I think it is a useful feature that nonetheless adds disproportionate
complexity -- for example, is there logic for when there is a task
downstream from an ad-hoc task, and the ad-hoc task isn't being run?
Perhaps there is a way to reengineer it around current Airflow idioms.
Maybe we can start by
The idea there is to be able to ship on-demand tasks along with your DAG.
Is it not used because it's not documented?
Deprecating may be harder than maintaining it. We'd have to start warning
about deprecation in 2.0 soon and add the PR that removes this in the
[eventual] 2.0 branch.
I don't feel
Yea, it feels like a pretty edge-use case. It's not even documented. In the
interest of simplifying and and reducing bugs it seems like we might just
want to nuke this, or completely rethink the use cases.
On Tue, May 17, 2016 at 12:22 PM, Jeremiah Lowin wrote:
> Perhaps ad-hoc tasks could be re
Perhaps ad-hoc tasks could be refractored as ad-hoc DAGs? It sounds like
they are for infrequent initialization or maintainence tasks.
On Tue, May 17, 2016 at 11:21 AM Arthur Wiedmer wrote:
> We still have tasks in production that use this feature.
>
> Sometimes, it has been used for one off tas
We still have tasks in production that use this feature.
Sometimes, it has been used for one off tasks that create simple static
mapping tables (Tables loaded from a static file that also lives in source
control, creating a programmatically generated time dimension etc...).
Of course, maybe just
I'm all for it to remove it as I have difficulty imagining a use case and
increases complexity in several places, but maybe max has one and is still
using it?
Sent from my iPhone
> On 17 mei 2016, at 19:40, Chris Riccomini wrote:
>
> @Bolke/@Jeremiah
>
> When you make your changes to unify t
@Bolke/@Jeremiah
When you make your changes to unify the backfiller and scheduler, it sounds
like this can go away, right?
On Tue, May 17, 2016 at 10:38 AM, Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:
> The scheduler won't trigger where `adhoc=True`. The CLI's backfill/test/run
> is t
The scheduler won't trigger where `adhoc=True`. The CLI's backfill/test/run
is the only way to trigger where `adhoc=True`. For backfill specifically,
there's a `-a`, `--include_adhoc` flag to make these tasks in-scope to the
backfill.
Max
On Tue, May 17, 2016 at 10:14 AM, Chris Riccomini
wrote:
