Those semantics should not change with my specific proposal, but I think logically moving a DAG's start date back should backfill those old dagruns (not the current behavior which continues running from the current date). Interval changes are a bit of a hairy topic, I think those kinds of changes along with mutations of DAG topology (new tasks, task dependency changes, etc.) need to be thought out a little bit more and have a proposal drafted (e.g. I think that Airflow should support dags with tasks with different scheduling intervals). For the purposes of this proposal (and internally at Airbnb) we do not support interval changes and recommend a new DAG be created for these cases.
On Tue, Mar 7, 2017 at 1:52 PM, Bolke de Bruin <[email protected]> wrote: > Ok sounds good. What do you do with a dag that gets predated and with an > existing dag run? What happens if the interval changes, i.e. non cron > syntax? > > (Just thinking out loud) > > B. > > Sent from my iPhone > > > On 7 Mar 2017, at 22:27, Dan Davydov <[email protected]> > wrote: > > > > Sure thing. > > > > Current Behavior: > > - User creates DAG with default_args start date to 2015 > > - dagrun gets kicked off for 2015 > > - User changes default_args start date to 2016 > > - dagruns continue running for 2015 > > > > New Behavior: > > - User creates DAG with default_args start date to 2015 > > - dagrun gets kicked off for 2015 > > - User changes default_args start date to 2016 > > - *dagruns start running for the 2016 start date instead of 2015* > > > >> On Tue, Mar 7, 2017 at 11:49 AM, Bolke de Bruin <[email protected]> > wrote: > >> > >> Hey Dan, > >> > >> Im not sure if I am seeing a difference for #1 vs now, except you are > >> excluding backfills now from the calculation? Can you provide an > example? > >> > >> Bolke > >> > >>> On 7 Mar 2017, at 20:38, Dan Davydov <[email protected]> > >> wrote: > >>> > >>> A very common source of confusion for our users is when they specify > >>> start_date in default_args but not in their DAG arguments and then try > to > >>> change this start_date to move the execution of their DAG forward (e.g. > >>> from 2015 to 2016). This doesn't work because the logic that is used to > >>> calculate the "initial" start date of a dag differs from the logic to > >>> calculate subsequent dagrun start dates. > >>> > >>> Current Airflow Logic: > >>> DS to schedule initial dagrun: dag.start_date if it exists, else > >> min(start > >>> date of tasks_of_dag) > >>> DS to schedule subsequent dagruns: last_dagrun + scheduled_interval > >>> > >>> There are a couple ways of addressing this: > >>> 1. Change the definition of start date for subsequent dagruns to match > >> the > >>> "initial" dagrun start date (calculated from the minimum of task start > >>> dates) > >>> 2. Force explicit dag start dates > >>> > >>> I personally like 1. > >>> > >>> I also propose that we throw errors for DAGs that have tasks that > depend > >> on > >>> other tasks with start dates that occur after theirs (otherwise there > >> could > >>> be deadlocks). > >>> > >>> What do people think? > >> > >> >
