Sure thing. Current Behavior: - User creates DAG with default_args start date to 2015 - dagrun gets kicked off for 2015 - User changes default_args start date to 2016 - dagruns continue running for 2015
New Behavior: - User creates DAG with default_args start date to 2015 - dagrun gets kicked off for 2015 - User changes default_args start date to 2016 - *dagruns start running for the 2016 start date instead of 2015* On Tue, Mar 7, 2017 at 11:49 AM, Bolke de Bruin <[email protected]> wrote: > Hey Dan, > > Im not sure if I am seeing a difference for #1 vs now, except you are > excluding backfills now from the calculation? Can you provide an example? > > Bolke > > > On 7 Mar 2017, at 20:38, Dan Davydov <[email protected]> > wrote: > > > > A very common source of confusion for our users is when they specify > > start_date in default_args but not in their DAG arguments and then try to > > change this start_date to move the execution of their DAG forward (e.g. > > from 2015 to 2016). This doesn't work because the logic that is used to > > calculate the "initial" start date of a dag differs from the logic to > > calculate subsequent dagrun start dates. > > > > Current Airflow Logic: > > DS to schedule initial dagrun: dag.start_date if it exists, else > min(start > > date of tasks_of_dag) > > DS to schedule subsequent dagruns: last_dagrun + scheduled_interval > > > > There are a couple ways of addressing this: > > 1. Change the definition of start date for subsequent dagruns to match > the > > "initial" dagrun start date (calculated from the minimum of task start > > dates) > > 2. Force explicit dag start dates > > > > I personally like 1. > > > > I also propose that we throw errors for DAGs that have tasks that depend > on > > other tasks with start dates that occur after theirs (otherwise there > could > > be deadlocks). > > > > What do people think? > >
