It's a totally reasonable use case. As you said, currently it only moves forward, though it could also move backwards. You could add a DAG argument `schedule_past_dagruns=False` (we probably need a better name), that would enable this feature.
Here's the code that creates DagRuns, it seems like it may be easy to add a few lines to implement this feature. https://github.com/apache/incubator-airflow/blob/master/airflow/jobs.py#L761 Max On Mon, Apr 2, 2018 at 7:17 PM, David Capwell <[email protected]> wrote: > Nothing I know of. The scheduler finds the latest execution then creates > the next based off interval; this is also why update to start date have no > affect (doesn't try to fill gaps) > > On Mon, Apr 2, 2018, 11:26 AM Dennis O'Brien <[email protected]> > wrote: > > > Hi folks, > > > > I recently asked this question on gitter but didn't get any feedback. > > > > Anyone know if there is a way to get the scheduler to reverse the order > of > > the dag runs? By default a new DAG starts at start_date then moves > > sequentially forward in time until it is caught up (assuming > catchup=True). > > The same is true for a new DAG just enabled, or a DAG that is cleared, > and > > for a backfill. > > > > The behavior I'd like to get is for the scheduler to queue up the latest > > available, so it starts most recent, then moves back in time. If while > the > > backfill is running a more recent DAG run is eligible, that one should be > > queued next. > > > > Is there anyway to accomplish this? Is this a feature that others would > > find useful? > > > > For some background, I have some jobs that make predictions and do a long > > backfill for historical backtesting, and that can mean no new predictions > > for a week depending on the job and the time to backfill. Ideally the > most > > recent jobs would take precedence over the historical jobs. > > > > thanks, > > Dennis > > >
