It's a totally reasonable use case. As you said, currently it only moves
forward, though it could also move backwards. You could add a DAG argument
`schedule_past_dagruns=False` (we probably need a better name), that would
enable this feature.

Here's the code that creates DagRuns, it seems like it may be easy to add a
few lines to implement this feature.
https://github.com/apache/incubator-airflow/blob/master/airflow/jobs.py#L761

Max

On Mon, Apr 2, 2018 at 7:17 PM, David Capwell <dcapw...@gmail.com> wrote:

> Nothing I know of.  The scheduler finds the latest execution then creates
> the next based off interval; this is also why update to start date have no
> affect (doesn't try to fill gaps)
>
> On Mon, Apr 2, 2018, 11:26 AM Dennis O'Brien <den...@dennisobrien.net>
> wrote:
>
> > Hi folks,
> >
> > I recently asked this question on gitter but didn't get any feedback.
> >
> > Anyone know if there is a way to get the scheduler to reverse the order
> of
> > the dag runs? By default a new DAG starts at start_date then moves
> > sequentially forward in time until it is caught up (assuming
> catchup=True).
> > The same is true for a new DAG just enabled, or a DAG that is cleared,
> and
> > for a backfill.
> >
> > The behavior I'd like to get is for the scheduler to queue up the latest
> > available, so it starts most recent, then moves back in time. If while
> the
> > backfill is running a more recent DAG run is eligible, that one should be
> > queued next.
> >
> > Is there anyway to accomplish this?  Is this a feature that others would
> > find useful?
> >
> > For some background, I have some jobs that make predictions and do a long
> > backfill for historical backtesting, and that can mean no new predictions
> > for a week depending on the job and the time to backfill.  Ideally the
> most
> > recent jobs would take precedence over the historical jobs.
> >
> > thanks,
> > Dennis
> >
>

Reply via email to