Recently I looked into the topic of *Unpause/Pause without any catchup (no run of last recent) * and I have written some comments in github: https://github.com/apache/airflow/issues/9914#issuecomment-680934697
Basically, last-recent catchup appears to be a catchup but it is just a longer-delayed scheduling. More detailed elaboration is in the comment for anyone interested. I do not have a clean solution in mind yet. Zikun On Thu, 3 Sep 2020 at 19:44, Axovision Team <[email protected]> wrote: > With the already very deep focus on ETL processes, there is a rising > interest in applying apache airflow also as a full stack scheduling system > for all types of processes. We are applying apache airflow as a scheduling > system at AXOVISION not only for ETL use cases, but also as a full > replacement of cron jobs. But still there is functionality missing for a > full stack scheduling system. But we think that adding these feature is not > to complex, because the fundamentals are already there. So it is like > adding simple new features to airflow to reach this stage of full stack > scheduling system, aside of ETL process management. The most important > features to add would be: *Dynamic schedule intervals *and *Unpause/Pause > without any catchup (no run of last recent) * > > Description > There are several question (issues) on stack overflow, asking for the need > of a dynamic schedule interval. This means, the ability to change the > schedule interval after DAG creation programmatically via API or CLI. > Further, user alo asking for a detailed discussion on some stackoverflow > questions, which could not be find. > With the ability to dynamically change DAG schedule intervals, airflow can > increase user satisfaction and fully replace other custom cron like > schedule systems. > > reference stack overflow links: > > - > > https://stackoverflow.com/questions/63494560/airflow-schedule-interval-change > - > > https://stackoverflow.com/questions/63271671/can-we-parameterize-the-airflow-schedule-interval-dynamically-reading-from-the-v > - > > https://stackoverflow.com/questions/37294560/airflow-changing-the-crontab-time-for-a-dag-in-airflow > > Within the Pitfalls it is also mentioned: > > - https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls > > Use case / motivation > As a user I want to change the schedule interval of an already defined DAG, > so that the DAG can run dynamically at different time points depended on an > external condition. > > The motivation is simple: there are a lot of use cases not fitting the only > once cron like created definition of a schedule interval of a DAG. (e.g. > event driven schedules) > > Use Case: > In the morning there was a database update announced, due to ongoing work > in the backend, the database should be updated always 2 hours after the > announcement. Which could be always a different point in time, according to > when the event happened. So the database update DAG needs to be scheduled > on a new interval to run. > > Related Issues > I could not find any directly related issues to that. >
