Apache Oozie has LAST_ONLY execution policy in which only the current instance will run when catching up or backfilling.
https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a6.3._Synchronous_Coordinator_Application_Definition _____________________________ From: siddharth anand <[email protected]<mailto:[email protected]>> Sent: Wednesday, August 17, 2016 5:39 PM Subject: Re: ShortCircuitIfNotCurrentOperator To: <[email protected]<mailto:[email protected]>> Hmn... none that I know of, but it seems pretty useful for us! On Wed, Aug 17, 2016 at 4:43 PM, Felix Cheung <[email protected]<mailto:[email protected]>> wrote: > Cool. Other scheduler has a concept called LAST_ONLY? > > > > > > > On Wed, Aug 17, 2016 at 2:34 PM -0700, "siddharth anand" < > [email protected]<mailto:[email protected]><mailto:[email protected]>> wrote: > > For the longest time, we have been discussing supporting full cron > functionality in Airflow. Currently, Airflow supports cron syntax with one > pretty visible issue. > > For example, if you wanted cron to schedule a job every day at midnight, > you would specify 0 0 * * *. In Airflow, if you were to pause your dag for > a week and then unpause it, your dag would run the past 7 days' worth of > dag runs. However, a common use-case for cron schedules is to use it for > some routine job scheduling (e.g. full db backup, log file cleanup, > minutely alert condition checking) that only needs to run at midnight, but > doesn't need to "backfill" the days missed. > > We had originally envisioned building this into the scheduler, but that is > unnecessarily complex. Instead, we have had the ShortCircuitOperator around > for a while. I plan to extend this to make a > ShortCircuitIfNotCurrentOperator. This will skip downstream for dag runs > that are not current. > > https://issues.apache.org/jira/browse/AIRFLOW-198 > > FYI.. alternate names for this operator are welcome. > -s >
