We have an on-ongoing discussion at https://lists.apache.org/thread.html/2b12ae265795ff2e655a5161c972f5c7bbe60722a12849a0e2c5c55f%40%3Cdev.airflow.apache.org%3E if you'd like to add some thoughts over there.
On Thu, Sep 3, 2020 at 1:07 PM Jarek Potiuk <[email protected]> wrote: > I start thinking more and more that we also should support this kind > of case. We are not very far from it and people are using it this way > anyway, so it would be a bit turning a blind eye if we don't try to > accommodate - especially that it is not as difficult to do. > > I think maybe we then should have the ETL (or rather Data interval) > and non-Data Interval type of DAGs? > > And this actually goes hand-in-hand with the discussion about > "schedule at the end or beginning of the interval". > > Maybe introducing the two kinds of Dags might kill two birds with the > same stone? > > We could have "Data Interval DAGs" with cron specs pointing to the end > of the interval, and "non-data-interval" ones triggered according to > cron schedule. Those two seem related. > > Just a wild thought :) > > J. > > On Thu, Sep 3, 2020 at 1:44 PM Axovision Team > <[email protected]> wrote: > > > > With the already very deep focus on ETL processes, there is a rising > > interest in applying apache airflow also as a full stack scheduling > system > > for all types of processes. We are applying apache airflow as a > scheduling > > system at AXOVISION not only for ETL use cases, but also as a full > > replacement of cron jobs. But still there is functionality missing for a > > full stack scheduling system. But we think that adding these feature is > not > > to complex, because the fundamentals are already there. So it is like > > adding simple new features to airflow to reach this stage of full stack > > scheduling system, aside of ETL process management. The most important > > features to add would be: *Dynamic schedule intervals *and *Unpause/Pause > > without any catchup (no run of last recent) * > > > > Description > > There are several question (issues) on stack overflow, asking for the > need > > of a dynamic schedule interval. This means, the ability to change the > > schedule interval after DAG creation programmatically via API or CLI. > > Further, user alo asking for a detailed discussion on some stackoverflow > > questions, which could not be find. > > With the ability to dynamically change DAG schedule intervals, airflow > can > > increase user satisfaction and fully replace other custom cron like > > schedule systems. > > > > reference stack overflow links: > > > > - > > > https://stackoverflow.com/questions/63494560/airflow-schedule-interval-change > > - > > > https://stackoverflow.com/questions/63271671/can-we-parameterize-the-airflow-schedule-interval-dynamically-reading-from-the-v > > - > > > https://stackoverflow.com/questions/37294560/airflow-changing-the-crontab-time-for-a-dag-in-airflow > > > > Within the Pitfalls it is also mentioned: > > > > - https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls > > > > Use case / motivation > > As a user I want to change the schedule interval of an already defined > DAG, > > so that the DAG can run dynamically at different time points depended on > an > > external condition. > > > > The motivation is simple: there are a lot of use cases not fitting the > only > > once cron like created definition of a schedule interval of a DAG. (e.g. > > event driven schedules) > > > > Use Case: > > In the morning there was a database update announced, due to ongoing work > > in the backend, the database should be updated always 2 hours after the > > announcement. Which could be always a different point in time, according > to > > when the event happened. So the database update DAG needs to be scheduled > > on a new interval to run. > > > > Related Issues > > I could not find any directly related issues to that. > > > > -- > > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 >
