To solve that exact problem, we ran a DAG on a frequent schedule, that basically acted as a scheduler. It used a shell script to kick off other DAGS. Possibly a custom scheduler would be a more elegant solution.
Thanks, Ben -- Ben Tallman - 503.680.5709 On Thu, Jan 31, 2019 at 11:03 AM abhishek sharma <[email protected]> wrote: > Hi Ben, > > Just copying my comment form ticket. > > I think current airflow scheduler schedule DAGs only on time-basis (based > on cron schedule string). ***Is it correct understanding?* > > How to approach a scenario where I want to trigger a DAG based on some > event which is not so predictable/regular on time basis? > > - One is to use sensor and DAG flow will start by first running a sensor > task for checking an event and when an event happened actual processing > get > a start. > - Second, is to have a DAG with None schedule and it gets triggered by > some other application or utility which checks for an event occurrence . > > If your most of the flow(DAGs) are suppose to run on this logic, then > sensor doesn't make sense and hence left with the second approach only, > which is nothing but a custom scheduling of DAGs. ****Is it agreeable use > case for the custom scheduler?* > > Thanks, {{Abhishek}} > > > > On Thu, Jan 31, 2019 at 11:53 AM Ben Tallman <[email protected]> wrote: > > > Can you explain a bit more what you are thinking for a custom scheduler? > > It's been awhile, but we added support for cron schedules without > backfill > > awhile back, so I'm wondering what you are thinking of adding with this? > > > > Thanks, > > Ben > > > > -- > > Ben Tallman - 503.680.5709 > > > > > > On Thu, Jan 31, 2019 at 8:29 AM abhishek sharma < > > [email protected]> > > wrote: > > > > > Hi All, > > > > > > Created a ticket(https://issues.apache.org/jira/browse/AIRFLOW-3775) > for > > > supporting custom scheduler in Airflow. > > > > > > The idea is to have a scheduler base class which can be extended for > > > writing a custom scheduler. The logic of custom scheduling is user > > > specific, and at the DAGs task level we can mention the scheduler type, > > and > > > that scheduler will be used for starting a task. [Naive Idea] > > > > > > Can we please discuss whether we need such functionality in Airflow or > > not > > > and If yes then we will proceed with the design and implementation. > > > > > > Thanks > > > Abhishek Sharma > > > > > >
