[
https://issues.apache.org/jira/browse/AIRFLOW-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755213#comment-16755213
]
Abhishek Sharma commented on AIRFLOW-3775:
------------------------------------------
I think current airflow scheduler schedule DAGs only on time-basis (based on
cron schedule string). Is it correct understanding?
How to approach a scenario where I want to trigger a DAG based on some event
which is not so predictable/regular on time basis?
* One is to use sensor and DAG flow will start by first running a sensor task
for checking an event and when an event happened actual processing get a start.
* Second, is to have a DAG with None schedule and it gets triggered by some
other application or utility which checks for an event occurrence.
If your most of the flow(DAGs) are suppose to run on this logic, then sensor
doesn't make sense and hence left with the second approach, which is nothing a
custom scheduling of DAGs. I am sure many users have such requirements. Is it
agreeable use case or not?
I think the first logical step for this JIRA is to determine and decide whether
we should have such functionality in Airflow or not.
> Support for custom task scheduler in airflow, apart of time based [default]
> one.
> --------------------------------------------------------------------------------
>
> Key: AIRFLOW-3775
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3775
> Project: Apache Airflow
> Issue Type: Wish
> Components: scheduler
> Reporter: Abhishek Sharma
> Priority: Major
> Attachments: Screenshot1.png, screenshot2.png
>
>
> Currently, airflow scheduler picks DAGs for execution based on cron string
> mentioned in the schedule.
> I have a requirement in which I want to trigger the DAGs based on the event.
> Currently, I am managing such requirement by running an application which
> triggers DAGs when an event occurs. DAGs schedule with such requirement is
> NONE and concurrency is 1.
> Airflow considers all runs as manual triggers, and Airflow UI is messed up
> for that. Also, multiple applications required for triggering DAGs based on
> different condition.
> I propose to have to multiple schedulers support in Airflow, and like queue
> feature at a task level, we should have the scheduler option to at the task
> level, which by default would be time-based however on providing specific
> value will be picked by that instance of the scheduler.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
