[
https://issues.apache.org/jira/browse/AIRFLOW-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Sharma updated AIRFLOW-3775:
-------------------------------------
Description:
Currently, airflow scheduler picks DAGs for execution based on cron string
mentioned in the schedule.
I have a requirement in which I want to trigger the DAGs based on the event.
Currently, I am managing such requirement by running an application which
triggers DAGs when an event occurs. DAGs schedule with such requirement is
NONE and concurrency is 1.
Airflow considers all runs as manual triggers, and Airflow UI is messed up for
that. Also, multiple applications required for triggering DAGs based on
different condition.
I propose to have a multiple schedulers support in Airflow, and like queue
feature at a task level, we should have the scheduler option too at the task
level, which by default would be time-based however on providing specific value
will be picked by that instance of the scheduler.
was:
Currently, airflow scheduler picks DAGs for execution based on cron string
mentioned in the schedule.
I have a requirement in which I want to trigger the DAGs based on the event.
Currently, I am managing such requirement by running an application which
triggers DAGs when an event occurs. DAGs schedule with such requirement is
NONE and concurrency is 1.
Airflow considers all runs as manual triggers, and Airflow UI is messed up for
that. Also, multiple applications required for triggering DAGs based on
different condition.
I propose to have to multiple schedulers support in Airflow, and like queue
feature at a task level, we should have the scheduler option to at the task
level, which by default would be time-based however on providing specific value
will be picked by that instance of the scheduler.
> Support for custom task scheduler in airflow, apart of time based [default]
> one.
> --------------------------------------------------------------------------------
>
> Key: AIRFLOW-3775
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3775
> Project: Apache Airflow
> Issue Type: Wish
> Components: scheduler
> Reporter: Abhishek Sharma
> Priority: Major
> Attachments: Screenshot1.png, screenshot2.png
>
>
> Currently, airflow scheduler picks DAGs for execution based on cron string
> mentioned in the schedule.
> I have a requirement in which I want to trigger the DAGs based on the event.
> Currently, I am managing such requirement by running an application which
> triggers DAGs when an event occurs. DAGs schedule with such requirement is
> NONE and concurrency is 1.
> Airflow considers all runs as manual triggers, and Airflow UI is messed up
> for that. Also, multiple applications required for triggering DAGs based on
> different condition.
> I propose to have a multiple schedulers support in Airflow, and like queue
> feature at a task level, we should have the scheduler option too at the task
> level, which by default would be time-based however on providing specific
> value will be picked by that instance of the scheduler.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
