[
https://issues.apache.org/jira/browse/AIRFLOW-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Davydov updated AIRFLOW-536:
--------------------------------
Description:
At the moment Dags only get one dag scheduled per scheduler loop (which can be
quite slow, on the order of minutes), but what should happen is that all
pending dagruns for a dag should be scheduled.
In addition to dagruns being scheduled more quickly for dags that have multiple
dag runs queued up (e.g. paused dags), fixing this issue will enable a
"only_run_latest" option for dags (so e.g. if the dag is paused and then later
unpaused, only the most recent ds will be run), as well as facilitating
compatibility between backfills and the scheduler (if you run a backfill over
100 dag runs the backfill will be waiting on 100 scheduler cycles).
was:
At the moment Dags only get one dag scheduled per scheduler loop (which can be
quite slow, on the order of minutes), but what should happen is that all
pending dagruns for a dag should be scheduled.
In addition to dagruns being scheduled more quickly for dags that havem ultiple
dag runs queued up (e.g. paused dags), fixing this issue will enable a
"only_run_latest" option for dags (so e.g. if the dag is paused and then later
unpaused, only the most recent ds will be run), as well as facilitating
compatibility between backfills and the scheduler (if you run a backfill over
100 dag runs the backfill will be waiting on 100 scheduler cycles).
> Dags only get one dagrun is scheduled per scheduler loop
> --------------------------------------------------------
>
> Key: AIRFLOW-536
> URL: https://issues.apache.org/jira/browse/AIRFLOW-536
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Dan Davydov
>
> At the moment Dags only get one dag scheduled per scheduler loop (which can
> be quite slow, on the order of minutes), but what should happen is that all
> pending dagruns for a dag should be scheduled.
> In addition to dagruns being scheduled more quickly for dags that have
> multiple dag runs queued up (e.g. paused dags), fixing this issue will enable
> a "only_run_latest" option for dags (so e.g. if the dag is paused and then
> later unpaused, only the most recent ds will be run), as well as facilitating
> compatibility between backfills and the scheduler (if you run a backfill over
> 100 dag runs the backfill will be waiting on 100 scheduler cycles).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
