Brecht De Vlieger created AIRFLOW-6233:
------------------------------------------
Summary: Task get stuck on scheduled on LocalExecutor
Key: AIRFLOW-6233
URL: https://issues.apache.org/jira/browse/AIRFLOW-6233
Project: Apache Airflow
Issue Type: Bug
Components: scheduler
Affects Versions: 1.10.6
Reporter: Brecht De Vlieger
Attachments: DAG.png, airflow.cfg, expected_gantt.png, logs.tar.gz,
observed_gantt.png, test.py
Use case description:
One DAG with a sequence of tasks run by the LocalExecutor. The number of
parallel tasks (6) to be run is higher than the concurrency limit (3)
(parallelism and dag_concurrency). An image of the DAG layout is attached below.
Expected behaviour:
Scheduler uses the three available slots to run as many tasks concurrently as
possible. The DAG finishes in some finite time without restarting the
scheduler. Example Gantt chart attached below.
Observed behaviour:
The tasks are not optimally scheduled and after a while the tasks no longer
move from the 'scheduled' state to 'queued'. When the scheduler is restarted
the tasks are picked up again as expected. The Gantt chart is also in the
attachments. Note that the chart does not show the tasks that never get
scheduled. You can find the logs in the attachments.
How to reproduce:
Install vanilla airflow version 1.10.6 and run with LocalExecutor (I used a
mysql database). The airflow.cfg and dag file are added to the attachments. The
problem does not exist in 1.10.5.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
