Rolf Schroeder created AIRFLOW-2683:
---------------------------------------
Summary: priority_weights honored by LocalExecutor but not by
CeleryExecutor
Key: AIRFLOW-2683
URL: https://issues.apache.org/jira/browse/AIRFLOW-2683
Project: Apache Airflow
Issue Type: Bug
Components: celery, scheduler
Affects Versions: Airflow 1.7.1.3
Environment: Linux
Reporter: Rolf Schroeder
Attachments: airflow_priorities_celery.png,
airflow_priorities_dag.png, airflow_priorities_localexecutor.png,
test_priority.py
Hi,
I am running Airflow 1.7.1.3 with Celery 4.0.2 (and rabbitmq). I have come
across the following issue: It seems to me that task priorities are not honored
as expected when scheduling tasks via Celery. However, when using the
LocalExecutor, the priorities seem to work fine. This issue arises when low
prio tasks get scheduled/queued before high prio tasks. Celery will finish all
previously scheduled/queued low prio tasks before tackling the high prio ones.
In contrast, the LocalExecutor "correctly" interrupts processing of the low
prio tasks in favor of the high prio task. I fee like once Airflow has sent the
tasks to Celery, it "looses" control of the execution order. Details below. I
search in Jira and possibly the following issues are linked (although I am not
convinced)
(AIRFLOW-1510)[https://issues.apache.org/jira/browse/AIRFLOW-1510]
(AIRFLOW-584)[https://issues.apache.org/jira/browse/AIRFLOW-584]
I have the following test setup: Two independent initial tasks, each with 3
downstream tasks. One of the initial tasks takes longer but has high prio
downstream tasks. The other initial task has a short duration and low prio
downstream tasks. Both initial tasks start at the same time. My expectation is
that some of the low prio tasks get executed first (since their initial
upstream task is faster) but the moment the slower initial taks is done, its
high prio downstream tasks should get executed before the low prio downstream
tasks.
Here is a picture of the setup (forget about init0, this is just to make the
DAG look correct):
!airflow_priorities_dag.png!
I've been trying this with a LocalExecutor (AIRFLOW__CORE__PARALLELISM=2) and
two celery workers. When running the LocalExecutor, the high prio tasks get
indeed executed once their init task is done (as expected, the 'prio100' tasks
finish before the remaining 'prio10' tasks):
!airflow_priorities_localexecutor.png!
However, when using Celery, it seems that that the priorities are not honored
anymore. Once can see clearly that the low prio tasks (prio10) are all executed
before the high prio ones (prio100)
!airflow_priorities_celery.png!
Here is the corresponding code
[^test_priority.py]
I do not know whether this expected behavior or a bug? Is there any way to
"fix" this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
