I was doing backfill running a one task dag. There were over 1000 dag_runs
ready to run, I set concurrency=16 and max_active_run=16, I use Celery executor
and have 12 workers. Task instances were quickly scheduled, but they stuck
there and never get run. I did a search online and got this: [jira] [Updated]
(AIRFLOW-6190) Task instances queued and dequeued before worker is ready,
causing intermittently failed tasks
|
|
| |
[jira] [Updated] (AIRFLOW-6190) Task instances queued and dequeued befor...
|
|
|
I have the same problem. How can I fix the problem?
Thanks for your help!
Here is the sample log message:
[2020-03-24 18:58:58,786] {taskinstance.py:624} INFO - Dependencies not met for
<TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00 [scheduled]>,
dependency 'Task Instance State' FAILED: Task is in the 'scheduled' state which
is not a valid state for execution. The task must be cleared in order to be
run.[2020-03-24 18:58:58,791] {logging_mixin.py:112} INFO - [2020-03-24
18:58:58,791] {local_task_job.py:91} INFO - Task is not able to be
run[2020-03-24 18:59:52,968] {taskinstance.py:624} INFO - Dependencies not met
for <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
[scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
'scheduled' state which is not a valid state for execution. The task must be
cleared in order to be run.[2020-03-24 18:59:52,970] {logging_mixin.py:112}
INFO - [2020-03-24 18:59:52,969] {local_task_job.py:91} INFO - Task is not able
to be run[2020-03-24 19:01:26,563] {taskinstance.py:624} INFO - Dependencies
not met for <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
[scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
'scheduled' state which is not a valid state for execution. The task must be
cleared in order to be run.[2020-03-24 19:01:26,565] {logging_mixin.py:112}
INFO - [2020-03-24 19:01:26,565] {local_task_job.py:91} INFO - Task is not able
to be run[2020-03-24 19:05:09,928] {taskinstance.py:624} INFO - Dependencies
not met for <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
[scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
'scheduled' state which is not a valid state for execution. The task must be
cleared in order to be run.[2020-03-24 19:05:09,930] {logging_mixin.py:112}
INFO - [2020-03-24 19:05:09,930] {local_task_job.py:91} INFO - Task is not able
to be run[2020-03-24 19:09:08,609] {taskinstance.py:624} INFO - Dependencies
not met for <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
[scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
'scheduled' state which is not a valid state for execution. The task must be
cleared in order to be run.[2020-03-24 19:09:08,611] {logging_mixin.py:112}
INFO - [2020-03-24 19:09:08,611] {local_task_job.py:91} INFO - Task is not able
to be run[2020-03-24 19:11:39,150] {taskinstance.py:624} INFO - Dependencies
not met for <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
[scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
'scheduled' state which is not a valid state for execution. The task must be
cleared in order to be run.[2020-03-24 19:11:39,152] {logging_mixin.py:112}
INFO - [2020-03-24 19:11:39,152] {local_task_job.py:91} INFO - Task is not able
to be run[2020-03-24 19:13:51,094] {taskinstance.py:624} INFO - Dependencies
not met for <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
[scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
'scheduled' state which is not a valid state for execution. The task must be
cleared in order to be run.[2020-03-24 19:13:51,098] {logging_mixin.py:112}
INFO - [2020-03-24 19:13:51,097] {local_task_job.py:91} INFO - Task is not able
to be run[2020-03-24 19:15:42,225] {taskinstance.py:624} INFO - Dependencies
not met for <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
[scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
'scheduled' state which is not a valid state for execution. The task must be
cleared in order to be run.[2020-03-24 19:15:42,227] {logging_mixin.py:112}
INFO - [2020-03-24 19:15:42,227] {local_task_job.py:91} INFO - Task is not able
to be run
