Hi ,
We have faced the similar issue and debugged the scheduler code and found
that back fill don't have much options given( Back fill logic is different
in more scheduling point) .In this case once the DAG is stuck it is stuck
. Instead of using backfill you can use trigger_dag which has more logic
and scheduler will automatically re-submit the stuck DAG's and
corresponding tasks will complete
Thanks & Regards
Pavan
Thanks & Regards
Pavan
On Wed, Mar 25, 2020 at 9:23 AM heng gu <heng...@yahoo.com.invalid> wrote:
> I was doing backfill running a one task dag. There were over 1000 dag_runs
> ready to run, I set concurrency=16 and max_active_run=16, I use Celery
> executor and have 12 workers. Task instances were quickly scheduled, but
> they stuck there and never get run. I did a search online and got this:
> [jira] [Updated] (AIRFLOW-6190) Task instances queued and dequeued before
> worker is ready, causing intermittently failed tasks
>
> |
> |
> | |
> [jira] [Updated] (AIRFLOW-6190) Task instances queued and dequeued befor...
>
>
> |
>
> |
>
> |
>
>
>
> I have the same problem. How can I fix the problem?
> Thanks for your help!
> Here is the sample log message:
> [2020-03-24 18:58:58,786] {taskinstance.py:624} INFO - Dependencies not
> met for <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
> [scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
> 'scheduled' state which is not a valid state for execution. The task must
> be cleared in order to be run.[2020-03-24 18:58:58,791]
> {logging_mixin.py:112} INFO - [2020-03-24 18:58:58,791]
> {local_task_job.py:91} INFO - Task is not able to be run[2020-03-24
> 18:59:52,968] {taskinstance.py:624} INFO - Dependencies not met for
> <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
> [scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
> 'scheduled' state which is not a valid state for execution. The task must
> be cleared in order to be run.[2020-03-24 18:59:52,970]
> {logging_mixin.py:112} INFO - [2020-03-24 18:59:52,969]
> {local_task_job.py:91} INFO - Task is not able to be run[2020-03-24
> 19:01:26,563] {taskinstance.py:624} INFO - Dependencies not met for
> <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
> [scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
> 'scheduled' state which is not a valid state for execution. The task must
> be cleared in order to be run.[2020-03-24 19:01:26,565]
> {logging_mixin.py:112} INFO - [2020-03-24 19:01:26,565]
> {local_task_job.py:91} INFO - Task is not able to be run[2020-03-24
> 19:05:09,928] {taskinstance.py:624} INFO - Dependencies not met for
> <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
> [scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
> 'scheduled' state which is not a valid state for execution. The task must
> be cleared in order to be run.[2020-03-24 19:05:09,930]
> {logging_mixin.py:112} INFO - [2020-03-24 19:05:09,930]
> {local_task_job.py:91} INFO - Task is not able to be run[2020-03-24
> 19:09:08,609] {taskinstance.py:624} INFO - Dependencies not met for
> <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
> [scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
> 'scheduled' state which is not a valid state for execution. The task must
> be cleared in order to be run.[2020-03-24 19:09:08,611]
> {logging_mixin.py:112} INFO - [2020-03-24 19:09:08,611]
> {local_task_job.py:91} INFO - Task is not able to be run[2020-03-24
> 19:11:39,150] {taskinstance.py:624} INFO - Dependencies not met for
> <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
> [scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
> 'scheduled' state which is not a valid state for execution. The task must
> be cleared in order to be run.[2020-03-24 19:11:39,152]
> {logging_mixin.py:112} INFO - [2020-03-24 19:11:39,152]
> {local_task_job.py:91} INFO - Task is not able to be run[2020-03-24
> 19:13:51,094] {taskinstance.py:624} INFO - Dependencies not met for
> <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
> [scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
> 'scheduled' state which is not a valid state for execution. The task must
> be cleared in order to be run.[2020-03-24 19:13:51,098]
> {logging_mixin.py:112} INFO - [2020-03-24 19:13:51,097]
> {local_task_job.py:91} INFO - Task is not able to be run[2020-03-24
> 19:15:42,225] {taskinstance.py:624} INFO - Dependencies not met for
> <TaskInstance: N20_L2_Pull.N20_L2_Pull 2020-03-08T19:00:00+00:00
> [scheduled]>, dependency 'Task Instance State' FAILED: Task is in the
> 'scheduled' state which is not a valid state for execution. The task must
> be cleared in order to be run.[2020-03-24 19:15:42,227]
> {logging_mixin.py:112} INFO - [2020-03-24 19:15:42,227]
> {local_task_job.py:91} INFO - Task is not able to be run
>
>
