[
https://issues.apache.org/jira/browse/AIRFLOW-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman updated AIRFLOW-6175:
-
Description:
There is a bug caused by scheduler_jobs refactor which leads to task failure
and scheduler locking.
Essentially when a there is an overflow of tasks going into the scheduler, the
tasks are set back to scheduled, but are not removed from the executor's
queued_tasks queue.
This means that the executor will attempt to run tasks that are in the
scheduled state, but those tasks will fail dependency checks. Eventually the
queue is filled with scheduled tasks, and the scheduler can no longer run.
> Executor changes queued tasks to scheduled without removing from task queue
> ---
>
> Key: AIRFLOW-6175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6175
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.6
>Reporter: Daniel Imberman
>Priority: Blocker
>
> There is a bug caused by scheduler_jobs refactor which leads to task failure
> and scheduler locking.
> Essentially when a there is an overflow of tasks going into the scheduler,
> the tasks are set back to scheduled, but are not removed from the executor's
> queued_tasks queue.
> This means that the executor will attempt to run tasks that are in the
> scheduled state, but those tasks will fail dependency checks. Eventually the
> queue is filled with scheduled tasks, and the scheduler can no longer run.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
