[
https://issues.apache.org/jira/browse/AIRFLOW-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556081#comment-16556081
]
Eric Horne commented on AIRFLOW-1941:
-------------------------------------
Another side-effect is the scheduler loses it's list of currently running
tasks, which means it can allow more tasks that parallelism would otherwise
limit. If running tasks is 20, parallelism is 32 and queued tasks is 0, upon
immediate restart, the scheduler will believe running tasks is 0 – which
ultimately leads to it computing open_slots to be the value of parallelism (in
this case 32), even though there really are still 20 tasks running. In this
scenario it would be possible for the scheduler to schedule up to 32 additional
tasks for a total of 52 tasks running, even though parallelism is set to only
32.
A potential solution here is to repopulate the self.running list with all the
currently running tasks in the __init__ of BaseExecutor. I am not familiar
enough with the code to know if it is easy or wise to perform database lookups
to pull all running task instance back into the self.running initialization,
but that seems like the obvious direction.
> Scheduler / executor loses tasks on restart when enforcing parallelism limit
> ----------------------------------------------------------------------------
>
> Key: AIRFLOW-1941
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1941
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.8.1, 1.9.0
> Environment: Linux
> Reporter: Joseph Harris
> Priority: Major
>
> When running the scheduler with a limited number of cycles - eg:
> {{airflow scheduler -n 30}}
> and with {{PARALLELISM=32}} set in airflow.cfg
> The Executor checks that {{len(self.running) < PARALLELISM}} before calling
> {{execute_async()}}
> https://github.com/apache/incubator-airflow/blob/master/airflow/executors/base_executor.py#L98
> When {{self.running}} is full for an extended period of time, the scheduler
> can exit without having scheduled the remaining tasks in
> {{self.queued_tasks}}. When it restarts, the lots tasks in
> {{self.queued_tasks}} don't get scheduled again, and get stuck in the queued
> state until manually kicked.
> We experienced issues with this when exiting tasks with clashing PIDs caused
> the CeleryExecutor's {{self.running}} to become full of zombie jobs that
> could not complete.
> * The Executor should not hold 'queued' tasks for an extended period of time,
> as it may exit for any reason. The parallelism constraint should be checked
> alongside other dependencies.
> * When shutting down 'gracefully', the scheduler should at least log a
> warning if there are any tasks in self.queued_tasks
> * Parallelism should be set to infinity if a queue-based/distributed executor
> is being used (more risky)
> This may be a common cause of tasks getting stuck in the 'queued' state when
> running Celery.
> Although AIRFLOW-900 is resolved in 1.9.0, this issue is still present, and
> the scheduler is still at risk of exiting without having scheduled tasks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
