[
https://issues.apache.org/jira/browse/AIRFLOW-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953595#comment-16953595
]
Ash Berlin-Taylor edited comment on AIRFLOW-5686 at 6/11/20, 8:41 AM:
----------------------------------------------------------------------
If anyone else runs in to this the workaround I used to fix it was to run this
SQL after upgrading:
{code:sql}
WITH inactive_dags AS (SELECT dag_id FROM dag WHERE is_active = false)
UPDATE ONLY task_instance SET state='removed' FROM inactive_dags
WHERE task_instance.dag_id=inactive_dags.dag_id
AND (state NOT IN ('failed', 'upstream_failed', 'success') or state IS NULL);
{code}
was (Author: ashb):
If anyone else runs in to this the workaround I used to fix it was to run this
SQL after upgrading:
{code:sql}
WITH inactive_dags AS (SELCT dag_id FROM dag WHERE is_active = false)
UPDATE ONLY task_instance SET state='removed' FROM inactive_dags
WHERE task_instance.dag_id=inactive_dags.dag_id
AND (state NOT IN ('failed', 'upstream_failed', 'success') or state IS NULL);
{code}
> Deleting DAG can leave scheduled/queued tasks consuming pool slots
> ------------------------------------------------------------------
>
> Key: AIRFLOW-5686
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5686
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.5
> Reporter: Ash Berlin-Taylor
> Priority: Critical
> Fix For: 2.0.0
>
>
> If you delete a dag file when it had tasks in the scheduled or queued state,
> those tasks instances are never touched again.
> With the slight tweak in Pool (making the default pool an explicit one) this
> now matters, and this ends up with the scheduler being "blocked" from running
> new tasks wiht this message:
> {noformat}
> Figuring out tasks to run in Pool(name=default_pool) with -9022 open slots
> and 45 task instances ready to be queued {noformat}
> The fix should be to set any task instance in a non-terminal state (None,
> queued, scheduled, running, up_for_retry etc.) to "removed" inside
> DAG.deactivate_stale_dags (which is already called on scheduler shutdown).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
