ephraimbuddy commented on issue #13542: URL: https://github.com/apache/airflow/issues/13542#issuecomment-845106707
> I can observe the same problem with version 2.0.2: > > * Tasks fail, because a DAG/task has gone missing (we are using dynamically created DAGs, and they can go missing) > * The scheduler keeps those queued > * The pool gradually fills up with these queued tasks > * The whole operation stops, because of this behaviour > > My current remedy: > > * Manually remove those queued tasks > > My desired solution: > > When a DAG/task goes missing while it is queued, it should end up in a failed state. Can you be able to try this with airflow master? We recently merged this https://github.com/apache/airflow/pull/15929. I'm thinking it will help in 'The scheduler keeps those queued' part -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
