StewartThomson opened a new issue #18446:
URL: https://github.com/apache/airflow/issues/18446
### Apache Airflow version
2.1.2
### Operating System
Docker on Amazon Linux 2
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### What happened
Dynamic dags sporadically result in a scheduler error of:
`ERROR - Executor reports task instance <TaskInstance: {task_id} 2021-09-21
00:00:00+00:00 [queued]> finished (failed) although the task says its queued.
(Info: None) Was the task killed externally?`
This pretty much breaks the whole DAG. The task instance gets stuck in a
queued state, and the DAG is unable to create new DAG runs.
It's worth noting that nothing is killing the task, and this only happens
for dynamic dags. No other DAGs in the system are affected by these new dags.
### What you expected to happen
I expect the tasks to not be stuck queued, and I really expect the DAG to be
able to create new dag runs.
### How to reproduce
1. Create a file that produces multiple dynamic DAGs
2. Create multiple DAG runs on any of the dynamic dags
3. Wait. Some number of tasks will become stuck in a queued state, the
scheduler will produce the error above, and no new DAG runs will be able to be
created.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
