newTypeGeek opened a new issue, #45796: URL: https://github.com/apache/airflow/issues/45796
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.5.3 ### What happened? # Background - Airflow is deployed in kuberenetes (k8s) cluster - The DAG consists of daily tasks that uses `KubernetesPodOperator` - The DAG is set with `max_active_runs=1` (because we don't have concurrent write) # Issue When we backfill the data, we go into the `airflow-scheduler` bash shell in k8s, and execute `airflow backfill` command, hoping that the backfill would complete over the weekend. At some point, the shell is exited. Then the next Monday, I still find the the airflow web UI shows the DAG is still running. However, when I check the kubernetes, no such Pod is running. Since DAG is set with `max_active_runs=1`, it cannot keep consuming the latest data. # Question How could I prevent such inconsistency between Airflow and the actual running tasks in Kubernetes? ### What you think should happen instead? _No response_ ### How to reproduce - Airflow running in Kubernetes cluster - Have a DAG (`max_active_run = 1`) with tasks using `KubernetesPodOperator` - Execute into `airflow-scheduler` bash shell, and run `airflow backfill` command (Note that it is expected that the backfill takes a long time to run, say > 30 minutes) - Then, at some point during backfill, you exit the bash shell It is expected that the Airflow web UI is still observing a running task, but no such Pod in kubernetes cluster ### Operating System Debian GNU/Linux 11 (bullseye) ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
