romina-nikolova opened a new issue, #72803: URL: https://github.com/apache/airflow/issues/72803
### Under which category would you file this issue? Providers ### Apache Airflow version 3.3.1 ### What happened and how to reproduce it? KubernetesExecutor issues 2-3 delete_pod API calls for the same finished worker pod instead of one. The first call succeeds; the rest return 404 from the Kubernetes API. All calls come from the same scheduler process for the same TaskInstanceKey, milliseconds apart. Observed on a production-scale cluster: over a 15-minute sample, 306 pods received 3 delete calls, 15 received 2, and only 4 received the intended 1. That's ~1.47 delete calls per pod and a 67% 404 rate, generating a large amount of unnecessary kube-apiserver audit-log volume (and associated CloudWatch cost on EKS) at scale. Mapped tasks (non-null map_index) appear to be the common factor in every case we captured. Root cause In the result-processing path (kubernetes_executor.py / kubernetes_executor_utils.py in apache-airflow-providers-cncf-kubernetes), delete_pod is invoked before the executor's self.running dedup check runs for that TaskInstanceKey. Because the same task-completion event gets processed more than once before the dedup check has a chance to short-circuit it, the pod delete is issued redundantly on each pass. ### What you think should happen instead? The dedup check against self.running should happen (or the running-set be updated) before delete_pod is called, so a given TaskInstanceKey only ever triggers one delete call, regardless of how many times its completion event is processed. ### Operating System _No response_ ### Deployment None ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers apache-airflow-providers-cncf-kubernetes==10.21.0 ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
