TPapajCin opened a new issue #19707: URL: https://github.com/apache/airflow/issues/19707
### Apache Airflow version 2.2.2 (latest released) ### Operating System Debian 10 ### Versions of Apache Airflow Providers apache-airflow-providers-apache-cassandra==2.0.1 \ apache-airflow-providers-apache-hive==2.0.2 \ apache-airflow-providers-celery==2.1.0 \ apache-airflow-providers-cncf-kubernetes==2.0.2 \ apache-airflow-providers-ftp==2.0.1 \ apache-airflow-providers-http==2.0.1 \ apache-airflow-providers-imap==2.0.1 \ apache-airflow-providers-jdbc==2.0.1 \ apache-airflow-providers-mysql==2.1.1 \ apache-airflow-providers-papermill==2.0.1 \ apache-airflow-providers-postgres==2.2.0 \ apache-airflow-providers-sftp==2.1.1 \ apache-airflow-providers-sqlite==2.0.1 \ apache-airflow-providers-ssh==2.1.1 \ apache-airflow-providers-google==5.1.0 \ apache-airflow-providers-apache-beam==3.1.0 \ ### Deployment Other Docker-based deployment ### Deployment details Kubernetes - GKE KubernetesCeleryExecutor Airflow configuration stored in configmap and secrets ### What happened I bumped into extremely frustrating bug that breaks my Airflow deployment. I have a CI/CD K8S pipeline that: - Creates new configmaps and secrets - Deploys new version of deoployment file with new configmaps and secrets - After deployment, deletes old configmaps and secrets. Pod Template contains also contains references to configmaps and secrets and here lies the problem. After upgrade from 2.1.3 to 2.2.2 very small portion of DAGs is using old configmap names, even from deployments that are ~2-3 deployems old. Out of 29 DAGs it happens to ~2-4 of them and often next deployments are fixing one but breaking other DAGs. "K8s Pod Spec" on Airflow shows correct configmap names but once they are deployed on k8s, the wrong one is used and this prevents pod from starting because these old configmaps and secrets do not exist anymore. I can't tell what could be the reason but I guess that Airflow must be storing configmap and secret names in database somehow. It did not happen in 2.1.3 version or any other versions before. ### What you expected to happen K8s pods deployed by Airflow should have correct configmap and secret names used. ### How to reproduce - Deploy airflow with kubernetes executor. - Create a pod template that contains a secret or configmap. - Get lots of DAGs - the more the better because it doesn't happen to every DAG. Let them run. - Update the pod template with new configmap or secret. - Restart Airflow instance with that new pod template. - Immediately after previous step, delete old configmaps or secrets that are no longer used(but were before) in pod template. - Wait for new tasks of DAGs to run. - Check that small amount of new pods is trying to still use old configmap/secret and they fail because of that. ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
