vghar-bh opened a new issue, #32540: URL: https://github.com/apache/airflow/issues/32540
### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened variables are getting duplicated on upgrading airflow to 2.5.3 from 2.4.1. after upgrade when the airflow-sync-variables pod runs its not finding the existing variables and then adding it again. **logs from airflow-sync-variables:** /home/airflow/.local/lib/python3.8/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9) [[34m2023-07-09 05:59:12,443[0m] {[34msync_variables.py:[0m357} INFO[0m - BEGIN: airflow variables sync[0m [[34m2023-07-09 05:59:14,521[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`TEMP-BUCKET` is missing, adding...[0m [[34m2023-07-09 05:59:14,534[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`TEMP-BUCKET` was successfully added.[0m [[34m2023-07-09 05:59:14,538[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`PERSPECTIVE-BUCKET` is missing, adding...[0m [[34m2023-07-09 05:59:14,546[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`PERSPECTIVE-BUCKET` was successfully added.[0m [[34m2023-07-09 05:59:14,550[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`TAMR_INPUT_BUCKET_NAME` is missing, adding...[0m [[34m2023-07-09 05:59:14,553[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`TAMR_INPUT_BUCKET_NAME` was successfully added.[0m [[34m2023-07-09 05:59:14,562[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`TAMR_OUTPUT_BUCKET_NAME` is missing, adding...[0m [[34m2023-07-09 05:59:14,573[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`TAMR_OUTPUT_BUCKET_NAME` was successfully added.[0m [[34m2023-07-09 05:59:14,577[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`RAW-BUCKET` is missing, adding...[0m [[34m2023-07-09 05:59:14,580[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`RAW-BUCKET` was successfully added.[0m [[34m2023-07-09 05:59:14,585[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`EXTRACTED-BUCKET` is missing, adding...[0m [[34m2023-07-09 05:59:14,589[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`EXTRACTED-BUCKET` was successfully added.[0m [[34m2023-07-09 05:59:14,592[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`CATEGORIZED-BUCKET` is missing, adding...[0m [[34m2023-07-09 05:59:14,596[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`CATEGORIZED-BUCKET` was successfully added.[0m [[34m2023-07-09 05:59:14,600[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`STRUCTURED-BUCKET` is missing, adding...[0m [[34m2023-07-09 05:59:14,603[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`STRUCTURED-BUCKET` was successfully added.[0m [[34m2023-07-09 05:59:14,608[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`STANDARD-BUCKET` is missing, adding...[0m [[34m2023-07-09 05:59:14,612[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`STANDARD-BUCKET` was successfully added.[0m [[34m2023-07-09 05:59:14,616[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`NPD_DATE` is missing, adding...[0m [[34m2023-07-09 05:59:14,619[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`NPD_DATE` was successfully added.[0m [[34m2023-07-09 05:59:14,623[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-KUBERNETES-ANNOTATION` is missing, adding...[0m [[34m2023-07-09 05:59:14,631[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-KUBERNETES-ANNOTATION` was successfully added.[0m [[34m2023-07-09 05:59:14,635[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-DRIVER-MEMORY` is missing, adding...[0m [[34m2023-07-09 05:59:14,638[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-DRIVER-MEMORY` was successfully added.[0m [[34m2023-07-09 05:59:14,641[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-EXECUTOR-MEMORY` is missing, adding...[0m [[34m2023-07-09 05:59:14,645[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-EXECUTOR-MEMORY` was successfully added.[0m [[34m2023-07-09 05:59:14,648[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-EXECUTOR-CORES` is missing, adding...[0m [[34m2023-07-09 05:59:14,653[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-EXECUTOR-CORES` was successfully added.[0m [[34m2023-07-09 05:59:14,656[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-NUM-EXECUTORS` is missing, adding...[0m [[34m2023-07-09 05:59:14,659[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-NUM-EXECUTORS` was successfully added.[0m [[34m2023-07-09 05:59:14,662[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`DATACURATION-OUTPUT-BUCKET` is missing, adding...[0m [[34m2023-07-09 05:59:14,665[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`DATACURATION-OUTPUT-BUCKET` was successfully added.[0m [[34m2023-07-09 05:59:14,671[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`ENT_NTFN_USER` is missing, adding...[0m [[34m2023-07-09 05:59:14,676[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`ENT_NTFN_USER` was successfully added.[0m [[34m2023-07-09 05:59:14,694[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` is missing, adding...[0m [[34m2023-07-09 05:59:14,698[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` was successfully added.[0m [[34m2023-07-09 05:59:14,710[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`MYID_BASE_URL` is missing, adding...[0m [[34m2023-07-09 05:59:14,714[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`MYID_BASE_URL` was successfully added.[0m [[34m2023-07-09 05:59:14,727[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`DATALAKE_BUCKET` is missing, adding...[0m [[34m2023-07-09 05:59:14,731[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`DATALAKE_BUCKET` was successfully added.[0m [[34m2023-07-09 05:59:14,735[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`BITPULSE_WEBBITS_DATALAKE_PATH` is missing, adding...[0m [[34m2023-07-09 05:59:14,744[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`BITPULSE_WEBBITS_DATALAKE_PATH` was successfully added.[0m [[34m2023-07-09 05:59:14,749[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_DOCKER_IMAGE` is missing, adding...[0m [[34m2023-07-09 05:59:14,753[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_DOCKER_IMAGE` was successfully added.[0m [[34m2023-07-09 05:59:14,757[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_NAMESPACE` is missing, adding...[0m [[34m2023-07-09 05:59:14,760[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_NAMESPACE` was successfully added.[0m [[34m2023-07-09 05:59:14,765[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_MAPPING_FILE_PATH` is missing, adding...[0m [[34m2023-07-09 05:59:14,768[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_MAPPING_FILE_PATH` was successfully added.[0m [[34m2023-07-09 05:59:14,774[0m] {[34msync_variables.py:[0m360} INFO[0m - END: airflow variables sync[0m [[34m2023-07-09 06:00:14,864[0m] {[34msync_variables.py:[0m357} INFO[0m - BEGIN: airflow variables sync[0m [[34m2023-07-09 06:00:14,917[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`STRUCTURED-BUCKET` is missing, adding...[0m [[34m2023-07-09 06:00:14,921[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`STRUCTURED-BUCKET` was successfully added.[0m [[34m2023-07-09 06:00:14,952[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`SPARK-EXECUTOR-CORES` is missing, adding...[0m [[34m2023-07-09 06:00:14,958[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`SPARK-EXECUTOR-CORES` was successfully added.[0m [[34m2023-07-09 06:00:14,990[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` is missing, adding...[0m [[34m2023-07-09 06:00:14,994[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` was successfully added.[0m [[34m2023-07-09 06:00:15,013[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`MYID_REDIRECT_BASE_URL` is missing, adding...[0m [[34m2023-07-09 06:00:15,016[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`MYID_REDIRECT_BASE_URL` was successfully added.[0m [[34m2023-07-09 06:00:15,035[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_DOCKER_IMAGE` is missing, adding...[0m [[34m2023-07-09 06:00:15,040[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_DOCKER_IMAGE` was successfully added.[0m [[34m2023-07-09 06:00:15,044[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`UOM_NAMESPACE` is missing, adding...[0m [[34m2023-07-09 06:00:15,048[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`UOM_NAMESPACE` was successfully added.[0m [[34m2023-07-09 06:00:15,058[0m] {[34msync_variables.py:[0m360} INFO[0m - END: airflow variables sync[0m [[34m2023-07-09 06:01:15,132[0m] {[34msync_variables.py:[0m357} INFO[0m - BEGIN: airflow variables sync[0m [[34m2023-07-09 06:01:15,256[0m] {[34msync_variables.py:[0m336} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` is missing, adding...[0m [[34m2023-07-09 06:01:15,260[0m] {[34msync_variables.py:[0m348} INFO[0m - Variable=`ENT_NTFN_CONSUMER_COUNT` was successfully added.[0m [[34m2023-07-09 06:01:15,323[0m] {[34msync_variables.py:[0m360} INFO[0m - END: airflow variables sync[0m [[34m2023-07-09 06:02:15,419[0m] {[34msync_variables.py:[0m357} INFO[0m - BEGIN: airflow variables sync[0m ### What you think should happen instead After upgrade when the airflow-sync-variables pod runs it should finding the existing variables and skip creating it again. ### How to reproduce downgrade the airflow to 2.4.1 where its working upgrade it to airflow 2.5.3 once we sync the airflow-sync-variables pod the variables will be duplicated in the UI and in database variable table. ### Operating System N/A ### Versions of Apache Airflow Providers **on airflow 2.4.1 before upgrade** apache-airflow-providers-amazon 5.1.0 apache-airflow-providers-apache-livy 2.2.3 apache-airflow-providers-celery 3.0.0 apache-airflow-providers-cncf-kubernetes 4.4.0 apache-airflow-providers-common-sql 1.2.0 apache-airflow-providers-docker 3.2.0 apache-airflow-providers-elasticsearch 4.2.1 apache-airflow-providers-ftp 3.1.0 apache-airflow-providers-google 8.3.0 apache-airflow-providers-grpc 3.0.0 apache-airflow-providers-hashicorp 3.1.0 apache-airflow-providers-http 4.0.0 apache-airflow-providers-imap 3.0.0 apache-airflow-providers-microsoft-azure 4.3.0 apache-airflow-providers-mysql 3.2.1 apache-airflow-providers-odbc 3.1.2 apache-airflow-providers-postgres 5.2.2 apache-airflow-providers-redis 3.0.0 apache-airflow-providers-sendgrid 3.0.0 apache-airflow-providers-sftp 4.1.0 apache-airflow-providers-slack 5.1.0 apache-airflow-providers-sqlite 3.2.1 apache-airflow-providers-ssh 3.2.0 **on 2.5.3 after upgrade** apache-airflow-providers-amazon 5.1.0 apache-airflow-providers-apache-livy 2.2.3 apache-airflow-providers-celery 3.2.0 apache-airflow-providers-cncf-kubernetes 7.0.0 apache-airflow-providers-common-sql 1.5.1 apache-airflow-providers-docker 3.7.0 apache-airflow-providers-elasticsearch 4.5.0 apache-airflow-providers-ftp 3.4.1 apache-airflow-providers-google 10.1.1 apache-airflow-providers-grpc 3.2.0 apache-airflow-providers-hashicorp 3.4.0 apache-airflow-providers-http 4.4.1 apache-airflow-providers-imap 3.2.1 apache-airflow-providers-microsoft-azure 6.1.1 apache-airflow-providers-odbc 3.3.0 apache-airflow-providers-postgres 5.5.0 apache-airflow-providers-redis 3.2.0 apache-airflow-providers-sendgrid 3.2.0 apache-airflow-providers-sftp 4.3.0 apache-airflow-providers-slack 7.3.0 apache-airflow-providers-snowflake 4.1.0 apache-airflow-providers-sqlite 3.4.1 apache-airflow-providers-ssh 3.7.0 ### Deployment Official Apache Airflow Helm Chart ### Deployment details Airflow version 2.5.3 currently working with helm charts from Airflow Helm Chart (User Community) https://artifacthub.io/packages/helm/airflow-helm/airflow/8.7.0 using Argocd as GitOps CD tool for Kubernetes. ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
