dskoda1 opened a new issue #22529: URL: https://github.com/apache/airflow/issues/22529
### Apache Airflow version 2.2.3 ### What happened We are leveraging the ability to fetch secrets in the cfg files at runtime, by adding the `_cmd` suffix to certain configs, such as `sql_alchemy_conn_cmd` and `broker_url_cmd`, as described in [the docs](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html). This is working fine for the scheduler and webserver components - additionally, for the worker, all but the `sql_alchemy_conn_cmd` are working. Unless we set `AIRFLOW__CORE__SQL_ALCHEMY_CONN` as an environment variable, the worker will always try and connect to the (default) sqlite connection string for the `sql_alchemy_conn`. Sadly, if we run `airflow config list`, though we don't have it in our cfg file, `sql_alchemy_conn` is present and set to the default sqlite connection string. ### What you think should happen instead Airflow worker should behave the same as webserver and scheduler, and leverage `sql_alchemy_conn_cmd` given it is present in the cfg file, and not fall back to the default value for `sql_alchemy_conn`. ### How to reproduce In a config file, set `sql_alchemy_conn_cmd` to a legitimate db backend (i.e. not sqlite). Start webserver, scheduler, verify they are able to connect. Start a celery worker, and validate it starts up ok. Next, run any dag (we ran any of the example dags as long as they aren't dummy operators), "tutorial" dag is fine here. Click into a task that is ran and check logs, and see the following error: ``` Traceback (most recent call last): File "/bb/bin/package/c/csi-airflow2/1.1.0/libexec/csi-airflow2/dependencies/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/bb/bin/package/c/csi-airflow2/1.1.0/libexec/csi-airflow2/dependencies/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: no such table: csi_airflow2_db.dag_run ``` indicating it is trying to connect to sqlite rather than the configured DB backend. ### Operating System RHEL 7.6 ### Versions of Apache Airflow Providers ``` apache-airflow-providers-celery==2.1.0 # via apache-airflow apache-airflow-providers-docker==2.4.0 # via -r requirements.in apache-airflow-providers-ftp==2.0.1 # via apache-airflow apache-airflow-providers-http==2.0.2 # via apache-airflow apache-airflow-providers-imap==2.1.0 # via apache-airflow apache-airflow-providers-postgres==2.4.0 # via apache-airflow apache-airflow-providers-sqlite==2.0.1 # via apache-airflow ``` ### Deployment Virtualenv installation ### Deployment details Deployed using internal company tooling, roughly a virtualenv / self contained application ### Anything else the celery worker is able to parse and use other configs which use `_cmd`, except for sql_alchemy_conn. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
