HannoGao opened a new issue #21895: URL: https://github.com/apache/airflow/issues/21895
### Apache Airflow version 2.2.4 (latest released) ### What happened ``` from airflow.models import Variable var = Variable.get(key_var) ``` When this is called during a special function during the normal scheduling job e.g. from somewhere in scheduler_job e.g. here [scheduler_job](https://github.com/apache/airflow/blob/737c0efff929710c0a5abbaa8de3575a55297aa5/airflow/jobs/scheduler_job.py#L413) gives the following error: in [variable.py](https://github.com/apache/airflow/blob/6b0ca646ec849af91fe8a10d3d5656cafa3ed4bd/airflow/models/variable.py#L256) ``` [2022-03-01 12:11:05,768] {variable.py:271} ERROR - Unable to retrieve variable from secrets backend (MetastoreBackend). Checking subsequent secrets backend. Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/airflow/models/variable.py", line 267, in get_variable_from_secrets var_val = secrets_backend.get_variable(key=key) File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 70, in wrapper return func(*args, session=session, **kwargs) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 32, in create_session session.commit() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1046, in commit self.transaction.commit() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 504, in commit self._prepare_impl() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 472, in _prepare_impl self.session.dispatch.before_commit(self.session) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/event/attr.py", line 322, in __call__ fn(*args, **kw) File "/usr/local/lib/python3.8/dist-packages/airflow/utils/sqlalchemy.py", line 265, in _validate_commit raise RuntimeError("UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!") RuntimeError: UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS! ``` This is not the case, when calling the same functionality via the RestAPI. Getting variables during the scheduling was possibile e.g. in Airflow 1. ### What you expected to happen RuntimeError("UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!") in sqlalchemy. Probably due to the fact, the function is called during scheduling. ### How to reproduce Set and Get a "test Variable in the scheduler and trigger a dag. ### Operating System container with ubuntu 20.04 ### Versions of Apache Airflow Providers _No response_ ### Deployment Other Docker-based deployment ### Deployment details microk8s version 1.18/stable, container for scheduler, webserver and postgres ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
