anavrotski opened a new issue, #61518: URL: https://github.com/apache/airflow/issues/61518
### Apache Airflow version 3.1.7 ### If "Other Airflow 3 version" selected, which one? _No response_ ### What happened? After several hours since deploy - Internal Server Error. ``` wait-for-airflow-migrations wait-for-airflow-migrations /home/airflow/.local/lib/python3.12/site-packages/airflow/providers/common/compat/security/permissions.py:30 RemovedInAirflow4Warning: The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html wait-for-airflow-migrations 2026-02-05T16:44:38.119574Z [info [] Context impl MySQLImpl. [alembic.runtime.migration] loc=migration.py:210 wait-for-airflow-migrations 2026-02-05T16:44:38.119946Z [info [] Will assume non-transactional DDL. [alembic.runtime.migration] loc=migration.py:213 api-server | return meth( api-server | ^^^^^ api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection api-server | return connection._execute_clauseelement( api-server | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement api-server | ret = self._execute_context( api-server | ^^^^^^^^^^^^^^^^^^^^^^ api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context api-server | conn = self._revalidate_connection() api-server | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection api-server | self._invalid_transaction() api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction api-server | raise exc.PendingRollbackError( api-server | sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b) api-server +------------------------------------ api-server api-server During handling of the above exception, another exception occurred: api-server api-server Traceback (most recent call last): api-server File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi api-server result = await app( # type: ignore[func-returns-value] api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__ api-server await super().__call__(scope, receive, send) api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__ api-server await self.middleware_stack(scope, receive, send) api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__ api-server raise exc api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__ api-server await self.app(scope, receive, _send) api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__ api-server await responder(scope, receive, send) api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__ api-server await super().__call__(scope, receive, send) api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__ api-server await self.app(scope, receive, self.send_with_compression) api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__ api-server await self.app(scope, receive, send) api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in __call__ api-server with recv_stream, send_stream, collapse_excgroups(): api-server ^^^^^^^^^^^^^^^^^^^^ api-server File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__ api-server self.gen.throw(value) api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups api-server raise exc api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in __call__ api-server response = await self.dispatch_func(request, call_next) api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch api-server new_user, current_user = await self._refresh_user(current_token) api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user api-server user = await resolve_user_from_token(current_token) api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token api-server return await get_auth_manager().get_user_from_token(token_str) api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token api-server return self.deserialize_user(payload) api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in __call__ api-server return wrapper(self._obj, *args, **kwargs) api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper api-server v = method(self, *args, **kwargs) api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user api-server return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one() api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars api-server return self._proxied.scalars( api-server ^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars api-server return self._execute_internal( api-server ^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal api-server result: Result[Any] = compile_state_cls.orm_execute_statement( api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement api-server result = conn.execute( api-server ^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute api-server return meth( api-server ^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection api-server return connection._execute_clauseelement( api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement api-server ret = self._execute_context( api-server ^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context api-server conn = self._revalidate_connection() api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection api-server self._invalid_transaction() api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction api-server raise exc.PendingRollbackError( api-server sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b) stream closed: EOF for airflow/airflow-api-server-f6b468f8-45gwp (wait-for-airflow-migrations) api-server INFO: 172.25.127.162:45116 - "GET /api/v2/version HTTP/1.1" 200 OK api-server INFO: 172.25.127.162:45132 - "GET /api/v2/version HTTP/1.1" 200 OK ``` ### What you think should happen instead? _No response_ ### How to reproduce Just deploy and wait ~ 16 hours. ### Operating System linux ### Versions of Apache Airflow Providers _No response_ ### Deployment Official Apache Airflow Helm Chart ### Deployment details Official helm chart 1.1.8 ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
