ido177 opened a new issue, #57065:
URL: https://github.com/apache/airflow/issues/57065
### Apache Airflow version
3.1.0
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
After upgrading to `apache-airflow-providers-fab==3.0.0`, which uses Flask
AppBuilder 5, we started observing intermittent authentication-related crashes
that make the Airflow Api Server unavailable.
The issue appears to be triggered by an expired or invalid JWT token, after
which the Flask AppBuilder authentication manager fails to recover gracefully
and raises a PendingRollbackError in SQLAlchemy.
```
2025-10-22T07:25:37.653971Z [error ] JWT token is not valid: Signature
has expired [airflow.api_fastapi.auth.managers.base_auth_manager]
loc=base_auth_manager.py:107
INFO: 10.67.70.212:57914 - "GET /ui/config HTTP/1.1" 401 Unauthorized
2025-10-22T07:25:37.824145Z [error ] Exception on /login/ [GET]
[airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py",
line 2529, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py",
line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py",
line 1821, in full_dispatch_request
rv = self.preprocess_request()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py",
line 2313, in preprocess_request
rv = self.ensure_sync(before_func)()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/security_manager.py",
line 57, in before_request
g.user = get_auth_manager().get_user()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py",
line 278, in get_user
if current_user.is_anonymous and getattr(g, "user", None) is not None
and not g.user.is_anonymous:
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line
316, in __get__
obj = instance._get_current_object()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line
520, in _get_current_object
return get_name(local()) # type: ignore
^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line
25, in <lambda>
current_user = LocalProxy(lambda: _get_user())
^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line
370, in _get_user
current_app.login_manager._load_user()
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_login/login_manager.py",
line 364, in _load_user
user = self._user_callback(user_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py",
line 1384, in load_user
user = self.get_user_by_id(int(pk))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py",
line 1390, in get_user_by_id
return self.session.get(self.user_model, pk)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py",
line 2853, in get
return self._get_impl(
^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py",
line 2975, in _get_impl
return db_load_fn(
^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/loading.py",
line 530, in load_on_pk_identity
session.execute(
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py",
line 1717, in execute
result = conn._execute_20(statement, params or {}, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
line 1710, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py",
line 334, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
line 1577, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
line 1808, in _execute_context
conn = self._revalidate_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
line 650, in _revalidate_connection
self._invalid_transaction()
File
"/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py",
line 622, in _invalid_transaction
raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid
transaction is rolled back. (Background on this error at:
https://sqlalche.me/e/14/8s2b)
```
After this occurs, the Api Server becomes unresponsive until it’s restarted.
### What you think should happen instead?
The system should gracefully handle expired JWT tokens - e.g., invalidate
the session, redirect the user to /login, and not leave SQLAlchemy sessions in
a broken state.
### How to reproduce
Unfortunately, this issue is intermittent - it’s not clear how to
consistently reproduce it.
It tends to occur when:
- The user session has expired and the UI tries to refresh(/ui/config or
/login/),
- Or when multiple requests hit the webserver simultaneously after an idle
period.
### Operating System
Debian
### Versions of Apache Airflow Providers
apache-airflow==3.1.0
apache-airflow-providers-google==18.0.0
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-slack==9.3.0
apache-airflow-providers-vertica==4.1.2
apache-airflow-providers-apache-livy==4.4.3
apache-airflow-providers-ssh==4.1.4
apache-airflow-providers-common-sql==1.28.1
apache-airflow-providers-cncf-kubernetes==10.8.2
apache-airflow-providers-hashicorp==4.3.2
apache-airflow-providers-standard==1.9.0
apache-airflow-providers-postgres==6.3.0
apache-airflow-providers-apache-cassandra==3.8.2
apache-airflow-providers-fab==3.0.0
apache-airflow-providers-amazon==9.15.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
airflow 3.1.0 in k8s deployed with official chart. We're also use pgbouncer
for connection pooling
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
