devgonvarun opened a new issue, #57954:
URL: https://github.com/apache/airflow/issues/57954
### Apache Airflow version
3.1.2
### If "Other Airflow 2/3 version" selected, which one?
3.1.1
### What happened?
When upgrading from Airflow 3.1.0 to 3.1.1 on kubernetes, the pods are all
installed and in running state but the Airflow UI shows "Internal Server
Error". On checking the logs of the api server pod I found the following error.
`2025-11-06T08:43:38.802104Z [error ] Exception when querying database
(MessagePack data is malformed: trailing characters (byte 1)).Retrying (1/3) in
0.30s. [airflow.providers.fab.www.app] loc=_utils.py:59
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py",
line 52, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/sqlalchemy/sqlalchemy.py",
line 152, in _retrieve_session_data
return self.serializer.decode(serialized_session_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/session.py",
line 37, in decode
return decoder.decode(data)
^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: MessagePack data is malformed: trailing characters
(byte 1)
2025-11-06T08:43:39.105905Z [error ] Exception when querying database
(MessagePack data is malformed: trailing characters (byte 1)).Retrying (2/3) in
0.60s. [airflow.providers.fab.www.app] loc=_utils.py:59
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py",
line 52, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/sqlalchemy/sqlalchemy.py",
line 152, in _retrieve_session_data
return self.serializer.decode(serialized_session_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/session.py",
line 37, in decode
return decoder.decode(data)
^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: MessagePack data is malformed: trailing characters
(byte 1)
2025-11-06T08:43:39.708232Z [error ] Exception on /login/ [GET]
[airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py",
line 2528, in wsgi_app
ctx.push()
File "/home/airflow/.local/lib/python3.12/site-packages/flask/ctx.py",
line 375, in push
self.session = session_interface.open_session(self.app, self.request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/base.py", line
350, in open_session
saved_session_data = self._retrieve_session_data(store_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py",
line 56, in wrapper
raise e
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py",
line 52, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/sqlalchemy/sqlalchemy.py",
line 152, in _retrieve_session_data
return self.serializer.decode(serialized_session_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/session.py",
line 37, in decode
return decoder.decode(data)
^^^^^^^^^^^^^^^^^^^^
msgspec.DecodeError: MessagePack data is malformed: trailing characters
(byte 1)
INFO: 10.244.0.114:46448 - "GET
/auth/login/?next=https://airflow.tst.batch.bawag.lcl/ HTTP/1.1" 500 Internal
Server Error
ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File
"/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py",
line 409, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py",
line 1082, in __call__
| await super().__call__(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py",
line 113, in __call__
| await self.middleware_stack(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py",
line 186, in __call__
| raise exc
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py",
line 164, in __call__
| await self.app(scope, receive, _send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py",
line 29, in __call__
| await responder(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py",
line 130, in __call__
| await super().__call__(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py",
line 46, in __call__
| await self.app(scope, receive, self.send_with_compression)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py",
line 85, in __call__
| await self.app(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py",
line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive,
send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py",
line 53, in wrapped_app
| raise exc
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py",
line 42, in wrapped_app
| await app(scope, receive, sender)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line
716, in __call__
| await self.middleware_stack(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line
736, in app
| await route.handle(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line
462, in handle
| await self.app(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py",
line 1082, in __call__
| await super().__call__(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py",
line 113, in __call__
| await self.middleware_stack(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py",
line 186, in __call__
| raise exc
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py",
line 164, in __call__
| await self.app(scope, receive, _send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py",
line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive,
send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py",
line 53, in wrapped_app
| raise exc
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py",
line 42, in wrapped_app
| await app(scope, receive, sender)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line
716, in __call__
| await self.middleware_stack(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line
736, in app
| await route.handle(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line
462, in handle
| await self.app(scope, receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py",
line 81, in __call__
| await responder(receive, send)
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py",
line 106, in __call__
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py",
line 781, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2528, in
wsgi_app
| ctx.push()
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask/ctx.py", line 375, in
push
| self.session = session_interface.open_session(self.app,
self.request)
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/base.py", line
350, in open_session
| saved_session_data = self._retrieve_session_data(store_id)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py",
line 56, in wrapper
| raise e
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/_utils.py",
line 52, in wrapper
| return func(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask_session/sqlalchemy/sqlalchemy.py",
line 152, in _retrieve_session_data
| return self.serializer.decode(serialized_session_data)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/session.py",
line 37, in decode
| return decoder.decode(data)
| ^^^^^^^^^^^^^^^^^^^^
| msgspec.DecodeError: MessagePack data is malformed: trailing
characters (byte 1)
|
| During handling of the above exception, another exception occurred:
|
| Traceback (most recent call last):
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py",
line 109, in __call__
| await anyio.to_thread.run_sync(self.wsgi, environ,
self.start_response)
| File
"/home/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line
56, in run_sync
| return await get_async_backend().run_sync_in_worker_thread(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py",
line 2485, in run_sync_in_worker_thread
| return await future
| ^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py",
line 976, in run
| result = context.run(func, *args)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py",
line 147, in wsgi
| for chunk in self.app(environ, start_response):
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2552, in
__call__
| return self.wsgi_app(environ, start_response)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/werkzeug/middleware/proxy_fix.py",
line 187, in __call__
| return self.app(environ, start_response)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2532, in
wsgi_app
| response = self.handle_exception(e)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1727, in
handle_exception
| server_error = self.ensure_sync(handler)(server_error)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/views.py",
line 82, in show_traceback
| return render_template("airflow/traceback.html"), 500
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask/templating.py", line
147, in render_template
| return _render(app, template, context)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask/templating.py", line
128, in _render
| app.update_template_context(context)
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 997, in
update_template_context
| context.update(func())
| ^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line
405, in _user_context_processor
| return dict(current_user=_get_user())
| ^^^^^^^^^^^
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line
370, in _get_user
| current_app.login_manager._load_user()
| File
"/home/airflow/.local/lib/python3.12/site-packages/flask_login/login_manager.py",
line 362, in _load_user
| user_id = session.get("_user_id")
| ^^^^^^^^^^^
| AttributeError: 'NoneType' object has no attribute 'get'
+------------------------------------`
Seems to me a serialisation error in the sessions table. After i cleaned the
session table the error went away and everything worked correctly.
### What you think should happen instead?
The upgrade should work without errors and without the need of resetting the
sessions table.
### How to reproduce
Upgrade from Airflow 3.1.0 to 3.1.1 in kubernetes. I am using Azure AD as
the OAuth provider to create custom airflow roles in my webserver config.
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
