chsa21 commented on issue #70222:
URL: https://github.com/apache/airflow/issues/70222#issuecomment-5088017137
Hi, thanks for the analysis.
The issue occurred again on Saturday.
Regarding your questions:
- Executor: LocalExecutor
- Unfortunately, we don't currently log the decoded JWT claims (iat/exp), so
we can't provide those values.
- I've attached an anonymized excerpt from the API server log. It shows the
sequence of:
1. JWT reissue middleware failed to refresh token with
ExpiredSignatureError,
2. the subsequent 403 on the /execution/... endpoint, and
3. Failed to validate JWT for the following request.
This seems consistent with the race condition you described, where the token
expires during request processing and no refreshed token is returned.
Please let me know if there is any other information that would help confirm
the root cause.
Log:
2026-07-24T23:37:03.964536Z [warning ] JWT reissue middleware failed to
refresh token [airflow.api_fastapi.execution_api.app] error='Signature has
expired' loc=app.py:158
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: Traceback (most
recent call last):
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/airflow/api_fastapi/execution_api/app.py",
line 141, in dispatch
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: claims = await
validator.avalidated_claims(token, {})
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/airflow/api_fastapi/auth/tokens.py",
line 331, in avalidated_claims
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: claims =
jwt.decode(
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: unvalidated,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ...<5 lines>...
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
leeway=self.leeway,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: )
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py",
line 368, in decode
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: decoded =
self.decode_complete(
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: jwt,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ...<8 lines>...
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: leeway=leeway,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: )
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py",
line 275, in decode_complete
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
self._validate_claims(
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
~~~~~~~~~~~~~~~~~~~~~^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: payload,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ^^^^^^^^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ...<4 lines>...
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
subject=subject,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
^^^^^^^^^^^^^^^^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: )
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py",
line 408, in _validate_claims
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
self._validate_exp(payload, now, leeway)
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py",
line 511, in _validate_exp
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: raise
ExpiredSignatureError("Signature has expired")
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
jwt.exceptions.ExpiredSignatureError: Signature has expired
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
2026-07-24T23:37:03.965604Z [info ] request finished
[http.access] client_addr=? duration_us=3424 loc=http_access_log.py:98
method=PATCH
path=/execution/task-instances/019f95df-f1c7-757e-985e-cf333fb5cb35/run query=
status_code=403
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
2026-07-24T23:37:03.978324Z [warning ] Failed to validate JWT
[airflow.api_fastapi.execution_api.security]
correlation_id=019f967d-54e8-7c2b-9452-4e5637085cdb loc=security.py:127
token=eyJ***
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: Traceback (most
recent call last):
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/airflow/api_fastapi/execution_api/security.py",
line 125, in __call__
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: claims = await
validator.avalidated_claims(creds.credentials, dict(self.required_claims))
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/airflow/api_fastapi/auth/tokens.py",
line 331, in avalidated_claims
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: claims =
jwt.decode(
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: unvalidated,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ...<5 lines>...
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
leeway=self.leeway,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: )
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py",
line 368, in decode
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: decoded =
self.decode_complete(
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: jwt,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ...<8 lines>...
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: leeway=leeway,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: )
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py",
line 275, in decode_complete
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
self._validate_claims(
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
~~~~~~~~~~~~~~~~~~~~~^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: payload,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ^^^^^^^^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ...<4 lines>...
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
subject=subject,
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
^^^^^^^^^^^^^^^^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: )
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: ^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py",
line 408, in _validate_claims
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
self._validate_exp(payload, now, leeway)
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: File
"path/python_venvs/airflow_3.2.2/lib64/python3.14/site-packages/jwt/api_jwt.py",
line 511, in _validate_exp
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]: raise
ExpiredSignatureError("Signature has expired")
Jul 25 01:37:03 XXXXX start_airflow_component.sh[PID]:
jwt.exceptions.ExpiredSignatureError: Signature has expired
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
