anmolxlight opened a new pull request, #70227: URL: https://github.com/apache/airflow/pull/70227
closes: #70222 ## Problem Intermittent `ExpiredSignatureError` / `Failed to validate JWT` on `/execution/...` for tasks that are not necessarily older than `[execution_api] jwt_expiration_time`. Root cause is a TOCTOU race in `JWTReissueMiddleware`: 1. `JWTBearer` validates the bearer token (still valid) and caches claims on the ASGI request scope. 2. The request handler runs (e.g. heartbeat). 3. Middleware re-calls `avalidated_claims` on the same bearer string. 4. If the token crossed `exp` during the request, that second validation raises `ExpiredSignatureError`. 5. The exception is swallowed, so no `Refreshed-API-Token` header is returned. 6. The client's next call arrives with a fully expired token → 403, and after enough failed heartbeats the task is killed. This matches the intermittent multi-task failures reported in #70222 and the long-running LocalExecutor case in #67939. ## Fix Reuse the claims already validated and cached by `JWTBearer` (`request.scope["ti_token"]`) instead of re-parsing/re-validating the Authorization header. If remaining lifetime is below the existing refresh threshold, issue a fresh token from those claims. No expiry leeway is added. Workload tokens remain non-refreshable. ## Tests - Updated threshold reissue tests to populate the ASGI scope the way production does. - Added a mid-request expiry regression test (custom route advances the clock past `exp` during the handler) asserting `Refreshed-API-Token` is still issued and `avalidated_claims` is not called by middleware. ## Local verification ```bash uv run --project airflow-core pytest airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py -xvs ruff check airflow-core/src/airflow/api_fastapi/execution_api/app.py airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py ruff format --check airflow-core/src/airflow/api_fastapi/execution_api/app.py airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py ``` All 5 tests passed. ## Notes Related open PR #68499 targets the same race for #67939. This PR is based on current `main`, includes ashb's requested mid-request time-travel regression shape, and closes #70222. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
