seanghaeli opened a new pull request, #76332: URL: https://github.com/apache/airflow/pull/76332
AwsEksExecutor signs every EKS token with the AWS credentials it read when it built its Kubernetes client. When those are static keys that something outside the process keeps rotating, such as a credentials file rewritten by a sidecar, every token signed after the first keys expire is rejected. The 401 isn't handled, so `_adopt_completed_pods` raises and the scheduler exits. The ECS, Batch and Lambda executors already survive this setup, because they rebuild their client when credentials expire. - Each new EKS token is signed with credentials from a new session of the AWS connection, so rotated keys are picked up. Refreshable sources (IRSA, Pod Identity, instance profile, assume-role connections) work as before; they are just resolved again for each token, every 14 minutes. - When the cluster answers 401, the client drops the token and retries the request once with a new one. That covers a token signed just before its keys expired, which would otherwise be reused for up to 14 minutes. A 401 means the request was rejected before anything was done, so retrying a create is safe. Both the sync client and the async one (`async_pod_creation`) do this. How to review: `_EksApiClient.call_api` and `refresh_api_key` in `executors/eks/_client_factory.py`, then the two new tests. A real EKS cluster and a Breeze scheduler with AwsEksExecutor. Credentials come from a file rewritten every 5 minutes with a new 15-minute STS session, and one task pod sleeps for 21 minutes. On main, the same run dies 18 seconds after the scheduler's first key expires, with `UnauthorizedError: 401` in `_adopt_completed_pods`, while a fresh key is already on disk. ```text # after this PR --- credentials file, rewritten every 5 min with a 15-min STS session 22:30:21 wrote key ...GPMB, expires 22:45:21 22:35:22 wrote key ...LJB4, expires 22:50:22 22:40:23 wrote key ...QKQQ, expires 22:55:23 22:45:24 wrote key ...SNYM, expires 23:00:24 22:50:25 wrote key ...XMJU, expires 23:05:25 22:55:26 wrote key ...GRLT, expires 23:10:26 23:00:27 wrote key ...XCNQ, expires 23:15:26 23:05:28 wrote key ...FYAD, expires 23:20:27 23:10:28 wrote key ...EVTI, expires 23:25:28 23:15:29 wrote key ...WMHP, expires 23:30:29 --- scheduler (AwsEksExecutor), started 22:31:05 with key ...GPMB 22:31:24 Event: eks-credential-rotation-outlive-the-first-session-poln20oy is Running 22:52:25 Event: eks-credential-rotation-outlive-the-first-session-poln20oy Succeeded 0 x 401, 0 watcher restarts, 0 scheduler crashes [45:20] done; scheduler alive at end: yes --- CloudTrail GetCallerIdentity (EKS token checks) by signing key ...GPMB 4 checks 22:31:07 -> 22:39:09 ...QKQQ 6 checks 22:45:09 -> 22:53:59 ...GRLT 6 checks 22:59:12 -> 23:08:28 ...EVTI 3 checks 23:13:15 -> 23:14:40 ``` --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [X] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
