1fanwang opened a new pull request, #68998: URL: https://github.com/apache/airflow/pull/68998
## Why When `KubernetesExecutor` fails to create a worker pod, it re-queues the task on an exceeded-quota / stale-version conflict, `500`, or `429`, but lets **`502` / `503` / `504` fall through and fail the task immediately.** Those are exactly the gateway / unavailable / timeout codes that spike when the API server or an admission webhook is under load. Concretely, the API server returns `503` **with `Retry-After: 1`** on every graceful shutdown (rollout, node drain, control-plane autoscale — see `apiserver` `waitgroup.go`), so a worker pod CREATE that is in flight during any apiserver roll currently kills the task outright instead of retrying. `client-go` retries all 5xx and honours `Retry-After`; the executor does not. ## What Add `502` / `503` / `504` to the transient set, so a task that hits one is re-queued (bounded by `task_publish_max_retries`) instead of failed. When the response carries `Retry-After` (a shutting-down apiserver always sends it on `503`), pause pod creation until then via `create_pods_after`, exactly as the `429` path already does; other transient 5xx without the header retry on the next scheduler loop like `500`. The `Retry-After` parse is guarded so a malformed value can't crash the scheduler loop. This gap was noticed while reviewing #68480 (concurrent pod creation). It is pre-existing on the current sync path and independent of that change, so it goes as its own PR; the two touch the same error handler, so whichever lands second will need a trivial rebase. ## Tests `test_run_next_exception_requeue` gains `502` / `504` (re-queue, retry next loop), `503 + Retry-After` (re-queue, loop backs off), and `503` with retries exhausted (fails). Each fails without this change. ## E2E Live `kind` (real API server, real `KubernetesExecutor.sync()`), with a proxy injecting `503 + Retry-After: 1` on the first CREATE per task — identical config (`task_publish_max_retries=3`), only the fix differs: | Run (`retries=3`) | behaviour | **pods created** | |---|---|---| | before (`main`) | `503` → task failed | **0 / 6** | | after (this PR) | `503` → re-queued, retried | **6 / 6** | <details><summary>Raw — after</summary> ``` loop 0..5: queue 6->6 failed_events=0 # each loop: one task hits 503+Retry-After, re-queues, loop backs off loop 6: queue 6->0 failed_events=0 # backoff elapsed -> retries succeed, all pods created FINAL failed_events=0 queue_remaining=0 # injector: INJECT 503 (Retry-After=1) x6 (first attempt) -> PASS 201 x6 (retry) # cluster: retrydag-t000..t005 Running ``` </details> ## Risk Behaviour change on the default path: a `502` / `503` / `504` now re-queues instead of failing the task — bounded by `task_publish_max_retries` (default `0`, i.e. inert unless retries are already enabled). The existing `429` / `500` / quota / conflict handling is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
