seanmuth opened a new pull request, #69058: URL: https://github.com/apache/airflow/pull/69058
When a worker pod is destroyed before the task process starts — a node drain, autoscaler scale-down, node boot race, or transient image pull failure — the task instance is still in `queued` state and no task code has run. Today the KubernetesExecutor reports this to the scheduler as a normal `FAILED`, which consumes a user-configured task retry and raises a misleading failure alert for work that never executed. This adds a transparent, executor-level requeue for that case. In `_change_state`, a pod that reports `FAILED` while its task instance is still `QUEUED` is requeued onto the existing `task_queue` (the same mechanism `task_publish_max_retries` already uses for pod *creation* failures) without writing to the event buffer, so the scheduler never observes the failure and no task-level retry is consumed. Behavior is bounded and configurable: - `pod_launch_failure_retries` (default `1`, `-1` unlimited, `0` disables) — how many times a task is transparently requeued before failing normally. - `pod_launch_failure_excluded_container_reasons` (default `Error`) — container reasons that opt out of the requeue path and consume a normal retry instead. The default excludes `Error`, which covers a container that started executing but whose worker process exited before writing `running` to the DB (most likely an Airflow-specific startup error rather than a transient infrastructure event). The `ti_state == QUEUED` check is the authoritative signal: a task that was actually executing would already have transitioned to `running`, so OOM-kills and other mid-execution failures are unaffected. Deferrable-operator resume pods are covered for free — when the triggerer fires, the TI returns to `queued`, so a resume pod killed before `execute_complete` starts is requeued rather than discarding already-completed external work. closes: #69052 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.8) Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
