noamst-monday opened a new pull request, #70595:
URL: https://github.com/apache/airflow/pull/70595
## Problem
`KubernetesPodOperator` in synchronous (`deferrable=False`) mode polls pod
status every ~2 seconds inside `await_pod_completion`. When a pod transitions
to `Succeeded`, there is up to a 2-second window before KPO observes the
terminal phase. The `airflow kubernetes cleanup-pods` command currently deletes
Succeeded/Failed/Evicted pods **immediately** — there is no minimum-age guard
for terminal states.
If the cleanup job fires during that window, KPO's next `read_pod` call
returns 404 and the task is marked **FAILED**, even though the pod completed
successfully (exit code 0).
A `--min-pending-minutes` guard already exists for Pending pods (default 30
m, minimum 5 m). No equivalent exists for terminal states.
### Real-world reproduction
This was confirmed via Kubernetes API server audit logs on a production EKS
cluster running `apache-airflow-providers-cncf-kubernetes==10.19.0` on Airflow
3.2.2. The cleanup CronJob was set to run every 5 minutes.
Timeline of a representative failure:
```
13:15:12Z KPO polls pod → phase Running
13:15:14Z Container exits → exit code 0 (pod phase → Succeeded)
13:15:17Z cleanup-pods deletes pod → 3 s after completion
13:15:18Z KPO polls pod → 404 Not Found → task FAILED
```
The pod had succeeded; the task failure was a false positive. Multiple DAGs
exhibited the same pattern on the same day.
Airflow task log excerpt:
```
[INFO] Pod run-o1sxc2on has phase Running
[ERROR] Task failed with exception
ApiException: (404) Reason: Not Found
HTTP response body: {"message":"pods \"run-o1sxc2on\" not
found","reason":"NotFound","code":404}
```
Audit log at deletion time confirmed `phase: Succeeded`,
`containerStatuses[0].state.terminated.exitCode: 0`, `finishedAt: 13:15:14Z`.
**Reducing the CronJob frequency is a partial mitigation** (lowers
probability) but does not eliminate the race — the window is a function of the
poll interval, not the CronJob frequency.
### Related
PR #69269 fixed a similar 404 race in `is_istio_enabled` for the deferrable
`trigger_reentry` path (merged into 10.20.0). This PR addresses the
complementary gap: preventing the race at source by not deleting
recently-completed pods.
## Solution
Add `--min-completed-minutes` (default `0`, preserving existing behaviour)
to the `cleanup-pods` command. When set to any positive value,
Succeeded/Failed/Evicted pods are skipped unless their completion time exceeds
the threshold.
Completion time is derived from
`max(containerStatuses[*].state.terminated.finishedAt)` — the latest container
finish time across all containers — which is the most accurate signal for "when
the pod entered a terminal state". Falls back to `metadata.creationTimestamp`
for pods evicted before any container started.
Setting `--min-completed-minutes=5` gives KPO a 5-minute observation window
— 150× wider than the 2 s poll interval — closing the race completely in
practice.
## Changes
| File | Change |
|---|---|
| `cli/definition.py` | Add `ARG_MIN_COMPLETED_MINUTES` (default `0`,
`allow_zero=True`); wire into cleanup-pods args |
| `cli/kubernetes_command.py` | Add `_get_pod_completion_time()` helper;
gate terminal-pod deletion by age |
| `tests/.../test_kubernetes_command.py` | 4 new unit tests (too-young
Succeeded, old-enough Succeeded, default=0 backward compat, too-young
Failed/Never) |
| `docs/changelog.rst` | Entry under 10.21.0 |
CLI reference docs (`cli-ref.rst`) use `.. argparse::` and pick up the new
flag automatically.
## Testing
```bash
pytest
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/cli/test_kubernetes_command.py::TestCleanUpPodsCommand
-v
```
All existing tests continue to pass (default=0 preserves current behaviour).
Four new tests cover the new flag.
## Usage
In the cleanup CronJob, add the flag:
```bash
airflow kubernetes cleanup-pods --namespace airflow --min-completed-minutes 5
```
Or in the Airflow Helm chart values:
```yaml
cleanup:
args: ["bash", "-c", "exec airflow kubernetes cleanup-pods --namespace {{
.Release.Namespace }} --min-completed-minutes 5"]
```
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Sonnet 4.6)
Generated-by: Claude Sonnet 4.6 following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
