Sanjays2402 opened a new pull request, #73709: URL: https://github.com/apache/airflow/pull/73709
Hi! This addresses #72803, where repeated Kubernetes watcher events for one TaskInstanceKey each fired their own delete_pod, so a single finished worker pod could be deleted two or three times (the extras just 404 against the API server; the issue reports a 67% 404 rate on deletes). The root cause is ordering: `_change_state` called delete_pod (or patched the pod done) before the `self.running` dedup check ran, so the check could never short-circuit the repeat. This moves the running-set removal ahead of the pod API calls, so the first completion for a key wins and repeats are dropped before another delete is issued. Two related paths get the same treatment: - Repeated pre-execution Failed events for an already-requeued pod are now dropped before any pod API call (previously each duplicate fired delete_pod on its way to the requeue dedup). - Completed pods adopted from a dead scheduler are never tracked in `self.running`; they keep their intended one-time cleanup and still report nothing to the scheduler. I also verified the neighboring behavior is preserved: adopted running pods are added to `self.running` by `adopt_launched_task`, so their completions still flow through the normal path, and the pre-execution requeue still re-adds the key to `self.running` so the requeued attempt stays tracked. Note: #72826 proposes an alternative fix for the same issue using a UID-keyed deletion history. This PR takes the approach the issue itself suggests (reorder the existing dedup before the pod calls), with no new executor state. Fixes #72803. Reproduction: a focused test feeds the same successful completion result through `_change_state` three times. Before the fix, delete_pod was called three times (three "Deleted pod associated with the TI" log lines); after the fix, exactly once. Tests (via `uv run --project providers/cncf/kubernetes pytest`): - 2 new regression tests: `test_change_state_ignores_duplicate_completion_event` and `test_change_state_pre_execution_failure_duplicate_does_not_redelete_pod` - All 18 `_change_state` tests and both `test_sync_processes_completed_pods_once*` tests pass (20 passed) - `ruff format` and `ruff check` clean on both changed files --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes - Muse Spark (Meta) Generated-by: Muse Spark (Meta) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
