amoghrajesh opened a new pull request, #77197:
URL: https://github.com/apache/airflow/pull/77197

    <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   <!--
   Thank you for contributing!
   
   Please provide above a brief description of the changes made in this pull 
request.
   Write a good git commit message following this guide: 
https://chris.beams.io/posts/git-commit/
   
   Please make sure that your code changes are covered with tests.
   And in case of new features or big changes remember to adjust the 
documentation.
   
   For user-facing UI changes, please attach before/after screenshots (or a 
short
   screen recording) so reviewers can assess the visual impact.
   
   Feel free to ping (in general) for the review if you do not see reaction for 
a few days
   (72 Hours is the minimum reaction time you can expect from volunteers) - we 
sometimes miss notifications.
   
   In case of an existing issue, reference it using one of the following:
   
   * closes: #ISSUE
   * related: #ISSUE
   -->
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, 
please
   change below checkbox to `[X]` followed by the name of the tool, uncomment 
the "Generated-by".
   -->
   
   - [ ] Yes (please specify the tool below)
   
   <!--
   Generated-by: [Tool Name] following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   -->
   
   ### What
   
   A durable `KubernetesPodOperator` is meant to let a retry pick up a pod that 
is still running instead of starting the work again. That does not work when 
the try ends because of `execution_timeout`, even with 
`on_kill_action="keep_pod"`. Every retry starts a new pod from zero, so a long 
job whose timeout is slightly too low can never finish.
   
   ### Current behaviour
   
   When `execution_timeout` fires, the timeout exception leaves `execute_sync`, 
and its `finally` block runs `cleanup` before the task runner calls `on_kill`. 
At that point `_killed` is still False, so `cleanup`:
   
   1. labels the pod as already checked, and
   2. deletes it per `on_finish_action` (default `delete_pod`).
   
   `on_kill_action` never gets a say. Even with `on_finish_action="keep_pod"`, 
the "already checked" label makes the next try reject the pod:
   
   ```
   Pod airflow/kpo-keep-pod-work-kehfcc39 from task state store was already 
checked by a prior attempt, creating a fresh pod instead of reattaching.
   ```
   
   ### Proposed change
   
   When `AirflowTaskTimeout` interrupts `execute_sync`, the operator is durable 
(`reattach_on_restart`), and `on_kill_action="keep_pod"`, `cleanup` now leaves 
the pod running and unlabelled. The next try then reattaches to it through the 
task state store, gets a fresh `execution_timeout` window, and waits for the 
remaining work. Every other path (success, pod failure, other exceptions, 
default `on_kill_action`) is unchanged.
   
   The `on_kill_action` docstring and the "Durable execution" section in 
`operators.rst` describe the new behaviour and its limits.
   
   ### Changes of Note
   
   - The decision is made in an `except AirflowTaskTimeout` in `execute_sync`, 
because the task runner only calls `on_kill` after the exception propagates. By 
then `cleanup` has already run.
   - This is opt-in through `on_kill_action="keep_pod"`. Keeping pods past a 
timeout by default would leave running pods behind whenever retries run out.
   - `AirflowTaskTimeout` is imported from 
`airflow.providers.common.compat.sdk`, so the provider's Airflow floor is 
unchanged.
   
   ### User implications / backcompat
   
   - Default behaviour is unchanged. A timeout still deletes the pod unless 
`on_kill_action="keep_pod"` is set.
   - Users who already set `on_kill_action="keep_pod"` with durable/reattach on 
(the default) will now see the pod survive a timeout, not just a manual kill. 
That includes Airflow `<` 3.3, where `reattach_on_restart` defaults to True and 
the retry reattaches through the label search.
   - If the last retry times out, its pod is left running and has to be cleaned 
up manually. This is documented.
   - For a timed-out try that keeps its pod, `on_finish_action`, 
`on_pod_teardown` and the failure XCom push and event logging are skipped. 
`on_pod_cleanup` callbacks still run.
   
   ### Testing
   
   End to end on a kind cluster (`breeze k8s`, LocalExecutor). The task is a 
pod that needs 240 s of work, with `execution_timeout=90s`, `retries=3` and 
`durable=True`. The same Dag ran once with `on_kill_action="keep_pod"` and once 
with `on_kill_action="delete_pod"` as a control.
   
   #### Before
   
   Both variants behaved the same:
   
   | | `keep_pod` | `delete_pod` |
   |---|---|---|
   | Pods created | 4 (new pod every try) | 4 |
   | Result | failed after 4 tries | failed after 4 tries |
   
   All 4 tries failed, each on a new pod:
   
   <img width="3200" height="2000" alt="01_before_task_tries" 
src="https://github.com/user-attachments/assets/fd3cf502-e0ba-4c36-a720-549966fe8efa";
 />
   
   
   Try 2 found the kept pod but rejected it as "already checked":
   
   <img width="3200" height="2000" alt="02_before_try2_already_checked" 
src="https://github.com/user-attachments/assets/4fe1747b-e0b6-4c51-905c-0d65e78e761a";
 />
   
   
   #### After
   
   | | `keep_pod` | `delete_pod` |
   |---|---|---|
   | Pods created | **1** | 4 |
   | Tries 2 and 3 | `Found matching pod kpo-keep-pod-work-4s7uxflh`, 
reattached | new pod |
   | Result | **success on try 3**, 4 s after the pod finished its 240 s of 
work | failed after 4 tries |
   
   Try 1 timed out and kept the pod for the next try:
   
   <img width="3200" height="2000" alt="04_after_try1_keeping_pod" 
src="https://github.com/user-attachments/assets/71e942d5-83c5-48c2-9b36-79162d7f2404";
 />
   
   
   Try 2 reattached to the same pod, timed out again, and kept it:
   
   <img width="3200" height="2000" alt="05_after_try2_found_matching_pod" 
src="https://github.com/user-attachments/assets/1aae8e9e-abef-45ad-be8d-6654109da97a";
 />
   
   
   Try 3 reattached and succeeded once the pod finished (`started at 09:25:42`, 
`finished at 09:29:42`):
   
   <img width="3200" height="2000" alt="03_after_task_tries" 
src="https://github.com/user-attachments/assets/cb0aa24e-91ef-4ce1-84ef-19aa8f93cdbd";
 />
   
   
   A third run, watched with `kubectl get pods`. One `keep_pod` pod (yellow) 
stays alive across all tries, while `delete_pod` (blue) gets a new pod every 
try:
   
   <img width="2172" height="1420" alt="06_pod_list" 
src="https://github.com/user-attachments/assets/588f7d02-b463-47b1-ab03-932cb57c7320";
 />
   
   ---
   
   * Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information. Note: commit author/co-author name and email in commits 
become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: 
`{pr_number}.significant.rst`, in 
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
 You can add this file in a follow-up commit after the PR is created so you 
know the PR number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to