kaxil commented on code in PR #77197:
URL: https://github.com/apache/airflow/pull/77197#discussion_r4604273857


##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py:
##########
@@ -932,6 +941,13 @@ def execute_sync(self, context: Context):
             self.remote_pod = self.pod_manager.await_pod_completion(
                 self.pod, istio_enabled, self.base_container_name, 
self.do_xcom_push
             )
+        except AirflowTaskTimeout:
+            # The task runner calls on_kill only after this propagates, so 
cleanup has to know now
+            # that the pod must be left running for the next try to reattach.
+            self._keep_pod_after_timeout = (
+                self.reattach_on_restart and self.on_kill_action == 
OnKillAction.KEEP_POD

Review Comment:
   This keeps the pod even when no retry will follow (`retries=0`, or the last 
retry), and then the log line's "so the next try can reattach" is wrong. Since 
`reattach_on_restart` is on by default, an existing Dag that set 
`on_kill_action="keep_pod"` for debugging (as the "Pod cleanup on kill" docs 
suggest) plus an `execution_timeout` now leaves a pod running after every 
timeout, often a hung one that the timeout was there to stop. Would adding 
`context["ti"].try_number <= context["ti"].max_tries` here work? Then the docs 
sentence about cleaning up the last pod by hand can go too.



##########
providers/cncf/kubernetes/docs/operators.rst:
##########
@@ -243,6 +243,11 @@ persisted, the operator falls back to the same label 
search ``reattach_on_restar
 used, so a running pod from a prior attempt is still found and reattached to 
rather than
 duplicated. Once an identity is persisted, subsequent retries skip the label 
search entirely.
 
+By default, a try interrupted by ``execution_timeout`` deletes the pod (per 
``on_finish_action``), so the retry
+starts a fresh pod. To let the retry reattach instead, set 
``on_kill_action="keep_pod"``: the pod is left running
+and is not marked as already checked. With retries exhausted, the last pod is 
left running and must be cleaned
+up manually. This applies to sync mode with ``durable=True`` only.

Review Comment:
   Below 3.3 this keys off `reattach_on_restart` (default True) and `durable` 
is ignored, so "`durable=True` only" reads wrong for anyone on an older 
Airflow. Maybe "with `durable=True` (or `reattach_on_restart=True` below 3.3)"?



##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py:
##########
@@ -1411,6 +1427,15 @@ def cleanup(
         if self._killed or not remote_pod:
             return
 
+        if self._keep_pod_after_timeout:

Review Comment:
   What happens if the kept pod finishes during `retry_delay` (300 s by 
default)? The next try finds it `Succeeded` in `get_or_create_pod`, which 
deletes it under the default `on_finish_action="delete_pod"` and creates a 
fresh pod. With the default `do_xcom_push=False` nothing keeps the pod 
`Running` after the job ends, so a job that needs a bit more than one timeout 
window gets thrown away and rerun from zero on every retry, which is the case 
this change is meant to fix. It also runs the job to completion each time, 
where before it was killed at the timeout. Could `get_or_create_pod` treat a 
`Succeeded` pod as the finished run when `do_xcom_push` is off?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to