kaxil commented on code in PR #77197:
URL: https://github.com/apache/airflow/pull/77197#discussion_r4604273857
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py:
##########
@@ -932,6 +941,13 @@ def execute_sync(self, context: Context):
self.remote_pod = self.pod_manager.await_pod_completion(
self.pod, istio_enabled, self.base_container_name,
self.do_xcom_push
)
+ except AirflowTaskTimeout:
+ # The task runner calls on_kill only after this propagates, so
cleanup has to know now
+ # that the pod must be left running for the next try to reattach.
+ self._keep_pod_after_timeout = (
+ self.reattach_on_restart and self.on_kill_action ==
OnKillAction.KEEP_POD
Review Comment:
This keeps the pod even when no retry will follow (`retries=0`, or the last
retry), and then the log line's "so the next try can reattach" is wrong. Since
`reattach_on_restart` is on by default, an existing Dag that set
`on_kill_action="keep_pod"` for debugging (as the "Pod cleanup on kill" docs
suggest) plus an `execution_timeout` now leaves a pod running after every
timeout, often a hung one that the timeout was there to stop. Would adding
`context["ti"].try_number <= context["ti"].max_tries` here work? Then the docs
sentence about cleaning up the last pod by hand can go too.
##########
providers/cncf/kubernetes/docs/operators.rst:
##########
@@ -243,6 +243,11 @@ persisted, the operator falls back to the same label
search ``reattach_on_restar
used, so a running pod from a prior attempt is still found and reattached to
rather than
duplicated. Once an identity is persisted, subsequent retries skip the label
search entirely.
+By default, a try interrupted by ``execution_timeout`` deletes the pod (per
``on_finish_action``), so the retry
+starts a fresh pod. To let the retry reattach instead, set
``on_kill_action="keep_pod"``: the pod is left running
+and is not marked as already checked. With retries exhausted, the last pod is
left running and must be cleaned
+up manually. This applies to sync mode with ``durable=True`` only.
Review Comment:
Below 3.3 this keys off `reattach_on_restart` (default True) and `durable`
is ignored, so "`durable=True` only" reads wrong for anyone on an older
Airflow. Maybe "with `durable=True` (or `reattach_on_restart=True` below 3.3)"?
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py:
##########
@@ -1411,6 +1427,15 @@ def cleanup(
if self._killed or not remote_pod:
return
+ if self._keep_pod_after_timeout:
Review Comment:
What happens if the kept pod finishes during `retry_delay` (300 s by
default)? The next try finds it `Succeeded` in `get_or_create_pod`, which
deletes it under the default `on_finish_action="delete_pod"` and creates a
fresh pod. With the default `do_xcom_push=False` nothing keeps the pod
`Running` after the job ends, so a job that needs a bit more than one timeout
window gets thrown away and rerun from zero on every retry, which is the case
this change is meant to fix. It also runs the job to completion each time,
where before it was killed at the timeout. Could `get_or_create_pod` treat a
`Succeeded` pod as the finished run when `do_xcom_push` is off?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]