arkadiuszbach commented on issue #73006: URL: https://github.com/apache/airflow/issues/73006#issuecomment-6151576724
My merged PR [#69034](https://github.com/apache/airflow/pull/69034) addresses the worker-shutdown path discussed in this issue and [#71202](https://github.com/apache/airflow/issues/71202): when SIGTERM reaches the supervisor, it waits for the running task instead of forwarding SIGTERM and triggering the operator's `on_kill()`. For KPO, this allows the operator to continue monitoring its application pod and report its outcome. I verified this with Airflow 3.3.1 / Task SDK 1.3.1 / Kubernetes provider 10.21.1, applying only #69034's SDK changes and its corresponding worker configuration (`DUMB_INIT_SETSID=0`; baseline: `1`). All 441 installed package versions remained identical. In the unpatched control, deleting only the worker produced a false `SUCCESS` before the application completed. The following tests used KubernetesExecutor and synchronous KPO on local kind. **Worker** means the pod running the supervisor and KPO; **application** means the separate pod created by KPO. The workload was `alpine:3`, running `sh -c 'sleep 600 && echo done'`. We initiated deletion of the selected pod(s) using two shutdown budgets: - **700 seconds: graceful shutdown with enough time to finish.** Kubernetes sends SIGTERM and honours the configured grace period. We deliberately allow enough time for the 600-second application workload to complete and for the worker to report its outcome. The selected pod(s) finish before the deadline, so SIGKILL is not needed. - **30 seconds: insufficient time, followed by forced termination.** Kubernetes sends SIGTERM, then sends SIGKILL to containers still running when the 30-second grace period expires. This simulates the effect of a short shutdown window, such as Spot reclamation or another event that terminates a node before its workloads can finish. These are simulation parameters. The 30-second deadline is not a universal Spot or EKS managed-node-group limit; actual shutdown windows depend on the provider, configuration, and event, as described below. In both groups, Kubernetes waits within the applicable grace period; the difference is whether the workload can finish before that period expires. | Pod(s) selected for termination | Grace | Worker pod: phase / exit | Application pod: phase / exit | Airflow task state | |---|---:|---|---|---| | Worker only | 700 s | Succeeded / 0 | Succeeded / 0 | SUCCESS, after `done` | | Application only | 700 s | Succeeded / 0 | Succeeded / 0 | SUCCESS, after `done` | | Both | 700 s | Succeeded / 0 | Succeeded / 0 | SUCCESS, after `done` | | Worker only | 30 s | Failed / 137 | Still Running | UP_FOR_RETRY | | Application only | 30 s | Succeeded / 0 | Failed / 137 | UP_FOR_RETRY | | Both | 30 s | Failed / 137 | Failed / 137 | UP_FOR_RETRY | No false `SUCCESS` occurred in this patched matrix. The Dag allowed one retry; the table records first-attempt results. When the worker was killed, KubernetesExecutor observed the failed worker pod and the scheduler handled the failure; this did not depend on the dead KPO detecting anything or waiting for a heartbeat timeout. When only the application was killed, KPO detected its failure. A worker pod exiting 0 therefore does not necessarily mean the Airflow task succeeded. The application continued after SIGTERM in this particular image/command. That is not guaranteed for arbitrary applications: two additional tests used an application that handled SIGTERM and exited 143, and the task entered `UP_FOR_RETRY` before the 30-second deadline. #69034 does not add signal handling to code inside the application pod. An application that exits 0 after abandoning unfinished work can still appear successful. I also ran both worker-killed scenarios through an actual retry, with a 10-second retry delay and the default `durable=True` / reattachment behavior. After worker-only SIGKILL, attempt 2 reconnected to the original application pod: the same pod UID, the same container ID, and `restartCount=0`. No second application pod was created. After both pods were killed, the original application container finished with exit 137 before the replacement application started. Both retries completed with `SUCCESS`. Continuous Kubernetes watches recorded at most one running application container per test throughout both attempts. There are also limits to the fix: - The `kill -TERM <task-runner PID>` reproduction in this issue sends the signal directly to the task subprocess, bypassing the supervisor. To verify worker shutdown with #69034, initiate termination of the worker pod so the signal reaches the supervisor through the configured entrypoint. - SIGKILL or loss of the VM can still interrupt the worker and leave its application running without an active worker. The successful reattachment verified above depends on the tested configuration and on the existing pod being discoverable. - Mixed placement, such as a Spot worker and an on-demand application pod, can produce the asymmetric cases above. Each pod has its own shutdown budget; these six cases are not an exhaustive test of cloud events or signal ordering. `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` can prevent Cluster Autoscaler scale-down of a node hosting the pod; it does not prevent Spot reclamation or every managed-node maintenance event. Setting it to `"true"` allows Cluster Autoscaler to consider eviction but does not select a shutdown timeout. Our long/short tests used `false`/`true`, respectively, but explicit pod deletion bypasses that autoscaler decision. These were not cloud Spot or autoscaler tests. See the [Cluster Autoscaler FAQ](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node). The available time depends on the provider, node configuration, Kubernetes version, and event: - **AWS/EKS:** EC2 Spot stop/termination has a best-effort two-minute interruption notice, not two guaranteed minutes for each pod. EKS warns that pods may receive less time or no termination signal. Managed-node-group desired-capacity reductions and `AZRebalance` have a separate 15-minute eviction limit before node termination; an ASG lifecycle hook can extend eligible terminations, but cannot extend Spot reclamation. See [Spot notices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html), [managed node groups](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html), and [ASG termination limits](https://docs.aws.amazon.com/autoscaling/ec2/userguide/gracefully-handle-instance-termination.html). For managed-node-group version/AMI updates, the 15-minute drain timeout instead fails an unforced update with `PodEvictionFailure`; it is not a universal hard termination deadline. See [EKS update behavior](https://docs.aws.amazon.co m/eks/latest/userguide/managed-node-update-behavior.html). - **GKE:** Default Spot shutdown is 30 seconds total, split into 15 seconds for ordinary pods and 15 for critical system pods. Current documentation allows up to 120 seconds of total node shutdown time through node configuration on supported versions (Standard control plane >= `1.35.0-gke.1171000`); ordinary pods receive the total node budget minus the critical-pod allocation. Separately, autoscaler scale-down allows one hour from `1.32.7-gke.1079000`, versus 10 minutes on older versions. See [GKE Spot shutdown](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/spot-vms#termination_and_graceful_shutdown_of_spot_vms) and [GKE autoscaling](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#how_cluster_autoscaler_works). - **Azure/AKS:** Spot eviction notifications are best effort, up to 30 seconds; AKS also attempts automatic cordon/drain on `Preempt`. Autoscaler scale-down separately defaults to 600 seconds through the configurable `max-graceful-termination-sec` profile setting. See [Azure Spot eviction](https://learn.microsoft.com/en-us/azure/virtual-machines/spot-vms#eviction-policy), [AKS auto-drain](https://learn.microsoft.com/en-us/azure/aks/node-auto-drain), and [AKS autoscaling](https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler). Rolling upgrades have a different drain timeout: 30 minutes by default, configurable from 5 minutes to 24 hours; expiry with pods remaining stops the upgrade. See [AKS upgrade draining](https://learn.microsoft.com/en-us/azure/aks/upgrade-aks-node-pools-rolling#customize-node-drain-timeout). During normal graceful pod termination, Kubernetes honours the configured `terminationGracePeriodSeconds`. Configure it separately for the Airflow worker and KPO application pod, allowing enough time for the remaining work and result reporting. The default is 30 seconds, and `preStop` consumes part of this budget. See [Kubernetes pod termination](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination-flow). With #69034, the worker can continue monitoring the running task during this period. If the underlying node is terminated earlier, the available time is constrained by that event, such as the Spot and managed-node-group termination limits described above. Could you test by applying #69034's Task SDK changes to your existing test deployment and setting `DUMB_INIT_SETSID=0` in the actual KubernetesExecutor worker pod template, then restarting the affected components? You do not need to build from main or upgrade all packages to test the backport. Please include the node-removal event, executor/provider versions, and both pod grace periods when reporting the result. I am also considering adding integration tests to Airflow's test suite to cover these six termination scenarios. --- Drafted-by: Codex (GPT-6); reviewed by @arkadiuszbach before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
