dongjoon-hyun opened a new pull request, #981: URL: https://github.com/apache/spark-kubernetes-operator/pull/981
### What changes were proposed in this pull request? This PR is a follow-up of SPARK-60463 (https://github.com/apache/spark-kubernetes-operator/pull/980). It deletes the `HorizontalPodAutoscaler` (HPA) and `PodDisruptionBudget` (PDB) of a `SparkCluster` that failed with `SchedulingFailure` only after its Kueue `Workload` is released, like its `Service`s and `NetworkPolicy`. - `ClusterSuspendStep`: Move the deletion of the worker HPA and PDB out of `deleteMasterAndWorkers` into a new package-private helper, `deleteWorkerAutoscalerAndDisruptionBudget`. `deleteMasterAndWorkers` now only deletes the master and worker `StatefulSet`s and lists the master and worker pods. The suspend path (`releaseResources`) calls both helpers before it waits for the pods, so a suspended cluster still deletes its HPA and PDB before it releases its `Workload`. - `ClusterCleanUpStep`: Delete the HPA and PDB after `KueueWorkloadUtils.deleteWorkloadOf`, together with the `Service`s and `NetworkPolicy`. - Update the Javadoc and `docs/spark_custom_resources.md` accordingly. ### Why are the changes needed? The HPA and PDB hold no Kueue quota. However, `ClusterCleanUpStep` deleted them before it listed the pods and released the `Workload`. If deleting either of them keeps failing, the release never reaches `waitForPods` and `deleteWorkloadOf`, and the `Workload` keeps its quota until the `SparkCluster` is deleted. For example, this happens to an operator whose custom role cannot access `poddisruptionbudgets`. Such an operator also fails to apply the PDB in `ClusterInitStep` with `403`, which is how the cluster ends up in `SchedulingFailure` in the first place. The review of SPARK-60463 pointed out the same issue for the `Service`s and `NetworkPolicy`, and SPARK-60463 fixed it for them (https://github.com/apache/spark-kubernetes-operator/pull/980#discussion_r4349288708). The HPA and PDB were left for this follow-up because the suspend path shares their deletion. ### Does this PR introduce _any_ user-facing change? No for the released versions, because `ClusterCleanUpStep` is new in the unreleased 1.1.0. Compared to the `main` branch, a `SparkCluster` in `SchedulingFailure` now releases its Kueue `Workload` even when deleting its HPA or PDB fails. The failure is still reported with the `ClusterReleaseFailed` event, and the cluster stays in `SchedulingFailure` until the deletion succeeds. The behavior of the suspend path is not changed. ### How was this patch tested? Pass the CIs with the updated test case. `ClusterCleanUpStepTest.failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload` is now parameterized: the client throws `403` from `network()`, `autoscaling()` or `policy()`. Without the fix, the new cases fail because the `Workload` is not released. ``` ClusterCleanUpStepTest > failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(String) > [2] "autoscaling" FAILED org.opentest4j.AssertionFailedError at ClusterCleanUpStepTest.java:189 ClusterCleanUpStepTest > failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(String) > [3] "policy" FAILED org.opentest4j.AssertionFailedError at ClusterCleanUpStepTest.java:189 ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
