This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git


The following commit(s) were added to refs/heads/main by this push:
     new 49971079 [SPARK-60475] Delete the HPA and PDB of a `SparkCluster` 
after releasing its Kueue `Workload`
49971079 is described below

commit 4997107921b05bc0588e0b35c8ff85215142d199
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 10:25:50 2026 -0700

    [SPARK-60475] Delete the HPA and PDB of a `SparkCluster` after releasing 
its Kueue `Workload`
    
    ### What changes were proposed in this pull request?
    
    This PR is a follow-up of SPARK-60463 
(https://github.com/apache/spark-kubernetes-operator/pull/980). It deletes the 
`HorizontalPodAutoscaler` (HPA) and `PodDisruptionBudget` (PDB) of a 
`SparkCluster` only after its Kueue `Workload` is released. This applies both 
when the cluster failed with `SchedulingFailure` and when it is suspended by 
`spec.suspend` or by the eviction of its `Workload`.
    
    - `ClusterSuspendStep`: Move the deletion of the worker HPA and PDB out of 
`deleteMasterAndWorkers` into a new package-private helper, 
`deleteWorkerAutoscalerAndDisruptionBudget`. `deleteMasterAndWorkers` now only 
deletes the master and worker `StatefulSet`s and lists the master and worker 
pods. `releaseResources` calls the new helper after 
`KueueWorkloadUtils.deleteWorkloadOf`. A suspended cluster still waits for the 
deletion before it is resumed or queued again.
    - `ClusterCleanUpStep`: Call the new helper after 
`KueueWorkloadUtils.deleteWorkloadOf`, together with the deletion of the 
`Service`s and `NetworkPolicy`.
    - Update the Javadoc and `docs/spark_custom_resources.md` accordingly.
    
    ### Why are the changes needed?
    
    The HPA and PDB hold no Kueue quota. However, `ClusterCleanUpStep` and 
`ClusterSuspendStep` deleted them before they released the `Workload`. If 
deleting either of them keeps failing, the release never reaches 
`deleteWorkloadOf`, and the `Workload` keeps its quota. A failed cluster keeps 
it until the `SparkCluster` is deleted. A suspended cluster keeps it while it 
is suspended, and an evicted cluster keeps the quota which its preemptor waits 
for.
    
    This happens to an operator which cannot access `horizontalpodautoscalers` 
or `poddisruptionbudgets`. Its clusters can still run, because 
`ClusterInitStep` applies the HPA and PDB only when the spec asks for them. For 
example, a queued cluster never has an HPA, since 
`KueueWorkloadFactory.buildWorkload` rejects `minWorkers < maxWorkers`, and a 
cluster with `minWorkers: 0` has no PDB. The release deletes them by name, so 
it gets `403` anyway. Such an operator also fails with `403` when [...]
    
    The review of SPARK-60463 pointed out the same issue for the `Service`s and 
`NetworkPolicy`, and SPARK-60463 fixed it for them 
(https://github.com/apache/spark-kubernetes-operator/pull/980#discussion_r4349288708).
    
    ### Does this PR introduce _any_ user-facing change?
    
    No for the released versions, because `ClusterCleanUpStep` and the 
`spec.suspend` of `SparkCluster` are new in the unreleased 1.1.0. Compared to 
the `main` branch, a `SparkCluster` in `SchedulingFailure` or `Suspended` now 
releases its Kueue `Workload` even when deleting its HPA or PDB fails. The 
failure is still reported with the `ClusterReleaseFailed` or 
`SuspendReleaseFailed` event. The cluster stays in `SchedulingFailure` or 
`Suspended` until the deletion succeeds.
    
    ### How was this patch tested?
    
    Pass the CIs with the updated test cases. In each case, the client throws 
`403` from `autoscaling()` or `policy()`.
    
    - 
`ClusterCleanUpStepTest.failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload`
 is parameterized for `network()`, `autoscaling()` and `policy()`.
    - 
`ClusterSuspendStepTest.failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload`
 is new. It covers a cluster suspended by `spec.suspend` and one suspended by 
an eviction.
    
    Without the fix, the new cases fail because the `Workload` is not released.
    
    ```
    ClusterCleanUpStepTest > 
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(String) > [2] 
"autoscaling" FAILED
    ClusterCleanUpStepTest > 
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(String) > [3] 
"policy" FAILED
    ClusterSuspendStepTest > 
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(boolean, String) > 
[1] "false", "autoscaling" FAILED
    ClusterSuspendStepTest > 
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(boolean, String) > 
[2] "false", "policy" FAILED
    ClusterSuspendStepTest > 
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(boolean, String) > 
[3] "true", "autoscaling" FAILED
    ClusterSuspendStepTest > 
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(boolean, String) > 
[4] "true", "policy" FAILED
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #981 from dongjoon-hyun/SPARK-60475.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/spark_custom_resources.md                     | 10 ++--
 .../reconcilesteps/ClusterCleanUpStep.java         | 14 +++---
 .../reconcilesteps/ClusterSuspendStep.java         | 54 +++++++++++++++-------
 .../reconcilesteps/ClusterCleanUpStepTest.java     | 20 +++++---
 .../reconcilesteps/ClusterSuspendStepTest.java     | 36 +++++++++++++++
 5 files changed, 101 insertions(+), 33 deletions(-)

diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index e9d95fb3..769d62f0 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -591,7 +591,9 @@ spec:
   HorizontalPodAutoscaler and PodDisruptionBudget of its workers if any. Its 
Services and
   NetworkPolicy are kept. Any application running on the cluster is terminated 
with it, since there
   is no graceful decommission. A failure to delete them, other than one which 
may clear on its own,
-  publishes the `SuspendReleaseFailed` 
[event](configuration.md#kubernetes-events) by default.
+  publishes the `SuspendReleaseFailed` 
[event](configuration.md#kubernetes-events) by default. The
+  HorizontalPodAutoscaler and PodDisruptionBudget hold no quota, so they are 
deleted after the
+  Kueue `Workload`, if any, is released, and a failure to delete them does not 
hold the quota.
 * Setting it back to `false` moves the cluster to `Submitted` once its master 
and worker pods are
   gone and its Kueue `Workload`, if any, is released, and the master and 
workers are requested
   again from scratch, like a newly created cluster. Only pods labeled with the 
`master` or `worker`
@@ -944,6 +946,6 @@ the cluster as well, and the cluster moves to 
`ResourceReleased`, which is not r
 further. So delete and recreate the cluster to request its resources again. A 
failure to delete
 them or the `Workload`, other than one which may clear on its own, publishes 
the
 `ClusterReleaseFailed` [event](configuration.md#kubernetes-events) by default, 
and the cluster stays
-in `SchedulingFailure` until the deletion succeeds. The Services and the 
NetworkPolicy hold no
-quota, so they are deleted after the `Workload` is released, and a failure to 
delete them does not
-hold the quota.
+in `SchedulingFailure` until the deletion succeeds. The Services, the 
NetworkPolicy, the
+HorizontalPodAutoscaler and the PodDisruptionBudget hold no quota, so they are 
deleted after the
+`Workload` is released, and a failure to delete them does not hold the quota.
diff --git 
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStep.java
 
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStep.java
index 13fde4c9..dd9a62b3 100644
--- 
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStep.java
+++ 
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStep.java
@@ -63,12 +63,13 @@ public final class ClusterCleanUpStep extends 
ClusterReconcileStep {
    * <p>The Workload is released only after the master and worker pods are 
gone, or are still
    * terminating past their timeout, see {@link 
ClusterSuspendStep#waitForPods}. A pod stuck in
    * terminating does not hold the cluster beyond that, since no pod of its 
name is created again.
-   * The Services and the NetworkPolicy hold no quota, so they are deleted 
only after the Workload
-   * is released, like the resources of an application in {@link 
AppCleanUpStep}. Otherwise a
-   * failure to delete them, e.g. by an operator which may not access them and 
so failed to apply
-   * them, would hold the quota forever. Everything here is idempotent, so a 
release that failed is
-   * simply retried. A failure which is not expected to clear on its own is 
reported, since
-   * SchedulingFailure reports the failure to request the resources, not the 
one to release them.
+   * The Services, the NetworkPolicy, and the HorizontalPodAutoscaler and the 
PodDisruptionBudget of
+   * the workers hold no quota, so they are deleted only after the Workload is 
released, like the
+   * resources of an application in {@link AppCleanUpStep}. Otherwise a 
failure to delete them, e.g.
+   * by an operator which may not access them and so failed to apply them, 
would hold the quota
+   * forever. Everything here is idempotent, so a release that failed is 
simply retried. A failure
+   * which is not expected to clear on its own is reported, since 
SchedulingFailure reports the
+   * failure to request the resources, not the one to release them.
    *
    * @param context The SparkClusterContext for the cluster.
    * @param statusRecorder The SparkClusterStatusRecorder for recording status 
updates.
@@ -94,6 +95,7 @@ public final class ClusterCleanUpStep extends 
ClusterReconcileStep {
       }
       // A Workload admitted before the queue label was removed is released as 
well.
       KueueWorkloadUtils.deleteWorkloadOf(client, cluster);
+      ClusterSuspendStep.deleteWorkerAutoscalerAndDisruptionBudget(client, 
cluster);
       
client.services().inNamespace(namespace).withName(getMasterServiceName(name)).delete();
       
client.services().inNamespace(namespace).withName(getWorkerServiceName(name)).delete();
       client
diff --git 
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
 
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
index a1db1961..8f1f3379 100644
--- 
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
+++ 
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
@@ -248,10 +248,15 @@ public final class ClusterSuspendStep extends 
ClusterReconcileStep {
    * application, which are force deleted past 
`forceTerminationGracePeriodMillis`, such a pod is
    * not force deleted, since its container may still run on a partitioned 
node while a pod of the
    * same StatefulSet identity is created again. Other pods which carry the 
cluster label, e.g. to
-   * reach the workers through their NetworkPolicy, do not count. Everything 
here is idempotent, so
-   * pods that are not gone yet, or a release that failed, are simply looked 
at again. A failure
-   * which is not expected to clear on its own is reported, since Suspended is 
a steady state which
-   * would not tell it apart from waiting for the pods to go.
+   * reach the workers through their NetworkPolicy, do not count. The 
HorizontalPodAutoscaler and
+   * the PodDisruptionBudget hold no quota, so they are deleted only after the 
Workload is released,
+   * like in {@link ClusterCleanUpStep}. Otherwise a failure to delete them, 
e.g. by an operator
+   * which may not access them since none of its clusters asks for them, would 
hold the quota
+   * forever. A resumed cluster still waits for them to be deleted, so that it 
does not keep one
+   * which its spec no longer asks for. Everything here is idempotent, so pods 
that are not gone
+   * yet, or a release that failed, are simply looked at again. A failure 
which is not expected to
+   * clear on its own is reported, since Suspended is a steady state which 
would not tell it apart
+   * from waiting for the pods to go.
    *
    * @param context The SparkClusterContext for the cluster.
    * @param statusRecorder The SparkClusterStatusRecorder for recording status 
updates.
@@ -276,6 +281,7 @@ public final class ClusterSuspendStep extends 
ClusterReconcileStep {
         // A Workload admitted before the queue label was removed is released 
as well.
         KueueWorkloadUtils.deleteWorkloadOf(client, cluster);
       }
+      deleteWorkerAutoscalerAndDisruptionBudget(client, cluster);
       if (!pods.isEmpty()) {
         log.debug("Waiting for the stuck pods of the suspended cluster to be 
deleted.");
         // This may last until a lost node is deleted, so the state says why, 
once per set of pods
@@ -317,10 +323,9 @@ public final class ClusterSuspendStep extends 
ClusterReconcileStep {
   }
 
   /**
-   * Deletes the master and worker StatefulSets of the given cluster, and the
-   * HorizontalPodAutoscaler and the PodDisruptionBudget of its workers, by 
name, and lists its
-   * master and worker pods, see {@link #releaseResources}. {@link 
ClusterCleanUpStep} releases a
-   * cluster which failed to request its resources the same way.
+   * Deletes the master and worker StatefulSets of the given cluster by name, 
and lists its master
+   * and worker pods, see {@link #releaseResources}. {@link 
ClusterCleanUpStep} releases a cluster
+   * which failed to request its resources the same way.
    *
    * @param client The KubernetesClient.
    * @param cluster The SparkCluster.
@@ -343,6 +348,30 @@ public final class ClusterSuspendStep extends 
ClusterReconcileStep {
         .inNamespace(namespace)
         .withName(getMasterStatefulSetName(name))
         .delete();
+    return ReconcilerUtils.podsOf(
+            client,
+            namespace,
+            LABEL_SPARK_CLUSTER_NAME,
+            name,
+            LABEL_SPARK_ROLE_MASTER_VALUE,
+            LABEL_SPARK_ROLE_WORKER_VALUE)
+        .list()
+        .getItems();
+  }
+
+  /**
+   * Deletes the HorizontalPodAutoscaler and the PodDisruptionBudget of the 
workers of the given
+   * cluster by name, see {@link #releaseResources}. They hold no quota, so 
they are deleted only
+   * after the Kueue Workload is released, here and in {@link 
ClusterCleanUpStep}.
+   *
+   * @param client The KubernetesClient.
+   * @param cluster The SparkCluster.
+   * @throws KubernetesClientException if a deletion fails.
+   */
+  static void deleteWorkerAutoscalerAndDisruptionBudget(
+      final KubernetesClient client, final SparkCluster cluster) {
+    String namespace = cluster.getMetadata().getNamespace();
+    String name = cluster.getMetadata().getName();
     client
         .autoscaling()
         .v2()
@@ -357,15 +386,6 @@ public final class ClusterSuspendStep extends 
ClusterReconcileStep {
         .inNamespace(namespace)
         .withName(getWorkerPodDisruptionBudgetName(name))
         .delete();
-    return ReconcilerUtils.podsOf(
-            client,
-            namespace,
-            LABEL_SPARK_CLUSTER_NAME,
-            name,
-            LABEL_SPARK_ROLE_MASTER_VALUE,
-            LABEL_SPARK_ROLE_WORKER_VALUE)
-        .list()
-        .getItems();
   }
 
   /**
diff --git 
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStepTest.java
 
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStepTest.java
index 5409e6cf..2f249b17 100644
--- 
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStepTest.java
+++ 
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStepTest.java
@@ -64,6 +64,7 @@ import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.params.ParameterizedTest;
 import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
 import org.mockito.ArgumentCaptor;
 
 import org.apache.spark.k8s.operator.Constants;
@@ -162,12 +163,19 @@ class ClusterCleanUpStepTest {
         ClusterStateSummary.ResourceReleased, 
captureAppendedState().getCurrentStateSummary());
   }
 
-  @Test
-  void failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload() {
+  @ParameterizedTest
+  @ValueSource(strings = {"network", "autoscaling", "policy"})
+  void failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(String 
apiGroup) {
     SparkCluster cluster = buildCluster(ClusterStateSummary.SchedulingFailure);
     KubernetesClient client = spy(kubernetesClient);
-    // e.g. an operator which may not access NetworkPolicies, which failed the 
cluster as well
-    doThrow(new KubernetesClientException("forbidden", 403, 
null)).when(client).network();
+    // e.g. an operator which may not access NetworkPolicies, 
HorizontalPodAutoscalers or
+    // PodDisruptionBudgets, which failed the cluster as well
+    KubernetesClientException forbidden = new 
KubernetesClientException("forbidden", 403, null);
+    switch (apiGroup) {
+      case "autoscaling" -> doThrow(forbidden).when(client).autoscaling();
+      case "policy" -> doThrow(forbidden).when(client).policy();
+      default -> doThrow(forbidden).when(client).network();
+    }
     stubContext(cluster, client);
     when(mockContext.getEventRecorder()).thenReturn(eventRecorder);
     
kubernetesClient.resource(KueueWorkloadFactory.buildWorkload(cluster)).create();
@@ -176,8 +184,8 @@ class ClusterCleanUpStepTest {
         ReconcileProgress.completeAndDefaultRequeue(),
         new ClusterCleanUpStep().reconcile(mockContext, recorder));
 
-    // The NetworkPolicy holds no quota, so the Workload is released even 
though it is not deleted,
-    // while the cluster stays in SchedulingFailure and reports the failure
+    // The resource holds no quota, so the Workload is released even though it 
is not deleted, while
+    // the cluster stays in SchedulingFailure and reports the failure
     Assertions.assertNull(getWorkload());
     verifyNoInteractions(recorder);
     ArgumentCaptor<EventRecord> event = 
ArgumentCaptor.forClass(EventRecord.class);
diff --git 
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
 
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
index c0d37d58..0f273976 100644
--- 
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
+++ 
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
@@ -24,6 +24,7 @@ import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.ArgumentMatchers.eq;
 import static org.mockito.Mockito.clearInvocations;
 import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.never;
 import static org.mockito.Mockito.spy;
@@ -429,6 +430,41 @@ class ClusterSuspendStepTest {
     verify(mockContext, never()).getEventRecorder();
   }
 
+  @ParameterizedTest
+  @CsvSource({"false, autoscaling", "false, policy", "true, autoscaling", 
"true, policy"})
+  void failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(
+      boolean evicted, String apiGroup) {
+    SparkCluster cluster =
+        evicted
+            ? buildEvictedCluster(false)
+            : buildKueueCluster(ClusterStateSummary.Suspended, true);
+    KubernetesClient client = spy(kubernetesClient);
+    // e.g. an operator which may not access HorizontalPodAutoscalers or 
PodDisruptionBudgets, since
+    // none of its clusters asks for them
+    KubernetesClientException forbidden = new 
KubernetesClientException("forbidden", 403, null);
+    if ("autoscaling".equals(apiGroup)) {
+      doThrow(forbidden).when(client).autoscaling();
+    } else {
+      doThrow(forbidden).when(client).policy();
+    }
+    stubContext(cluster, client);
+    when(mockContext.getEventRecorder()).thenReturn(eventRecorder);
+    createWorkload(cluster, evicted ? evictedStatus("Preempted") : 
admittedStatus());
+
+    Assertions.assertEquals(
+        ReconcileProgress.completeAndDefaultRequeue(),
+        new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+    // The resource holds no quota, so the Workload is released even though it 
is not deleted, e.g.
+    // for the workload which preempted the cluster, while the cluster stays 
Suspended and reports
+    // the failure
+    Assertions.assertNull(getWorkload());
+    verifyNoInteractions(recorder);
+    ArgumentCaptor<EventRecord> event = 
ArgumentCaptor.forClass(EventRecord.class);
+    verify(eventRecorder).record(event.capture());
+    Assertions.assertEquals(EventUtils.REASON_SUSPEND_RELEASE_FAILED, 
event.getValue().reason());
+  }
+
   @ParameterizedTest
   @ValueSource(ints = {403, 429, 500})
   void persistentReleaseFailureIsReportedAndRetried(int code) {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to