This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 49971079 [SPARK-60475] Delete the HPA and PDB of a `SparkCluster`
after releasing its Kueue `Workload`
49971079 is described below
commit 4997107921b05bc0588e0b35c8ff85215142d199
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 10:25:50 2026 -0700
[SPARK-60475] Delete the HPA and PDB of a `SparkCluster` after releasing
its Kueue `Workload`
### What changes were proposed in this pull request?
This PR is a follow-up of SPARK-60463
(https://github.com/apache/spark-kubernetes-operator/pull/980). It deletes the
`HorizontalPodAutoscaler` (HPA) and `PodDisruptionBudget` (PDB) of a
`SparkCluster` only after its Kueue `Workload` is released. This applies both
when the cluster failed with `SchedulingFailure` and when it is suspended by
`spec.suspend` or by the eviction of its `Workload`.
- `ClusterSuspendStep`: Move the deletion of the worker HPA and PDB out of
`deleteMasterAndWorkers` into a new package-private helper,
`deleteWorkerAutoscalerAndDisruptionBudget`. `deleteMasterAndWorkers` now only
deletes the master and worker `StatefulSet`s and lists the master and worker
pods. `releaseResources` calls the new helper after
`KueueWorkloadUtils.deleteWorkloadOf`. A suspended cluster still waits for the
deletion before it is resumed or queued again.
- `ClusterCleanUpStep`: Call the new helper after
`KueueWorkloadUtils.deleteWorkloadOf`, together with the deletion of the
`Service`s and `NetworkPolicy`.
- Update the Javadoc and `docs/spark_custom_resources.md` accordingly.
### Why are the changes needed?
The HPA and PDB hold no Kueue quota. However, `ClusterCleanUpStep` and
`ClusterSuspendStep` deleted them before they released the `Workload`. If
deleting either of them keeps failing, the release never reaches
`deleteWorkloadOf`, and the `Workload` keeps its quota. A failed cluster keeps
it until the `SparkCluster` is deleted. A suspended cluster keeps it while it
is suspended, and an evicted cluster keeps the quota which its preemptor waits
for.
This happens to an operator which cannot access `horizontalpodautoscalers`
or `poddisruptionbudgets`. Its clusters can still run, because
`ClusterInitStep` applies the HPA and PDB only when the spec asks for them. For
example, a queued cluster never has an HPA, since
`KueueWorkloadFactory.buildWorkload` rejects `minWorkers < maxWorkers`, and a
cluster with `minWorkers: 0` has no PDB. The release deletes them by name, so
it gets `403` anyway. Such an operator also fails with `403` when [...]
The review of SPARK-60463 pointed out the same issue for the `Service`s and
`NetworkPolicy`, and SPARK-60463 fixed it for them
(https://github.com/apache/spark-kubernetes-operator/pull/980#discussion_r4349288708).
### Does this PR introduce _any_ user-facing change?
No for the released versions, because `ClusterCleanUpStep` and the
`spec.suspend` of `SparkCluster` are new in the unreleased 1.1.0. Compared to
the `main` branch, a `SparkCluster` in `SchedulingFailure` or `Suspended` now
releases its Kueue `Workload` even when deleting its HPA or PDB fails. The
failure is still reported with the `ClusterReleaseFailed` or
`SuspendReleaseFailed` event. The cluster stays in `SchedulingFailure` or
`Suspended` until the deletion succeeds.
### How was this patch tested?
Pass the CIs with the updated test cases. In each case, the client throws
`403` from `autoscaling()` or `policy()`.
-
`ClusterCleanUpStepTest.failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload`
is parameterized for `network()`, `autoscaling()` and `policy()`.
-
`ClusterSuspendStepTest.failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload`
is new. It covers a cluster suspended by `spec.suspend` and one suspended by
an eviction.
Without the fix, the new cases fail because the `Workload` is not released.
```
ClusterCleanUpStepTest >
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(String) > [2]
"autoscaling" FAILED
ClusterCleanUpStepTest >
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(String) > [3]
"policy" FAILED
ClusterSuspendStepTest >
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(boolean, String) >
[1] "false", "autoscaling" FAILED
ClusterSuspendStepTest >
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(boolean, String) >
[2] "false", "policy" FAILED
ClusterSuspendStepTest >
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(boolean, String) >
[3] "true", "autoscaling" FAILED
ClusterSuspendStepTest >
failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(boolean, String) >
[4] "true", "policy" FAILED
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #981 from dongjoon-hyun/SPARK-60475.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/spark_custom_resources.md | 10 ++--
.../reconcilesteps/ClusterCleanUpStep.java | 14 +++---
.../reconcilesteps/ClusterSuspendStep.java | 54 +++++++++++++++-------
.../reconcilesteps/ClusterCleanUpStepTest.java | 20 +++++---
.../reconcilesteps/ClusterSuspendStepTest.java | 36 +++++++++++++++
5 files changed, 101 insertions(+), 33 deletions(-)
diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index e9d95fb3..769d62f0 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -591,7 +591,9 @@ spec:
HorizontalPodAutoscaler and PodDisruptionBudget of its workers if any. Its
Services and
NetworkPolicy are kept. Any application running on the cluster is terminated
with it, since there
is no graceful decommission. A failure to delete them, other than one which
may clear on its own,
- publishes the `SuspendReleaseFailed`
[event](configuration.md#kubernetes-events) by default.
+ publishes the `SuspendReleaseFailed`
[event](configuration.md#kubernetes-events) by default. The
+ HorizontalPodAutoscaler and PodDisruptionBudget hold no quota, so they are
deleted after the
+ Kueue `Workload`, if any, is released, and a failure to delete them does not
hold the quota.
* Setting it back to `false` moves the cluster to `Submitted` once its master
and worker pods are
gone and its Kueue `Workload`, if any, is released, and the master and
workers are requested
again from scratch, like a newly created cluster. Only pods labeled with the
`master` or `worker`
@@ -944,6 +946,6 @@ the cluster as well, and the cluster moves to
`ResourceReleased`, which is not r
further. So delete and recreate the cluster to request its resources again. A
failure to delete
them or the `Workload`, other than one which may clear on its own, publishes
the
`ClusterReleaseFailed` [event](configuration.md#kubernetes-events) by default,
and the cluster stays
-in `SchedulingFailure` until the deletion succeeds. The Services and the
NetworkPolicy hold no
-quota, so they are deleted after the `Workload` is released, and a failure to
delete them does not
-hold the quota.
+in `SchedulingFailure` until the deletion succeeds. The Services, the
NetworkPolicy, the
+HorizontalPodAutoscaler and the PodDisruptionBudget hold no quota, so they are
deleted after the
+`Workload` is released, and a failure to delete them does not hold the quota.
diff --git
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStep.java
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStep.java
index 13fde4c9..dd9a62b3 100644
---
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStep.java
+++
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStep.java
@@ -63,12 +63,13 @@ public final class ClusterCleanUpStep extends
ClusterReconcileStep {
* <p>The Workload is released only after the master and worker pods are
gone, or are still
* terminating past their timeout, see {@link
ClusterSuspendStep#waitForPods}. A pod stuck in
* terminating does not hold the cluster beyond that, since no pod of its
name is created again.
- * The Services and the NetworkPolicy hold no quota, so they are deleted
only after the Workload
- * is released, like the resources of an application in {@link
AppCleanUpStep}. Otherwise a
- * failure to delete them, e.g. by an operator which may not access them and
so failed to apply
- * them, would hold the quota forever. Everything here is idempotent, so a
release that failed is
- * simply retried. A failure which is not expected to clear on its own is
reported, since
- * SchedulingFailure reports the failure to request the resources, not the
one to release them.
+ * The Services, the NetworkPolicy, and the HorizontalPodAutoscaler and the
PodDisruptionBudget of
+ * the workers hold no quota, so they are deleted only after the Workload is
released, like the
+ * resources of an application in {@link AppCleanUpStep}. Otherwise a
failure to delete them, e.g.
+ * by an operator which may not access them and so failed to apply them,
would hold the quota
+ * forever. Everything here is idempotent, so a release that failed is
simply retried. A failure
+ * which is not expected to clear on its own is reported, since
SchedulingFailure reports the
+ * failure to request the resources, not the one to release them.
*
* @param context The SparkClusterContext for the cluster.
* @param statusRecorder The SparkClusterStatusRecorder for recording status
updates.
@@ -94,6 +95,7 @@ public final class ClusterCleanUpStep extends
ClusterReconcileStep {
}
// A Workload admitted before the queue label was removed is released as
well.
KueueWorkloadUtils.deleteWorkloadOf(client, cluster);
+ ClusterSuspendStep.deleteWorkerAutoscalerAndDisruptionBudget(client,
cluster);
client.services().inNamespace(namespace).withName(getMasterServiceName(name)).delete();
client.services().inNamespace(namespace).withName(getWorkerServiceName(name)).delete();
client
diff --git
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
index a1db1961..8f1f3379 100644
---
a/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
+++
b/spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStep.java
@@ -248,10 +248,15 @@ public final class ClusterSuspendStep extends
ClusterReconcileStep {
* application, which are force deleted past
`forceTerminationGracePeriodMillis`, such a pod is
* not force deleted, since its container may still run on a partitioned
node while a pod of the
* same StatefulSet identity is created again. Other pods which carry the
cluster label, e.g. to
- * reach the workers through their NetworkPolicy, do not count. Everything
here is idempotent, so
- * pods that are not gone yet, or a release that failed, are simply looked
at again. A failure
- * which is not expected to clear on its own is reported, since Suspended is
a steady state which
- * would not tell it apart from waiting for the pods to go.
+ * reach the workers through their NetworkPolicy, do not count. The
HorizontalPodAutoscaler and
+ * the PodDisruptionBudget hold no quota, so they are deleted only after the
Workload is released,
+ * like in {@link ClusterCleanUpStep}. Otherwise a failure to delete them,
e.g. by an operator
+ * which may not access them since none of its clusters asks for them, would
hold the quota
+ * forever. A resumed cluster still waits for them to be deleted, so that it
does not keep one
+ * which its spec no longer asks for. Everything here is idempotent, so pods
that are not gone
+ * yet, or a release that failed, are simply looked at again. A failure
which is not expected to
+ * clear on its own is reported, since Suspended is a steady state which
would not tell it apart
+ * from waiting for the pods to go.
*
* @param context The SparkClusterContext for the cluster.
* @param statusRecorder The SparkClusterStatusRecorder for recording status
updates.
@@ -276,6 +281,7 @@ public final class ClusterSuspendStep extends
ClusterReconcileStep {
// A Workload admitted before the queue label was removed is released
as well.
KueueWorkloadUtils.deleteWorkloadOf(client, cluster);
}
+ deleteWorkerAutoscalerAndDisruptionBudget(client, cluster);
if (!pods.isEmpty()) {
log.debug("Waiting for the stuck pods of the suspended cluster to be
deleted.");
// This may last until a lost node is deleted, so the state says why,
once per set of pods
@@ -317,10 +323,9 @@ public final class ClusterSuspendStep extends
ClusterReconcileStep {
}
/**
- * Deletes the master and worker StatefulSets of the given cluster, and the
- * HorizontalPodAutoscaler and the PodDisruptionBudget of its workers, by
name, and lists its
- * master and worker pods, see {@link #releaseResources}. {@link
ClusterCleanUpStep} releases a
- * cluster which failed to request its resources the same way.
+ * Deletes the master and worker StatefulSets of the given cluster by name,
and lists its master
+ * and worker pods, see {@link #releaseResources}. {@link
ClusterCleanUpStep} releases a cluster
+ * which failed to request its resources the same way.
*
* @param client The KubernetesClient.
* @param cluster The SparkCluster.
@@ -343,6 +348,30 @@ public final class ClusterSuspendStep extends
ClusterReconcileStep {
.inNamespace(namespace)
.withName(getMasterStatefulSetName(name))
.delete();
+ return ReconcilerUtils.podsOf(
+ client,
+ namespace,
+ LABEL_SPARK_CLUSTER_NAME,
+ name,
+ LABEL_SPARK_ROLE_MASTER_VALUE,
+ LABEL_SPARK_ROLE_WORKER_VALUE)
+ .list()
+ .getItems();
+ }
+
+ /**
+ * Deletes the HorizontalPodAutoscaler and the PodDisruptionBudget of the
workers of the given
+ * cluster by name, see {@link #releaseResources}. They hold no quota, so
they are deleted only
+ * after the Kueue Workload is released, here and in {@link
ClusterCleanUpStep}.
+ *
+ * @param client The KubernetesClient.
+ * @param cluster The SparkCluster.
+ * @throws KubernetesClientException if a deletion fails.
+ */
+ static void deleteWorkerAutoscalerAndDisruptionBudget(
+ final KubernetesClient client, final SparkCluster cluster) {
+ String namespace = cluster.getMetadata().getNamespace();
+ String name = cluster.getMetadata().getName();
client
.autoscaling()
.v2()
@@ -357,15 +386,6 @@ public final class ClusterSuspendStep extends
ClusterReconcileStep {
.inNamespace(namespace)
.withName(getWorkerPodDisruptionBudgetName(name))
.delete();
- return ReconcilerUtils.podsOf(
- client,
- namespace,
- LABEL_SPARK_CLUSTER_NAME,
- name,
- LABEL_SPARK_ROLE_MASTER_VALUE,
- LABEL_SPARK_ROLE_WORKER_VALUE)
- .list()
- .getItems();
}
/**
diff --git
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStepTest.java
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStepTest.java
index 5409e6cf..2f249b17 100644
---
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStepTest.java
+++
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterCleanUpStepTest.java
@@ -64,6 +64,7 @@ import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
import org.mockito.ArgumentCaptor;
import org.apache.spark.k8s.operator.Constants;
@@ -162,12 +163,19 @@ class ClusterCleanUpStepTest {
ClusterStateSummary.ResourceReleased,
captureAppendedState().getCurrentStateSummary());
}
- @Test
- void failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload() {
+ @ParameterizedTest
+ @ValueSource(strings = {"network", "autoscaling", "policy"})
+ void failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(String
apiGroup) {
SparkCluster cluster = buildCluster(ClusterStateSummary.SchedulingFailure);
KubernetesClient client = spy(kubernetesClient);
- // e.g. an operator which may not access NetworkPolicies, which failed the
cluster as well
- doThrow(new KubernetesClientException("forbidden", 403,
null)).when(client).network();
+ // e.g. an operator which may not access NetworkPolicies,
HorizontalPodAutoscalers or
+ // PodDisruptionBudgets, which failed the cluster as well
+ KubernetesClientException forbidden = new
KubernetesClientException("forbidden", 403, null);
+ switch (apiGroup) {
+ case "autoscaling" -> doThrow(forbidden).when(client).autoscaling();
+ case "policy" -> doThrow(forbidden).when(client).policy();
+ default -> doThrow(forbidden).when(client).network();
+ }
stubContext(cluster, client);
when(mockContext.getEventRecorder()).thenReturn(eventRecorder);
kubernetesClient.resource(KueueWorkloadFactory.buildWorkload(cluster)).create();
@@ -176,8 +184,8 @@ class ClusterCleanUpStepTest {
ReconcileProgress.completeAndDefaultRequeue(),
new ClusterCleanUpStep().reconcile(mockContext, recorder));
- // The NetworkPolicy holds no quota, so the Workload is released even
though it is not deleted,
- // while the cluster stays in SchedulingFailure and reports the failure
+ // The resource holds no quota, so the Workload is released even though it
is not deleted, while
+ // the cluster stays in SchedulingFailure and reports the failure
Assertions.assertNull(getWorkload());
verifyNoInteractions(recorder);
ArgumentCaptor<EventRecord> event =
ArgumentCaptor.forClass(EventRecord.class);
diff --git
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
index c0d37d58..0f273976 100644
---
a/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
+++
b/spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterSuspendStepTest.java
@@ -24,6 +24,7 @@ import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.clearInvocations;
import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.spy;
@@ -429,6 +430,41 @@ class ClusterSuspendStepTest {
verify(mockContext, never()).getEventRecorder();
}
+ @ParameterizedTest
+ @CsvSource({"false, autoscaling", "false, policy", "true, autoscaling",
"true, policy"})
+ void failedReleaseOfResourceWithoutQuotaDoesNotHoldKueueWorkload(
+ boolean evicted, String apiGroup) {
+ SparkCluster cluster =
+ evicted
+ ? buildEvictedCluster(false)
+ : buildKueueCluster(ClusterStateSummary.Suspended, true);
+ KubernetesClient client = spy(kubernetesClient);
+ // e.g. an operator which may not access HorizontalPodAutoscalers or
PodDisruptionBudgets, since
+ // none of its clusters asks for them
+ KubernetesClientException forbidden = new
KubernetesClientException("forbidden", 403, null);
+ if ("autoscaling".equals(apiGroup)) {
+ doThrow(forbidden).when(client).autoscaling();
+ } else {
+ doThrow(forbidden).when(client).policy();
+ }
+ stubContext(cluster, client);
+ when(mockContext.getEventRecorder()).thenReturn(eventRecorder);
+ createWorkload(cluster, evicted ? evictedStatus("Preempted") :
admittedStatus());
+
+ Assertions.assertEquals(
+ ReconcileProgress.completeAndDefaultRequeue(),
+ new ClusterSuspendStep().reconcile(mockContext, recorder));
+
+ // The resource holds no quota, so the Workload is released even though it
is not deleted, e.g.
+ // for the workload which preempted the cluster, while the cluster stays
Suspended and reports
+ // the failure
+ Assertions.assertNull(getWorkload());
+ verifyNoInteractions(recorder);
+ ArgumentCaptor<EventRecord> event =
ArgumentCaptor.forClass(EventRecord.class);
+ verify(eventRecorder).record(event.capture());
+ Assertions.assertEquals(EventUtils.REASON_SUSPEND_RELEASE_FAILED,
event.getValue().reason());
+ }
+
@ParameterizedTest
@ValueSource(ints = {403, 429, 500})
void persistentReleaseFailureIsReportedAndRetried(int code) {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]