dongjoon-hyun opened a new pull request, #980:
URL: https://github.com/apache/spark-kubernetes-operator/pull/980
### What changes were proposed in this pull request?
This PR aims to release the resources of a `SparkCluster` which fails with
`SchedulingFailure`, and then to terminate it as `ResourceReleased` instead of
`Failed`.
- A new `ClusterCleanUpStep` reconciles a cluster in `SchedulingFailure`,
which `ClusterInitStep` records with the cause as before.
- It deletes the master and worker `StatefulSet`s and `Service`s, the
`NetworkPolicy`, and the `HorizontalPodAutoscaler` and `PodDisruptionBudget` of
the workers by name, with the name getters of `SparkClusterResourceSpec`
(SPARK-60438), since the spec may not build, which may be the failure itself.
- Once the master and worker pods are gone, or are still terminating 5
minutes after their grace period ended, it deletes the Kueue `Workload` and
moves the cluster to `ResourceReleased`, whose message points back to the
preceding `SchedulingFailure` state for the cause.
- A failure of the release is retried with the default interval, and
reported with the new `ClusterReleaseFailed` warning event unless it may clear
on its own.
- `ClusterSuspendStep` shares the deletion by name
(`deleteMasterAndWorkers`) and the wait for the pods (`waitForPods`) with it.
- A `Failed` cluster, which an earlier version leaves behind, is left as it
is, so that an upgrade does not delete a master which may be in use.
### Why are the changes needed?
`ClusterInitStep` applies the `Service`s, the master and worker
`StatefulSet`s, the `NetworkPolicy`, the `HorizontalPodAutoscaler` and the
`PodDisruptionBudget` in order. When a request is rejected, e.g. with `403` by
a `ResourceQuota` or an admission policy, or with `422`, the cluster moves to
`SchedulingFailure` and then to `Failed`, which is never reconciled again. As a
result,
- the master and workers which were requested before the rejection keep
running,
- with `spark.kubernetes.operator.kueue.enabled`, the Kueue `Workload` keeps
holding the quota, and
- neither `spec.suspend` nor a spec change releases them, since
`ClusterSuspendStep` does not run in `Failed`. Only deleting the `SparkCluster`
does.
The same happens when the spec of a queued cluster is changed to `minWorkers
< maxWorkers` while its `Workload` waits for the admission, since
`KueueWorkloadFactory.buildWorkload` throws `UnsupportedOperationException` for
it. In contrast, the Kueue flavor conflict path of the same cluster
(`KueueWorkloadUtils.holdForAdmission`) and `SparkApplication`
(`AppCleanUpStep`) release their resources.
`ResourceReleased` is used as the final state because:
- It states that the resources are released. It has existed in
`ClusterStateSummary` and the CRD since 0.1.0, so no CRD update is needed, and
`isTerminated()` makes `ClusterTerminatedStep` end the reconciliation.
- A `SparkApplication` also moves from `SchedulingFailure` to
`ResourceReleased` after releasing its resources.
- It tells a released cluster apart from a `Failed` cluster which an earlier
version left behind with its resources.
- The cause stays in the preceding `SchedulingFailure` state and its warning
event.
### Does this PR introduce _any_ user-facing change?
Yes. Compared with
[1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0)
(2026-07-23), a `SparkCluster` which fails with `SchedulingFailure` releases
the resources which it may have requested and ends in `ResourceReleased`
instead of `Failed`. `docs/migration_guide.md` describes it, including that a
`Failed` cluster which 1.0.0 left behind has to be deleted to release its
resources.
The Kueue integration and the K8s events are not released yet, so releasing
the `Workload` and the `ClusterReleaseFailed` event need no migration item.
`architecture.md`, `configuration.md` and `spark_custom_resources.md` are
updated accordingly.
### How was this patch tested?
Pass the CIs with the newly added test cases. The following test cases of
`ClusterInitStepTest` reconcile a cluster with `SparkClusterReconciler`, and
they fail without this fix.
- `rejectedRequestReleasesResourcesRequestedBeforeIt`: the worker
`StatefulSet` is rejected after the master was requested.
- `rejectedRequestReleasesKueueWorkloadOncePodsAreGone`: the
`PodDisruptionBudget` is rejected after the `Workload` was admitted.
- `unsupportedSpecOfQueuedClusterReleasesPendingKueueWorkload`: the spec of
a queued cluster is changed to `minWorkers < maxWorkers`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
