This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 6a06de45 [SPARK-60017] Fix `Workload` access revocation and
`Suspended` downgrade notes
6a06de45 is described below
commit 6a06de45cb868847c80660758a5940e2a26b412e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Oct 6 21:28:53 2026 -0700
[SPARK-60017] Fix `Workload` access revocation and `Suspended` downgrade
notes
### What changes were proposed in this pull request?
This PR aims to fix two inaccurate notes in
`docs/spark_custom_resources.md`.
- With the Kueue integration enabled but no access to `Workload`s, a
suspended `SparkCluster` and any stopping `SparkApplication` are held until the
access is restored or the integration is disabled, not until the `Workload` is
gone.
- Resuming a `SparkCluster` before downgrading the operator is enough only
when `spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled`
drops the `Suspended` state from its history.
### Why are the changes needed?
- The operator keeps deleting the `Workload` by name, which is denied
without the access even after the `Workload` is gone.
- An operator version without the `Suspended` state cannot read a
`SparkCluster` which has it in its state transition history.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change for features which are not released
yet in
[1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0)
(2026-07-23).
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #945 from dongjoon-hyun/SPARK-60017.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/spark_custom_resources.md | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index 673973b8..ef0911ea 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -584,8 +584,11 @@ spec:
a lost node, no longer holds the Kueue `Workload`, so that it does not hold
the quota forever. The
cluster still stays `Suspended` until such a pod is gone, since it keeps the
name of the master or
worker to create, and the message of its `Suspended` state names such pods.
-* An operator version without the `Suspended` state cannot read a
`SparkCluster` which is
- `Suspended`, so resume or delete such clusters before downgrading the
operator.
+* An operator version without the `Suspended` state cannot read a
`SparkCluster` whose status
+ has it, even after the cluster is resumed. A resumed `SparkCluster` keeps it
unless
+ `spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled`
drops the history of
+ its previous run. So delete such clusters, or resume the suspended ones with
that setting
+ enabled, before downgrading the operator.
* Deleting a suspended resource works as usual.
* This is the building block for external job queueing systems. See
[Kueue](#kueue) for the
built-in integration.
@@ -718,8 +721,9 @@ spec:
by disabling `operatorRbac.kueue.enabled`, the operator no longer deletes
them, and a `Workload`
left behind keeps its quota until its owner is deleted. While the
integration stays enabled
without the access, a suspended `SparkCluster` does not resume, and a
stopping
- `SparkApplication` with a `Workload` neither terminates nor restarts, until
its `Workload` is
- gone or the access is restored. Before disabling either, let the queued
+ `SparkApplication` neither terminates nor restarts, until the access is
restored or the
+ integration is disabled, since the operator keeps deleting their `Workload`s
by name, which is
+ denied even once a `Workload` is gone. Before disabling either, let the
queued
`SparkApplication`s finish or suspend those which have not started yet, and
suspend the queued
`SparkCluster`s, so that the operator releases their `Workload`s itself.
Then list the remaining
ones with `kubectl get workloads -A -l spark.operator/spark-app-name` and
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
