This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 6d079691 [SPARK-59964] Fix workload RBAC to grant `deletecollection`
to Spark drivers
6d079691 is described below
commit 6d0796914b3014e7c12ef638cddc351ab3b68fd2
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Oct 5 08:16:32 2026 -0700
[SPARK-59964] Fix workload RBAC to grant `deletecollection` to Spark drivers
### What changes were proposed in this pull request?
This PR aims to grant `deletecollection` on `pods`, `services`,
`configmaps` and `persistentvolumeclaims` to the Spark workload ClusterRole and
Roles of the Helm chart, with Helm test assertions and a migration guide item.
### Why are the changes needed?
SPARK-57942 replaced the wildcard RBAC verbs with an explicit verb list
without `deletecollection` in
[1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0)
(2026-07-23). However, Spark drivers delete the resources of their application
by label selector, which K8s RBAC authorizes as `deletecollection`. So every
driver logs errors like the following when it stops, and leaves the resources
until the driver pod is deleted.
```
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
DELETE at:
https://kubernetes.default.svc/api/v1/namespaces/default/services?labelSelector=spark-app-selector%3Dpi-before-0.
Message: services is forbidden: User "system:serviceaccount:default:spark"
cannot deletecollection resource "services" in API group "" in the namespace
"default". ...
```
### Does this PR introduce _any_ user-facing change?
Yes. This fixes a regression of 1.0.0.
### How was this patch tested?
Pass the CIs with the new Helm test assertions. I also verified manually on
Minikube with the operator `1.0.0` and Spark `4.2.0` that the driver logs no
error and deletes its resources when it stops.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This patch had conflicts when merged, resolved by
Committer: Dongjoon Hyun <[email protected]>
Closes #931 from dongjoon-hyun/SPARK-59964.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark-kubernetes-operator/templates/tests/test-rbac.yaml | 4 ++++
.../spark-kubernetes-operator/templates/workload-rbac.yaml | 1 +
docs/migration_guide.md | 10 ++++++++++
3 files changed, 15 insertions(+)
diff --git
a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
index deefe934..74ff44ab 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
@@ -145,6 +145,10 @@ spec:
kubectl auth can-i update statefulsets.apps --subresource=scale
--namespace={{ .Release.Namespace }}
kubectl auth can-i get deployments.apps --subresource=scale
--namespace={{ .Release.Namespace }}
kubectl auth can-i update deployments.apps --subresource=scale
--namespace={{ .Release.Namespace }}
+ kubectl auth can-i deletecollection pods --namespace={{
.Release.Namespace }}
+ kubectl auth can-i deletecollection services --namespace={{
.Release.Namespace }}
+ kubectl auth can-i deletecollection configmaps --namespace={{
.Release.Namespace }}
+ kubectl auth can-i deletecollection persistentvolumeclaims
--namespace={{ .Release.Namespace }}
# Spark workloads must not reach beyond their own namespace. `set -e`
# ignores a failing `!` pipeline, so assert each denial explicitly.
diff --git
a/build-tools/helm/spark-kubernetes-operator/templates/workload-rbac.yaml
b/build-tools/helm/spark-kubernetes-operator/templates/workload-rbac.yaml
index 213a506e..5de37a91 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/workload-rbac.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/workload-rbac.yaml
@@ -33,6 +33,7 @@ rules:
- update
- patch
- delete
+ - deletecollection
- apiGroups:
- "apps"
resources:
diff --git a/docs/migration_guide.md b/docs/migration_guide.md
index 2b19793c..7370961a 100644
--- a/docs/migration_guide.md
+++ b/docs/migration_guide.md
@@ -73,6 +73,16 @@ affect building the project, CI, tests or examples.
failed to start, since creating the Deployment or scaling the StatefulSet
was forbidden. To keep
the rules of 1.0, set `workloadResources.clusterRole.create` and
`workloadResources.role.create`
to `false`, and grant them with your own Role or ClusterRole.
+- Since 1.1.0, the workload ClusterRole and Roles grant `deletecollection` on
`pods`, `services`,
+ `configmaps` and `persistentvolumeclaims`, which Spark drivers need to
delete the resources of
+ their application by label selector. 1.0 did not grant it, so drivers logged
errors like
+ `cannot deletecollection resource "services"` when they stopped, and left
the driver service,
+ the on-demand PVCs and the executor pods and configmaps until the driver pod
was deleted, e.g.
+ as long as `resourceRetainPolicy` retained it. With dynamic allocation, they
could not delete
+ unneeded pending executor pods either. If you manage the workload Role or
ClusterRole yourself,
+ add `deletecollection` to it as well. To keep the driver service and the
executor pods and
+ configmaps, set `spark.kubernetes.driver.service.deleteOnTermination` and
+ `spark.kubernetes.executor.deleteOnTermination` to `false`.
- Since 1.1.0, each option of
`operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
to the operator JVM as its own argument. 1.0 passed the whole value as one
argument, which the
JVM took as a single `-Dfile.encoding` system property, so the other options
had no effect. The
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
