This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 0872073d [SPARK-59953] Add Kueue examples and use them as the Kueue
E2E fixtures
0872073d is described below
commit 0872073d29f0fa7d6381f5e2957d91cdd8a268af
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 2 10:50:24 2026 -0700
[SPARK-59953] Add Kueue examples and use them as the Kueue E2E fixtures
### What changes were proposed in this pull request?
- Add Kueue examples: `kueue-single-clusterqueue-setup.yaml`,
`pi-on-kueue.yaml`, and `cluster-on-kueue.yaml`.
- Link them from `docs/spark_custom_resources.md` and `docs/operations.md`.
- Use them as the fixtures of the two admission-wait steps in the Kueue
E2E, and delete the two duplicated fixtures. The flavor injection checks move
to the retained-resources and PodsReady steps, because the example flavor is
empty.
### Why are the changes needed?
Users have no ready-to-apply Kueue example, unlike YuniKorn and Volcano.
Reusing them in the E2E keeps them working.
### Does this PR introduce _any_ user-facing change?
No. It only adds examples and documentation links for the unreleased Kueue
support.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
Closes #926 from dongjoon-hyun/SPARK-59953.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/operations.md | 3 +-
docs/spark_custom_resources.md | 9 +-
.../cluster-on-kueue.yaml | 34 ++-
examples/kueue-single-clusterqueue-setup.yaml | 47 ++++
.../pi-on-kueue.yaml | 24 +-
tests/e2e/kueue/chainsaw-test.yaml | 273 ++++++++++-----------
tests/e2e/kueue/spark-example-retained-queued.yaml | 2 +
7 files changed, 231 insertions(+), 161 deletions(-)
diff --git a/docs/operations.md b/docs/operations.md
index 581ef706..820403cf 100644
--- a/docs/operations.md
+++ b/docs/operations.md
@@ -41,7 +41,8 @@ under the License.
- **Kueue** (`workloads.kueue.x-k8s.io`, `resourceflavors.kueue.x-k8s.io`,
`workloadpriorityclasses.kueue.x-k8s.io`) — required only when
`operatorRbac.kueue.enabled` is
set. Kueue is not bundled with the operator; install it from
- [kueue.sigs.k8s.io](https://kueue.sigs.k8s.io/docs/installation/). Kueue's
+ [kueue.sigs.k8s.io](https://kueue.sigs.k8s.io/docs/installation/), and set
up its queues, e.g.
+ with
[kueue-single-clusterqueue-setup.yaml](../examples/kueue-single-clusterqueue-setup.yaml).
Kueue's
`integrations.externalFrameworks` does not need to list `SparkApplication`
or `SparkCluster`:
the operator manages the `Workload` objects for them, so Kueue does not have
to
recognize the Spark custom resources as job kinds.
`operatorRbac.kueue.enabled` also sets
diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index 3de5917d..0ddd26c8 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -597,16 +597,23 @@ labeled with `kueue.x-k8s.io/queue-name` is queued by
[Kueue](https://kueue.sigs
operator creates a Kueue `Workload` that describes the driver and executor (or
master and worker)
pod sets, and holds the creation of those resources until Kueue admits the
`Workload`.
+See
[kueue-single-clusterqueue-setup.yaml](../examples/kueue-single-clusterqueue-setup.yaml)
for the
+queues the administrator sets up, and
[pi-on-kueue.yaml](../examples/pi-on-kueue.yaml) and
+[cluster-on-kueue.yaml](../examples/cluster-on-kueue.yaml) for a
`SparkApplication` and a
+`SparkCluster` labeled with the `LocalQueue` of the setup.
+
```yaml
apiVersion: spark.apache.org/v1
kind: SparkApplication
metadata:
- name: spark-pi
+ name: pi-on-kueue
labels:
kueue.x-k8s.io/queue-name: spark-queue
spec:
mainClass: "org.apache.spark.examples.SparkPi"
jars: "local:///opt/spark/examples/jars/spark-examples.jar"
+ sparkConf:
+ spark.executor.instances: "1"
runtimeVersions:
sparkVersion: "4.2.0"
```
diff --git a/tests/e2e/kueue/spark-cluster-example-queued.yaml
b/examples/cluster-on-kueue.yaml
similarity index 53%
rename from tests/e2e/kueue/spark-cluster-example-queued.yaml
rename to examples/cluster-on-kueue.yaml
index f0dd701a..0585ba0c 100644
--- a/tests/e2e/kueue/spark-cluster-example-queued.yaml
+++ b/examples/cluster-on-kueue.yaml
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -6,22 +5,22 @@
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-#
-
+# Requires Kueue and `operatorRbac.kueue.enabled=true`. Apply
kueue-single-clusterqueue-setup.yaml
+# first. The operator holds the master and the worker until Kueue admits the
Workload of the
+# cluster, which keeps its quota until the cluster is deleted or suspended.
apiVersion: spark.apache.org/v1
kind: SparkCluster
metadata:
- name: spark-cluster-kueue-queued-test
- namespace: default
+ name: cluster-on-kueue
labels:
- kueue.x-k8s.io/queue-name: ($LOCAL_QUEUE)
+ kueue.x-k8s.io/queue-name: spark-queue
spec:
runtimeVersions:
sparkVersion: "4.2.0"
@@ -30,8 +29,27 @@ spec:
initWorkers: 1
minWorkers: 1
maxWorkers: 1
+ workerSpec:
+ statefulSetSpec:
+ template:
+ spec:
+ containers:
+ - name: worker
+ # Without these, the worker advertises the whole node to its
executors, well above
+ # the quota which Kueue accounts for it.
+ env:
+ - name: SPARK_WORKER_CORES
+ value: "1"
+ - name: SPARK_WORKER_MEMORY
+ value: "1g"
+ resources:
+ requests:
+ cpu: "1"
+ memory: "3Gi"
+ limits:
+ cpu: "1"
+ memory: "3Gi"
sparkConf:
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
- spark.master.ui.title: "Spark Cluster Kueue E2E Test"
spark.master.rest.host: "0.0.0.0"
spark.ui.reverseProxy: "true"
diff --git a/examples/kueue-single-clusterqueue-setup.yaml
b/examples/kueue-single-clusterqueue-setup.yaml
new file mode 100644
index 00000000..17581a2f
--- /dev/null
+++ b/examples/kueue-single-clusterqueue-setup.yaml
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# A single ClusterQueue setup for the Kueue examples, like the
single-clusterqueue-setup of Kueue.
+# Kueue must be installed, and this file must be applied before
pi-on-kueue.yaml and
+# cluster-on-kueue.yaml. The operator needs the Helm value
`operatorRbac.kueue.enabled=true`.
+# The quota fits both examples at once, since a SparkCluster holds its quota
until it is deleted
+# or suspended. Adjust it to the capacity of your cluster.
+apiVersion: kueue.x-k8s.io/v1beta2
+kind: ResourceFlavor
+metadata:
+ name: default-flavor
+---
+apiVersion: kueue.x-k8s.io/v1beta2
+kind: ClusterQueue
+metadata:
+ name: spark-cluster-queue
+spec:
+ namespaceSelector: {}
+ resourceGroups:
+ - coveredResources: ["cpu", "memory"]
+ flavors:
+ - name: default-flavor
+ resources:
+ - name: "cpu"
+ nominalQuota: 4
+ - name: "memory"
+ nominalQuota: 8Gi
+---
+apiVersion: kueue.x-k8s.io/v1beta2
+kind: LocalQueue
+metadata:
+ namespace: default
+ name: spark-queue
+spec:
+ clusterQueue: spark-cluster-queue
diff --git a/tests/e2e/kueue/spark-example-queued.yaml
b/examples/pi-on-kueue.yaml
similarity index 68%
rename from tests/e2e/kueue/spark-example-queued.yaml
rename to examples/pi-on-kueue.yaml
index 2abe6c8f..796dbe41 100644
--- a/tests/e2e/kueue/spark-example-queued.yaml
+++ b/examples/pi-on-kueue.yaml
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -6,34 +5,31 @@
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
-# http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-#
-
+# Requires Kueue and `operatorRbac.kueue.enabled=true`. Apply
kueue-single-clusterqueue-setup.yaml
+# first. The operator holds the driver until Kueue admits the Workload of the
application.
apiVersion: spark.apache.org/v1
kind: SparkApplication
metadata:
- name: spark-job-kueue-queued-test
- namespace: default
+ name: pi-on-kueue
labels:
- kueue.x-k8s.io/queue-name: ($LOCAL_QUEUE)
+ kueue.x-k8s.io/queue-name: spark-queue
spec:
mainClass: "org.apache.spark.examples.SparkPi"
jars: "local:///opt/spark/examples/jars/spark-examples.jar"
- # The Kueue admission takes one more reconciliation, so a queued application
that finishes
- # within the first rate limit window (5 reconciliations per 15 seconds) is
observed only after
- # its driver completed. This is a property of every queued resource, which
is documented in
- # docs/spark_custom_resources.md. This test runs longer than the window to
check the regular
- # state transitions.
- driverArgs: ["10000"]
+ # A queued application takes one more reconciliation for the admission. This
one runs long
+ # enough to watch its state transitions.
+ driverArgs: [ "10000" ]
sparkConf:
+ # Kueue requests the quota upfront, so dynamic allocation is not supported.
spark.executor.instances: "1"
- spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
+ spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
runtimeVersions:
sparkVersion: "4.2.0"
diff --git a/tests/e2e/kueue/chainsaw-test.yaml
b/tests/e2e/kueue/chainsaw-test.yaml
index 477e3566..ddd09070 100644
--- a/tests/e2e/kueue/chainsaw-test.yaml
+++ b/tests/e2e/kueue/chainsaw-test.yaml
@@ -115,34 +115,45 @@ spec:
content: |
kubectl delete sparkcluster qa --ignore-not-found=true
- name: spark-application-waits-for-kueue-admission
- # Each step uses its own queue objects, so that a step never waits for
Kueue to finish
- # deleting the objects of another step.
+ # The queues of the official example have fixed names, so that only this
step and the next one
+ # use them. This step creates them, and the cluster step deletes them,
since Kueue may not
+ # have finished deleting them when a step creates them again.
bindings:
- name: FLAVOR
- value: spark-e2e-app-flavor
+ value: default-flavor
- name: CLUSTER_QUEUE
- value: spark-e2e-app-cluster-queue
+ value: spark-cluster-queue
- name: LOCAL_QUEUE
- value: spark-e2e-app-queue
+ value: spark-queue
try:
- apply:
- file: kueue-queues.yaml
+ file: ../../../examples/kueue-single-clusterqueue-setup.yaml
+ # No quota at first, so that an admission has to wait
+ - script:
+ timeout: 30s
+ env:
+ - name: CLUSTER_QUEUE
+ value: ($CLUSTER_QUEUE)
+ content: |
+ kubectl patch clusterqueue "$CLUSTER_QUEUE" --type json -p \
+ '[{"op": "replace", "path":
"/spec/resourceGroups/0/flavors/0/resources/0/nominalQuota", "value": "0"},
+ {"op": "replace", "path":
"/spec/resourceGroups/0/flavors/0/resources/1/nominalQuota", "value": "0"}]'
- apply:
- file: spark-example-queued.yaml
+ file: ../../../examples/pi-on-kueue.yaml
- assert:
timeout: 60s
resource:
apiVersion: kueue.x-k8s.io/v1beta2
kind: Workload
metadata:
- name: sparkapplication-spark-job-kueue-queued-test
+ name: sparkapplication-pi-on-kueue
namespace: default
spec:
queueName: ($LOCAL_QUEUE)
- script:
timeout: 30s
content: |
- kubectl get workload sparkapplication-spark-job-kueue-queued-test -n
default -o json \
+ kubectl get workload sparkapplication-pi-on-kueue -n default -o json
\
| jq -r 'if ([.status.conditions[]? | select(.type == "Admitted"
and .status == "True")] | length) > 0 then "ADMITTED" else "PENDING" end'
check:
(contains($stdout, 'PENDING')): true
@@ -154,7 +165,7 @@ spec:
metadata:
namespace: default
labels:
- spark.operator/spark-app-name: spark-job-kueue-queued-test
+ spark.operator/spark-app-name: pi-on-kueue
- assert:
timeout: 60s
resource:
@@ -164,10 +175,10 @@ spec:
namespace: default
involvedObject:
kind: SparkApplication
- name: spark-job-kueue-queued-test
+ name: pi-on-kueue
type: Normal
reason: KueueAdmissionPending
- (contains(message || '',
'sparkapplication-spark-job-kueue-queued-test')): true
+ (contains(message || '', 'sparkapplication-pi-on-kueue')): true
# The ValidatingAdmissionPolicy of the chart lets the queued application
move to another
# queue, here and back again
- script:
@@ -177,47 +188,11 @@ spec:
value: ($LOCAL_QUEUE)
content: |
set -e
- kubectl label sparkapplication spark-job-kueue-queued-test -n
default --overwrite kueue.x-k8s.io/queue-name=another-queue
- kubectl label sparkapplication spark-job-kueue-queued-test -n
default --overwrite "kueue.x-k8s.io/queue-name=$LOCAL_QUEUE"
- - script:
- timeout: 30s
- env:
- - name: CLUSTER_QUEUE
- value: ($CLUSTER_QUEUE)
- content: |
- kubectl patch clusterqueue "$CLUSTER_QUEUE" --type json -p \
- '[{"op": "replace", "path":
"/spec/resourceGroups/0/flavors/0/resources/0/nominalQuota", "value": "4"},
- {"op": "replace", "path":
"/spec/resourceGroups/0/flavors/0/resources/1/nominalQuota", "value": "8Gi"}]'
- # The node labels and tolerations of the assigned ResourceFlavor are
injected into the driver
- # and the executor pods, which the driver creates.
- - assert:
- timeout: 5m
- resource:
- apiVersion: v1
- kind: Pod
- metadata:
- namespace: default
- labels:
- spark.operator/spark-app-name: spark-job-kueue-queued-test
- spark-role: driver
- spec:
- nodeSelector:
- kubernetes.io/os: linux
- (tolerations[?key == 'spark.apache.org/kueue-e2e'] | length(@)): 1
- - assert:
- timeout: 5m
- resource:
- apiVersion: v1
- kind: Pod
- metadata:
- namespace: default
- labels:
- spark.operator/spark-app-name: spark-job-kueue-queued-test
- spark-role: executor
- spec:
- nodeSelector:
- kubernetes.io/os: linux
- (tolerations[?key == 'spark.apache.org/kueue-e2e'] | length(@)): 1
+ kubectl label sparkapplication pi-on-kueue -n default --overwrite
kueue.x-k8s.io/queue-name=another-queue
+ kubectl label sparkapplication pi-on-kueue -n default --overwrite
"kueue.x-k8s.io/queue-name=$LOCAL_QUEUE"
+ # Applying the queues again restores the quota of the example
+ - apply:
+ file: ../../../examples/kueue-single-clusterqueue-setup.yaml
# Like Kueue built-in integrations, the operator reports that the driver
and the executor are
# ready, so that the waitForPodsReady timeout of Kueue does not evict the
Workload.
- assert:
@@ -226,7 +201,7 @@ spec:
apiVersion: kueue.x-k8s.io/v1beta2
kind: Workload
metadata:
- name: sparkapplication-spark-job-kueue-queued-test
+ name: sparkapplication-pi-on-kueue
namespace: default
status:
(conditions[?type == 'PodsReady' && status == 'True' && reason ==
'Started'] | length(@)): 1
@@ -235,7 +210,7 @@ spec:
- name: SPARK_APP_NAMESPACE
value: default
- name: SPARK_APPLICATION_NAME
- value: spark-job-kueue-queued-test
+ value: pi-on-kueue
timeout: 10m
file: "../assertions/spark-application/spark-state-transition.yaml"
- assert:
@@ -247,10 +222,10 @@ spec:
namespace: default
involvedObject:
kind: SparkApplication
- name: spark-job-kueue-queued-test
+ name: pi-on-kueue
type: Normal
reason: KueueAdmitted
- (contains(message || '',
'sparkapplication-spark-job-kueue-queued-test')): true
+ (contains(message || '', 'sparkapplication-pi-on-kueue')): true
# The Workload is deleted to release the quota once the application stops.
- error:
timeout: 60s
@@ -258,13 +233,13 @@ spec:
apiVersion: kueue.x-k8s.io/v1beta2
kind: Workload
metadata:
- name: sparkapplication-spark-job-kueue-queued-test
+ name: sparkapplication-pi-on-kueue
namespace: default
# Like a finished Kueue job, the terminated application keeps its queue
- script:
timeout: 30s
content: |
- kubectl label sparkapplication spark-job-kueue-queued-test -n
default kueue.x-k8s.io/queue-name-
+ kubectl label sparkapplication pi-on-kueue -n default
kueue.x-k8s.io/queue-name-
check:
($error != null): true
(contains($stderr, 'ValidatingAdmissionPolicy')): true
@@ -282,19 +257,8 @@ spec:
finally:
- script:
timeout: 120s
- env:
- - name: FLAVOR
- value: ($FLAVOR)
- - name: CLUSTER_QUEUE
- value: ($CLUSTER_QUEUE)
- - name: LOCAL_QUEUE
- value: ($LOCAL_QUEUE)
content: |
- kubectl delete sparkapplication spark-job-kueue-queued-test -n
default --ignore-not-found=true
- # Kueue finalizers may delay the deletion, which later steps do not
depend on.
- kubectl delete localqueue "$LOCAL_QUEUE" -n default
--ignore-not-found=true --wait=false
- kubectl delete clusterqueue "$CLUSTER_QUEUE" --ignore-not-found=true
--wait=false
- kubectl delete resourceflavor "$FLAVOR" --ignore-not-found=true
--wait=false
+ kubectl delete sparkapplication pi-on-kueue -n default
--ignore-not-found=true
- name: spark-application-with-retained-resources-finishes-its-workload
# Each step uses its own queue objects, so that a step never waits for
Kueue to finish
# deleting the objects of another step.
@@ -320,6 +284,36 @@ spec:
{"op": "replace", "path":
"/spec/resourceGroups/0/flavors/0/resources/1/nominalQuota", "value": "8Gi"}]'
- apply:
file: spark-example-retained-queued.yaml
+ # The node labels and tolerations of the assigned ResourceFlavor are
injected into the driver
+ # and the executor pods, which the driver creates.
+ - assert:
+ timeout: 5m
+ resource:
+ apiVersion: v1
+ kind: Pod
+ metadata:
+ namespace: default
+ labels:
+ spark.operator/spark-app-name: spark-job-kueue-retained-test
+ spark-role: driver
+ spec:
+ nodeSelector:
+ kubernetes.io/os: linux
+ (tolerations[?key == 'spark.apache.org/kueue-e2e'] | length(@)): 1
+ - assert:
+ timeout: 5m
+ resource:
+ apiVersion: v1
+ kind: Pod
+ metadata:
+ namespace: default
+ labels:
+ spark.operator/spark-app-name: spark-job-kueue-retained-test
+ spark-role: executor
+ spec:
+ nodeSelector:
+ kubernetes.io/os: linux
+ (tolerations[?key == 'spark.apache.org/kueue-e2e'] | length(@)): 1
- assert:
timeout: 10m
resource:
@@ -450,34 +444,45 @@ spec:
kubectl delete resourceflavor "$FLAVOR" --ignore-not-found=true
--wait=false
kubectl delete workloadpriorityclass "$PRIORITY_CLASS"
--ignore-not-found=true --wait=false
- name: spark-cluster-waits-for-kueue-admission
- # Each step uses its own queue objects, so that a step never waits for
Kueue to finish
- # deleting the objects of another step.
+ # The queues of the official example have fixed names, which the
application step created.
+ # This step deletes them at the end, since Kueue may not have finished
deleting them when a
+ # step creates them again.
bindings:
- name: FLAVOR
- value: spark-e2e-cluster-flavor
+ value: default-flavor
- name: CLUSTER_QUEUE
- value: spark-e2e-cluster-cluster-queue
+ value: spark-cluster-queue
- name: LOCAL_QUEUE
- value: spark-e2e-cluster-queue
+ value: spark-queue
try:
- apply:
- file: kueue-queues.yaml
+ file: ../../../examples/kueue-single-clusterqueue-setup.yaml
+ # No quota at first, so that an admission has to wait
+ - script:
+ timeout: 30s
+ env:
+ - name: CLUSTER_QUEUE
+ value: ($CLUSTER_QUEUE)
+ content: |
+ kubectl patch clusterqueue "$CLUSTER_QUEUE" --type json -p \
+ '[{"op": "replace", "path":
"/spec/resourceGroups/0/flavors/0/resources/0/nominalQuota", "value": "0"},
+ {"op": "replace", "path":
"/spec/resourceGroups/0/flavors/0/resources/1/nominalQuota", "value": "0"}]'
- apply:
- file: spark-cluster-example-queued.yaml
+ file: ../../../examples/cluster-on-kueue.yaml
- assert:
timeout: 60s
resource:
apiVersion: kueue.x-k8s.io/v1beta2
kind: Workload
metadata:
- name: sparkcluster-spark-cluster-kueue-queued-test
+ name: sparkcluster-cluster-on-kueue
namespace: default
spec:
queueName: ($LOCAL_QUEUE)
- script:
timeout: 30s
content: |
- kubectl get workload sparkcluster-spark-cluster-kueue-queued-test -n
default -o json \
+ kubectl get workload sparkcluster-cluster-on-kueue -n default -o
json \
| jq -r 'if ([.status.conditions[]? | select(.type == "Admitted"
and .status == "True")] | length) > 0 then "ADMITTED" else "PENDING" end'
check:
(contains($stdout, 'PENDING')): true
@@ -487,7 +492,7 @@ spec:
apiVersion: apps/v1
kind: StatefulSet
metadata:
- name: spark-cluster-kueue-queued-test-master
+ name: cluster-on-kueue-master
namespace: default
- assert:
timeout: 60s
@@ -498,55 +503,19 @@ spec:
namespace: default
involvedObject:
kind: SparkCluster
- name: spark-cluster-kueue-queued-test
+ name: cluster-on-kueue
type: Normal
reason: KueueAdmissionPending
- (contains(message || '',
'sparkcluster-spark-cluster-kueue-queued-test')): true
- - script:
- timeout: 30s
- env:
- - name: CLUSTER_QUEUE
- value: ($CLUSTER_QUEUE)
- content: |
- kubectl patch clusterqueue "$CLUSTER_QUEUE" --type json -p \
- '[{"op": "replace", "path":
"/spec/resourceGroups/0/flavors/0/resources/0/nominalQuota", "value": "4"},
- {"op": "replace", "path":
"/spec/resourceGroups/0/flavors/0/resources/1/nominalQuota", "value": "8Gi"}]'
- # The node labels and tolerations of the assigned ResourceFlavor are
injected into the master
- # and the worker pods.
- - assert:
- timeout: 5m
- resource:
- apiVersion: apps/v1
- kind: StatefulSet
- metadata:
- name: spark-cluster-kueue-queued-test-master
- namespace: default
- spec:
- template:
- spec:
- nodeSelector:
- kubernetes.io/os: linux
- (tolerations[?key == 'spark.apache.org/kueue-e2e'] |
length(@)): 1
- - assert:
- timeout: 5m
- resource:
- apiVersion: apps/v1
- kind: StatefulSet
- metadata:
- name: spark-cluster-kueue-queued-test-worker
- namespace: default
- spec:
- template:
- spec:
- nodeSelector:
- kubernetes.io/os: linux
- (tolerations[?key == 'spark.apache.org/kueue-e2e'] |
length(@)): 1
+ (contains(message || '', 'sparkcluster-cluster-on-kueue')): true
+ # Applying the queues again restores the quota of the example
+ - apply:
+ file: ../../../examples/kueue-single-clusterqueue-setup.yaml
- assert:
bindings:
- name: SPARK_APP_NAMESPACE
value: default
- name: SPARK_CLUSTER_NAME
- value: spark-cluster-kueue-queued-test
+ value: cluster-on-kueue
timeout: 10m
file: "../assertions/spark-cluster/spark-cluster-state-transition.yaml"
- assert:
@@ -554,7 +523,7 @@ spec:
- name: SPARK_APP_NAMESPACE
value: default
- name: SPARK_CLUSTER_NAME
- value: spark-cluster-kueue-queued-test
+ value: cluster-on-kueue
timeout: 10m
file:
"../assertions/spark-cluster/spark-cluster-worker-statefulset.yaml"
- assert:
@@ -566,16 +535,16 @@ spec:
namespace: default
involvedObject:
kind: SparkCluster
- name: spark-cluster-kueue-queued-test
+ name: cluster-on-kueue
type: Normal
reason: KueueAdmitted
- (contains(message || '',
'sparkcluster-spark-cluster-kueue-queued-test')): true
+ (contains(message || '', 'sparkcluster-cluster-on-kueue')): true
# The ValidatingAdmissionPolicy of the chart keeps the running cluster in
its queue, since its
# Workload holds the quota until the cluster is Suspended
- script:
timeout: 30s
content: |
- kubectl label sparkcluster spark-cluster-kueue-queued-test -n
default kueue.x-k8s.io/queue-name-
+ kubectl label sparkcluster cluster-on-kueue -n default
kueue.x-k8s.io/queue-name-
check:
($error != null): true
(contains($stderr, 'ValidatingAdmissionPolicy')): true
@@ -583,14 +552,14 @@ spec:
- script:
timeout: 30s
content: |
- kubectl patch sparkcluster spark-cluster-kueue-queued-test -n
default --type merge -p '{"spec":{"suspend":true}}'
+ kubectl patch sparkcluster cluster-on-kueue -n default --type merge
-p '{"spec":{"suspend":true}}'
- assert:
timeout: 60s
resource:
apiVersion: spark.apache.org/v1
kind: SparkCluster
metadata:
- name: spark-cluster-kueue-queued-test
+ name: cluster-on-kueue
namespace: default
status:
currentState:
@@ -603,8 +572,8 @@ spec:
value: ($LOCAL_QUEUE)
content: |
set -e
- kubectl label sparkcluster spark-cluster-kueue-queued-test -n
default --overwrite kueue.x-k8s.io/queue-name=another-queue
- kubectl label sparkcluster spark-cluster-kueue-queued-test -n
default --overwrite "kueue.x-k8s.io/queue-name=$LOCAL_QUEUE"
+ kubectl label sparkcluster cluster-on-kueue -n default --overwrite
kueue.x-k8s.io/queue-name=another-queue
+ kubectl label sparkcluster cluster-on-kueue -n default --overwrite
"kueue.x-k8s.io/queue-name=$LOCAL_QUEUE"
- error:
timeout: 120s
resource:
@@ -613,14 +582,14 @@ spec:
metadata:
namespace: default
labels:
- spark.operator/spark-cluster-name:
spark-cluster-kueue-queued-test
+ spark.operator/spark-cluster-name: cluster-on-kueue
- error:
timeout: 120s
resource:
apiVersion: kueue.x-k8s.io/v1beta2
kind: Workload
metadata:
- name: sparkcluster-spark-cluster-kueue-queued-test
+ name: sparkcluster-cluster-on-kueue
namespace: default
- assert:
timeout: 60s
@@ -636,14 +605,14 @@ spec:
- script:
timeout: 30s
content: |
- kubectl patch sparkcluster spark-cluster-kueue-queued-test -n
default --type merge -p '{"spec":{"suspend":false}}'
+ kubectl patch sparkcluster cluster-on-kueue -n default --type merge
-p '{"spec":{"suspend":false}}'
- assert:
timeout: 5m
resource:
apiVersion: spark.apache.org/v1
kind: SparkCluster
metadata:
- name: spark-cluster-kueue-queued-test
+ name: cluster-on-kueue
namespace: default
status:
currentState:
@@ -662,7 +631,7 @@ spec:
- name: SPARK_APP_NAMESPACE
value: default
- name: SPARK_CLUSTER_NAME
- value: spark-cluster-kueue-queued-test
+ value: cluster-on-kueue
timeout: 5m
file:
"../assertions/spark-cluster/spark-cluster-worker-statefulset.yaml"
catch:
@@ -687,7 +656,7 @@ spec:
- name: LOCAL_QUEUE
value: ($LOCAL_QUEUE)
content: |
- kubectl delete sparkcluster spark-cluster-kueue-queued-test -n
default --ignore-not-found=true
+ kubectl delete sparkcluster cluster-on-kueue -n default
--ignore-not-found=true
# Kueue finalizers may delay the deletion, which later steps do not
depend on.
kubectl delete localqueue "$LOCAL_QUEUE" -n default
--ignore-not-found=true --wait=false
kubectl delete clusterqueue "$CLUSTER_QUEUE" --ignore-not-found=true
--wait=false
@@ -877,6 +846,36 @@ spec:
namespace: default
status:
(conditions[?type == 'Admitted' && status == 'True'] | length(@)):
1
+ # The node labels and tolerations of the assigned ResourceFlavor are
injected into the master
+ # and the worker pods.
+ - assert:
+ timeout: 5m
+ resource:
+ apiVersion: apps/v1
+ kind: StatefulSet
+ metadata:
+ name: spark-cluster-kueue-pods-ready-test-master
+ namespace: default
+ spec:
+ template:
+ spec:
+ nodeSelector:
+ kubernetes.io/os: linux
+ (tolerations[?key == 'spark.apache.org/kueue-e2e'] |
length(@)): 1
+ - assert:
+ timeout: 5m
+ resource:
+ apiVersion: apps/v1
+ kind: StatefulSet
+ metadata:
+ name: spark-cluster-kueue-pods-ready-test-worker
+ namespace: default
+ spec:
+ template:
+ spec:
+ nodeSelector:
+ kubernetes.io/os: linux
+ (tolerations[?key == 'spark.apache.org/kueue-e2e'] |
length(@)): 1
# Admitted after the cluster above, so that the waitForPodsReady timeout
of Kueue, which the CI
# shortens to 2 minutes, has elapsed for both clusters once this one is
evicted.
- apply:
diff --git a/tests/e2e/kueue/spark-example-retained-queued.yaml
b/tests/e2e/kueue/spark-example-retained-queued.yaml
index b281e62d..80998717 100644
--- a/tests/e2e/kueue/spark-example-retained-queued.yaml
+++ b/tests/e2e/kueue/spark-example-retained-queued.yaml
@@ -25,6 +25,8 @@ metadata:
spec:
mainClass: "org.apache.spark.examples.SparkPi"
jars: "local:///opt/spark/examples/jars/spark-examples.jar"
+ # Runs long enough for the test to check the executor pod before it is
deleted
+ driverArgs: ["10000"]
applicationTolerations:
# The resources, including the Workload, are retained after the
application terminated, so
# Kueue has to release the quota on the `Finished` condition of the
Workload.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
