This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 63a15e9 [SPARK-55660] Add `On-Demand PVC` example
63a15e9 is described below
commit 63a15e9080523315bf21b79ed1e97fa03035bf15
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Feb 24 10:23:24 2026 -0800
[SPARK-55660] Add `On-Demand PVC` example
### What changes were proposed in this pull request?
This PR aims to add `On-Demand PVC` example.
### Why are the changes needed?
To illustrate how Apache Spark uses the PVC in the on-demand manner.
```bash
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY
VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete
WaitForFirstConsumer false 35d
$ kubectl apply -f examples/pi-with-on-demand-pvc.yaml
sparkapplication.spark.apache.org/pi-with-on-demand-pvc created
$ kubectl get sparkapp
NAME CURRENT STATE AGE
pi-with-on-demand-pvc RunningHealthy 6s
$ kubectl get pod -l spark-role
NAME READY STATUS
RESTARTS AGE
pi-with-on-demand-pvc-0-driver 1/1 Running 0
11s
spark-kubernetes-operator-67964b9b47-vql44 1/1 Running 0
6m17s
spark-pi-5d91cc9c90d4b8cf-exec-1 1/1 Running 0
7s
spark-pi-5d91cc9c90d4b8cf-exec-2 0/1 ContainerCreating 0
6s
spark-pi-5d91cc9c90d4b8cf-exec-3 1/1 Running 0
6s
$ kubectl get pvc
NAME STATUS VOLUME
CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS
AGE
spark-pi-5d91cc9c90d4b8cf-exec-1-pvc-0 Bound
pvc-a848829b-4b7e-4e55-a9b5-1d9230e67fc1 1Gi RWOP local-path
<unset> 12s
spark-pi-5d91cc9c90d4b8cf-exec-2-pvc-0 Bound
pvc-e5ef5a62-9238-4a67-84ce-d7f2ba4069aa 1Gi RWOP local-path
<unset> 11s
spark-pi-5d91cc9c90d4b8cf-exec-3-pvc-0 Bound
pvc-94db625d-5b16-4b4f-81c5-92fbad150d7e 1Gi RWOP local-path
<unset> 11s
$ kubectl get pv
NAME CAPACITY ACCESS MODES
RECLAIM POLICY STATUS CLAIM
STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-94db625d-5b16-4b4f-81c5-92fbad150d7e 1Gi RWOP Delete
Bound default/spark-pi-5d91cc9c90d4b8cf-exec-3-pvc-0 local-path
<unset> 11s
pvc-a848829b-4b7e-4e55-a9b5-1d9230e67fc1 1Gi RWOP Delete
Bound default/spark-pi-5d91cc9c90d4b8cf-exec-1-pvc-0 local-path
<unset> 12s
pvc-e5ef5a62-9238-4a67-84ce-d7f2ba4069aa 1Gi RWOP Delete
Bound default/spark-pi-5d91cc9c90d4b8cf-exec-2-pvc-0 local-path
<unset> 11s
```
### Does this PR introduce _any_ user-facing change?
No behavior change.
### How was this patch tested?
Manually tested with `Rancher Desktop`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: `Gemini 3.1 Pro (High)` on `Antigravity`
Closes #520 from dongjoon-hyun/SPARK-55660.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/pi-with-on-demand-pvc.yaml | 39 +++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/examples/pi-with-on-demand-pvc.yaml
b/examples/pi-with-on-demand-pvc.yaml
new file mode 100644
index 0000000..8cc8ae3
--- /dev/null
+++ b/examples/pi-with-on-demand-pvc.yaml
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: spark.apache.org/v1
+kind: SparkApplication
+metadata:
+ name: pi-with-on-demand-pvc
+spec:
+ mainClass: "org.apache.spark.examples.SparkPi"
+ jars: "local:///opt/spark/examples/jars/spark-examples.jar"
+ driverArgs: [ "20000" ]
+ sparkConf:
+ spark.dynamicAllocation.enabled: "true"
+ spark.dynamicAllocation.shuffleTracking.enabled: "true"
+ spark.dynamicAllocation.maxExecutors: "3"
+ spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
+ spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
+ spark.kubernetes.driver.pod.excludedFeatureSteps:
"org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
+
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path:
"/data"
+
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.readOnly:
"false"
+
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName:
"OnDemand"
+
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit:
"1Gi"
+
spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.storageClass:
"local-path"
+ applicationTolerations:
+ resourceRetainPolicy: OnFailure
+ ttlAfterStopMillis: 10000
+ runtimeVersions:
+ sparkVersion: "4.1.1"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
