This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 77e8a39 [SPARK-58194] Upgrade docker images to Apache Spark 4.2.0
77e8a39 is described below
commit 77e8a3925070574234956ea4cb0dfe86480037f8
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 17 08:27:02 2026 -0700
[SPARK-58194] Upgrade docker images to Apache Spark 4.2.0
### What changes were proposed in this pull request?
This PR aims to use Apache Spark 4.2.0 docker images instead of `4.1.2` and
`4.2.0-preview5`.
- Rename `pi-preview*` examples and the `pi-preview` E2E test to
`pi-java25*` with `apache/spark:4.2.0-scala-java25` image.
- Update all examples, E2E tests, benchmark script, and docs from `4.1.2`
to `4.2.0`.
- Update the default Spark cluster image of `SparkClusterResourceSpec` from
`apache/spark:4.1.2` to `apache/spark:4.2.0`.
Note that `pi-with-comet.yaml` stays on `4.1.2` because `Apache DataFusion
Comet` does not provide a Spark 4.2 artifact yet, like `pi-with-gluten.yaml`
stays on `4.0.3`.
### Why are the changes needed?
Since Apache Spark 4.2.0 is released, the examples and tests should use the
official release images instead of the preview and old versions. The build
dependency is already on Spark 4.2.0 in `gradle/libs.versions.toml`.
### Does this PR introduce _any_ user-facing change?
Yes, the default Spark cluster image becomes `apache/spark:4.2.0`, and the
examples are updated. However, this is a consistent behavior change with the
already-upgraded Spark 4.2.0 dependency.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #761 from dongjoon-hyun/SPARK-58194.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 2 +-
README.md | 6 +++---
docs/spark_custom_resources.md | 4 ++--
examples/cluster-on-yunikorn.yaml | 2 +-
examples/cluster-with-hpa-template.yaml | 2 +-
examples/cluster-with-hpa.yaml | 2 +-
examples/cluster-with-jws-filter.yaml | 2 +-
examples/cluster-with-template.yaml | 2 +-
examples/cluster.yaml | 2 +-
examples/dfs-read-write.yaml | 2 +-
examples/pi-java17.yaml | 2 +-
.../{pi-preview-deployment.yaml => pi-java25-deployment.yaml} | 6 +++---
...ize-plugin.yaml => pi-java25-with-executor-resize-plugin.yaml} | 6 +++---
examples/{pi-preview.yaml => pi-java25.yaml} | 6 +++---
examples/pi-on-volcano.yaml | 2 +-
examples/pi-on-yunikorn.yaml | 2 +-
examples/pi-python.yaml | 2 +-
examples/pi-statefulset.yaml | 2 +-
examples/pi-v1beta1.yaml | 2 +-
examples/pi-with-driver-timeout.yaml | 2 +-
examples/pi-with-eventlog.yaml | 2 +-
examples/pi-with-instanceconfig.yaml | 2 +-
examples/pi-with-on-demand-pvc.yaml | 2 +-
examples/pi-with-one-pod.yaml | 2 +-
examples/pi-with-spark-connect-plugin.yaml | 2 +-
examples/pi-with-template.yaml | 2 +-
examples/pi.yaml | 2 +-
examples/prod-cluster-with-three-workers.yaml | 2 +-
examples/qa-cluster-with-one-worker.yaml | 2 +-
examples/spark-connect-server-usw2-az2.yaml | 2 +-
examples/spark-connect-server-with-spark-cluster.yaml | 2 +-
examples/spark-connect-server.yaml | 2 +-
examples/spark-history-server-with-jws-filter.yaml | 2 +-
examples/spark-history-server.yaml | 2 +-
examples/spark-thrift-server.yaml | 2 +-
examples/sql.yaml | 2 +-
examples/stream-word-count.yaml | 2 +-
examples/word-count.yaml | 2 +-
.../org/apache/spark/k8s/operator/SparkClusterResourceSpec.java | 2 +-
tests/benchmark/sparkapps.sh | 2 +-
.../driver-start-timeout/spark-example-driver-start-timeout.yaml | 2 +-
tests/e2e/{pi-preview => pi-java25}/chainsaw-test.yaml | 8 ++++----
tests/e2e/{pi-preview => pi-java25}/spark-state-transition.yaml | 2 +-
tests/e2e/python/chainsaw-test.yaml | 4 ++--
.../resource-retain-duration/spark-example-retain-duration.yaml | 2 +-
.../spark-example-retain-on-failure.yaml | 2 +-
tests/e2e/resource-selector/chainsaw-test.yaml | 4 ++--
tests/e2e/spark-versions/chainsaw-test.yaml | 8 ++++----
tests/e2e/state-transition/spark-cluster-example-succeeded.yaml | 2 +-
tests/e2e/state-transition/spark-example-succeeded.yaml | 2 +-
tests/e2e/watched-namespaces-file/spark-example.yaml | 2 +-
tests/e2e/watched-namespaces/spark-example.yaml | 2 +-
52 files changed, 69 insertions(+), 69 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 9f7be94..ef9ca7d 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -133,7 +133,7 @@ jobs:
test-group: pi-with-gluten
- kubernetes-version: "1.36.0"
mode: static
- test-group: pi-preview
+ test-group: pi-java25
steps:
- name: Checkout repository
uses: actions/checkout@v6
diff --git a/README.md b/README.md
index 472ed86..26415ef 100644
--- a/README.md
+++ b/README.md
@@ -96,7 +96,7 @@ $ ./examples/submit-pi-to-prod.sh
{
"action" : "CreateSubmissionResponse",
"message" : "Driver successfully submitted as driver-20260110030233-0000",
- "serverSparkVersion" : "4.1.2",
+ "serverSparkVersion" : "4.2.0",
"submissionId" : "driver-20260110030233-0000",
"success" : true
}
@@ -105,7 +105,7 @@ $ curl
http://localhost:6066/v1/submissions/status/driver-20260110030233-0000/
{
"action" : "SubmissionStatusResponse",
"driverState" : "FINISHED",
- "serverSparkVersion" : "4.1.2",
+ "serverSparkVersion" : "4.2.0",
"submissionId" : "driver-20260110030233-0000",
"success" : true,
"workerHostPort" : "10.1.1.172:44233",
@@ -142,7 +142,7 @@ Events:
Normal Scheduling 1s yunikorn default/pi-on-yunikorn-0-driver
is queued and waiting for allocation
Normal Scheduled 1s yunikorn Successfully assigned
default/pi-on-yunikorn-0-driver to node docker-desktop
Normal PodBindSuccessful 1s yunikorn Pod
default/pi-on-yunikorn-0-driver is successfully bound to node docker-desktop
- Normal Pulled 0s kubelet Container image
"apache/spark:4.1.2-scala" already present on machine
+ Normal Pulled 0s kubelet Container image
"apache/spark:4.2.0-scala" already present on machine
Normal Created 0s kubelet Created container:
spark-kubernetes-driver
Normal Started 0s kubelet Started container
spark-kubernetes-driver
diff --git a/docs/spark_custom_resources.md b/docs/spark_custom_resources.md
index c0fc94a..2472f72 100644
--- a/docs/spark_custom_resources.md
+++ b/docs/spark_custom_resources.md
@@ -52,12 +52,12 @@ spec:
spark.dynamicAllocation.shuffleTracking.enabled: "true"
spark.dynamicAllocation.maxExecutors: "3"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
- spark.kubernetes.container.image: "apache/spark:4.1.2-scala"
+ spark.kubernetes.container.image: "apache/spark:4.2.0-scala"
applicationTolerations:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
```
After application is submitted, Operator will add status information to your
application based on
diff --git a/examples/cluster-on-yunikorn.yaml
b/examples/cluster-on-yunikorn.yaml
index 4834af4..3ddd22e 100644
--- a/examples/cluster-on-yunikorn.yaml
+++ b/examples/cluster-on-yunikorn.yaml
@@ -18,7 +18,7 @@ metadata:
name: cluster-on-yunikorn
spec:
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
clusterTolerations:
instanceConfig:
initWorkers: 1
diff --git a/examples/cluster-with-hpa-template.yaml
b/examples/cluster-with-hpa-template.yaml
index 2c53e93..4a4ce64 100644
--- a/examples/cluster-with-hpa-template.yaml
+++ b/examples/cluster-with-hpa-template.yaml
@@ -18,7 +18,7 @@ metadata:
name: cluster-with-hpa-template
spec:
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
clusterTolerations:
instanceConfig:
initWorkers: 1
diff --git a/examples/cluster-with-hpa.yaml b/examples/cluster-with-hpa.yaml
index 5982514..7170a4e 100644
--- a/examples/cluster-with-hpa.yaml
+++ b/examples/cluster-with-hpa.yaml
@@ -18,7 +18,7 @@ metadata:
name: cluster-with-hpa
spec:
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
clusterTolerations:
instanceConfig:
initWorkers: 3
diff --git a/examples/cluster-with-jws-filter.yaml
b/examples/cluster-with-jws-filter.yaml
index 9a265ff..685ceb6 100644
--- a/examples/cluster-with-jws-filter.yaml
+++ b/examples/cluster-with-jws-filter.yaml
@@ -18,7 +18,7 @@ metadata:
name: cluster-with-jws-filter
spec:
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
clusterTolerations:
instanceConfig:
initWorkers: 3
diff --git a/examples/cluster-with-template.yaml
b/examples/cluster-with-template.yaml
index 79c4deb..cb454fe 100644
--- a/examples/cluster-with-template.yaml
+++ b/examples/cluster-with-template.yaml
@@ -18,7 +18,7 @@ metadata:
name: cluster-with-template
spec:
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
clusterTolerations:
instanceConfig:
initWorkers: 1
diff --git a/examples/cluster.yaml b/examples/cluster.yaml
index cf9d89f..f887057 100644
--- a/examples/cluster.yaml
+++ b/examples/cluster.yaml
@@ -18,7 +18,7 @@ metadata:
name: cluster
spec:
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
clusterTolerations:
instanceConfig:
initWorkers: 3
diff --git a/examples/dfs-read-write.yaml b/examples/dfs-read-write.yaml
index 7147888..08f7d64 100644
--- a/examples/dfs-read-write.yaml
+++ b/examples/dfs-read-write.yaml
@@ -41,4 +41,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-java17.yaml b/examples/pi-java17.yaml
index c1d1013..ed451a3 100644
--- a/examples/pi-java17.yaml
+++ b/examples/pi-java17.yaml
@@ -28,4 +28,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-preview-deployment.yaml
b/examples/pi-java25-deployment.yaml
similarity index 90%
rename from examples/pi-preview-deployment.yaml
rename to examples/pi-java25-deployment.yaml
index 19e2db1..5e55a11 100644
--- a/examples/pi-preview-deployment.yaml
+++ b/examples/pi-java25-deployment.yaml
@@ -15,7 +15,7 @@
apiVersion: spark.apache.org/v1
kind: SparkApplication
metadata:
- name: pi-preview-deployment
+ name: pi-java25-deployment
spec:
mainClass: "org.apache.spark.examples.SparkPi"
driverArgs: ["20000"]
@@ -24,9 +24,9 @@ spec:
spark.executor.instances: "3"
spark.kubernetes.allocation.pods.allocator: "deployment"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
- spark.kubernetes.container.image: "apache/spark:4.2.0-preview5-scala"
+ spark.kubernetes.container.image: "apache/spark:4.2.0-scala-java25"
applicationTolerations:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.2.0-preview5"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-preview-with-executor-resize-plugin.yaml
b/examples/pi-java25-with-executor-resize-plugin.yaml
similarity index 90%
rename from examples/pi-preview-with-executor-resize-plugin.yaml
rename to examples/pi-java25-with-executor-resize-plugin.yaml
index efb302a..f6e8ecd 100644
--- a/examples/pi-preview-with-executor-resize-plugin.yaml
+++ b/examples/pi-java25-with-executor-resize-plugin.yaml
@@ -15,7 +15,7 @@
apiVersion: spark.apache.org/v1
kind: SparkApplication
metadata:
- name: pi-preview-with-executor-resize-plugin
+ name: pi-java25-with-executor-resize-plugin
spec:
mainClass: "org.apache.spark.examples.SparkPi"
driverArgs: ["200000"]
@@ -25,7 +25,7 @@ spec:
spark.dynamicAllocation.maxExecutors: "1"
spark.dynamicAllocation.shuffleTracking.enabled: "true"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
- spark.kubernetes.container.image: "apache/spark:4.2.0-preview5-scala"
+ spark.kubernetes.container.image: "apache/spark:4.2.0-scala-java25"
spark.kubernetes.executor.resizeInterval: "1min"
spark.kubernetes.executor.resizeThreshold: "0.25"
spark.plugins:
"org.apache.spark.scheduler.cluster.k8s.ExecutorResizePlugin"
@@ -33,4 +33,4 @@ spec:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.2.0-preview5"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-preview.yaml b/examples/pi-java25.yaml
similarity index 91%
rename from examples/pi-preview.yaml
rename to examples/pi-java25.yaml
index 7a70277..8fe6723 100644
--- a/examples/pi-preview.yaml
+++ b/examples/pi-java25.yaml
@@ -15,7 +15,7 @@
apiVersion: spark.apache.org/v1
kind: SparkApplication
metadata:
- name: pi-preview
+ name: pi-java25
spec:
mainClass: "org.apache.spark.examples.SparkPi"
jars: "local:///opt/spark/examples/jars/spark-examples.jar"
@@ -24,9 +24,9 @@ spec:
spark.dynamicAllocation.shuffleTracking.enabled: "true"
spark.dynamicAllocation.maxExecutors: "3"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
- spark.kubernetes.container.image: "apache/spark:4.2.0-preview5-scala"
+ spark.kubernetes.container.image: "apache/spark:4.2.0-scala-java25"
applicationTolerations:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.2.0-preview5"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-on-volcano.yaml b/examples/pi-on-volcano.yaml
index 6455360..551a13b 100644
--- a/examples/pi-on-volcano.yaml
+++ b/examples/pi-on-volcano.yaml
@@ -33,4 +33,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-on-yunikorn.yaml b/examples/pi-on-yunikorn.yaml
index 93ffc07..103c712 100644
--- a/examples/pi-on-yunikorn.yaml
+++ b/examples/pi-on-yunikorn.yaml
@@ -34,4 +34,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-python.yaml b/examples/pi-python.yaml
index 45643c5..16a6b49 100644
--- a/examples/pi-python.yaml
+++ b/examples/pi-python.yaml
@@ -27,4 +27,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-statefulset.yaml b/examples/pi-statefulset.yaml
index 62ab5e2..856c7f6 100644
--- a/examples/pi-statefulset.yaml
+++ b/examples/pi-statefulset.yaml
@@ -28,4 +28,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-v1beta1.yaml b/examples/pi-v1beta1.yaml
index 067d7c0..73d17e4 100644
--- a/examples/pi-v1beta1.yaml
+++ b/examples/pi-v1beta1.yaml
@@ -28,4 +28,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-with-driver-timeout.yaml
b/examples/pi-with-driver-timeout.yaml
index 57723d0..a9f02df 100644
--- a/examples/pi-with-driver-timeout.yaml
+++ b/examples/pi-with-driver-timeout.yaml
@@ -32,4 +32,4 @@ spec:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-with-eventlog.yaml b/examples/pi-with-eventlog.yaml
index d50cb02..9d3226c 100644
--- a/examples/pi-with-eventlog.yaml
+++ b/examples/pi-with-eventlog.yaml
@@ -40,4 +40,4 @@ spec:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-with-instanceconfig.yaml
b/examples/pi-with-instanceconfig.yaml
index 2f84f7f..5723921 100644
--- a/examples/pi-with-instanceconfig.yaml
+++ b/examples/pi-with-instanceconfig.yaml
@@ -31,4 +31,4 @@ spec:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-with-on-demand-pvc.yaml
b/examples/pi-with-on-demand-pvc.yaml
index f0e11f2..a33e0b2 100644
--- a/examples/pi-with-on-demand-pvc.yaml
+++ b/examples/pi-with-on-demand-pvc.yaml
@@ -35,4 +35,4 @@ spec:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-with-one-pod.yaml b/examples/pi-with-one-pod.yaml
index 6395131..e1afb48 100644
--- a/examples/pi-with-one-pod.yaml
+++ b/examples/pi-with-one-pod.yaml
@@ -26,4 +26,4 @@ spec:
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-with-spark-connect-plugin.yaml
b/examples/pi-with-spark-connect-plugin.yaml
index c3ae4ac..a39b1c2 100644
--- a/examples/pi-with-spark-connect-plugin.yaml
+++ b/examples/pi-with-spark-connect-plugin.yaml
@@ -30,4 +30,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi-with-template.yaml b/examples/pi-with-template.yaml
index eb6a9af..ef8d33a 100644
--- a/examples/pi-with-template.yaml
+++ b/examples/pi-with-template.yaml
@@ -38,4 +38,4 @@ spec:
priorityClassName: system-cluster-critical
terminationGracePeriodSeconds: 0
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/pi.yaml b/examples/pi.yaml
index 50280b8..eeee12a 100644
--- a/examples/pi.yaml
+++ b/examples/pi.yaml
@@ -29,4 +29,4 @@ spec:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/prod-cluster-with-three-workers.yaml
b/examples/prod-cluster-with-three-workers.yaml
index a5c8b53..4b2b74c 100644
--- a/examples/prod-cluster-with-three-workers.yaml
+++ b/examples/prod-cluster-with-three-workers.yaml
@@ -18,7 +18,7 @@ metadata:
name: prod
spec:
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
clusterTolerations:
instanceConfig:
initWorkers: 3
diff --git a/examples/qa-cluster-with-one-worker.yaml
b/examples/qa-cluster-with-one-worker.yaml
index b337e03..2882653 100644
--- a/examples/qa-cluster-with-one-worker.yaml
+++ b/examples/qa-cluster-with-one-worker.yaml
@@ -18,7 +18,7 @@ metadata:
name: qa
spec:
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
clusterTolerations:
instanceConfig:
initWorkers: 1
diff --git a/examples/spark-connect-server-usw2-az2.yaml
b/examples/spark-connect-server-usw2-az2.yaml
index 208944f..9716098 100644
--- a/examples/spark-connect-server-usw2-az2.yaml
+++ b/examples/spark-connect-server-usw2-az2.yaml
@@ -41,4 +41,4 @@ spec:
resourceRetainPolicy: OnFailure
ttlAfterStopMillis: 10000
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/spark-connect-server-with-spark-cluster.yaml
b/examples/spark-connect-server-with-spark-cluster.yaml
index 9f6655e..5f19b70 100644
--- a/examples/spark-connect-server-with-spark-cluster.yaml
+++ b/examples/spark-connect-server-with-spark-cluster.yaml
@@ -26,4 +26,4 @@ spec:
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
spark.ui.reverseProxy: "true"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/spark-connect-server.yaml
b/examples/spark-connect-server.yaml
index c0761c3..48da081 100644
--- a/examples/spark-connect-server.yaml
+++ b/examples/spark-connect-server.yaml
@@ -30,4 +30,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/spark-history-server-with-jws-filter.yaml
b/examples/spark-history-server-with-jws-filter.yaml
index c8a5f0d..8c612c7 100644
--- a/examples/spark-history-server-with-jws-filter.yaml
+++ b/examples/spark-history-server-with-jws-filter.yaml
@@ -39,7 +39,7 @@ spec:
spark.ui.filters: "org.apache.spark.ui.JWSFilter"
spark.org.apache.spark.ui.JWSFilter.param.secretKey:
"VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4="
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
applicationTolerations:
restartConfig:
restartPolicy: Always
diff --git a/examples/spark-history-server.yaml
b/examples/spark-history-server.yaml
index ab2e3b8..0ad14fc 100644
--- a/examples/spark-history-server.yaml
+++ b/examples/spark-history-server.yaml
@@ -36,7 +36,7 @@ spec:
spark.hadoop.fs.s3a.access.key: "test"
spark.hadoop.fs.s3a.secret.key: "test"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
applicationTolerations:
restartConfig:
restartPolicy: Always
diff --git a/examples/spark-thrift-server.yaml
b/examples/spark-thrift-server.yaml
index 36312f8..6fc05e0 100644
--- a/examples/spark-thrift-server.yaml
+++ b/examples/spark-thrift-server.yaml
@@ -28,7 +28,7 @@ spec:
spark.kubernetes.executor.podNamePrefix: "spark-thrift-server"
spark.scheduler.mode: "FAIR"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
applicationTolerations:
restartConfig:
restartPolicy: Always
diff --git a/examples/sql.yaml b/examples/sql.yaml
index dcfc84b..0eb05ea 100644
--- a/examples/sql.yaml
+++ b/examples/sql.yaml
@@ -27,4 +27,4 @@ spec:
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/stream-word-count.yaml b/examples/stream-word-count.yaml
index 3d833b8..8e368b5 100644
--- a/examples/stream-word-count.yaml
+++ b/examples/stream-word-count.yaml
@@ -37,4 +37,4 @@ spec:
spark.hadoop.fs.s3a.access.key: "test"
spark.hadoop.fs.s3a.secret.key: "test"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/examples/word-count.yaml b/examples/word-count.yaml
index 5a1b73c..8072bfe 100644
--- a/examples/word-count.yaml
+++ b/examples/word-count.yaml
@@ -28,4 +28,4 @@ spec:
applicationTolerations:
resourceRetainPolicy: OnFailure
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git
a/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
b/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
index 9debb38..8cadf00 100644
---
a/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
+++
b/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java
@@ -69,7 +69,7 @@ public class SparkClusterResourceSpec {
String clusterName = cluster.getMetadata().getName();
String scheduler = conf.get(Config.KUBERNETES_SCHEDULER_NAME().key(),
"default-scheduler");
String namespace = conf.get(Config.KUBERNETES_NAMESPACE().key(),
clusterNamespace);
- String image = conf.get(Config.CONTAINER_IMAGE().key(),
"apache/spark:4.1.2");
+ String image = conf.get(Config.CONTAINER_IMAGE().key(),
"apache/spark:4.2.0");
ClusterSpec spec = cluster.getSpec();
String version = spec.getRuntimeVersions().getSparkVersion();
StringBuilder options = new StringBuilder();
diff --git a/tests/benchmark/sparkapps.sh b/tests/benchmark/sparkapps.sh
index 8c85e71..53b2273 100755
--- a/tests/benchmark/sparkapps.sh
+++ b/tests/benchmark/sparkapps.sh
@@ -46,7 +46,7 @@ spec:
spark.kubernetes.driver.pod.excludedFeatureSteps:
"org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
spark.kubernetes.driver.request.cores: "100m"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
---
EOF
done
diff --git
a/tests/e2e/driver-start-timeout/spark-example-driver-start-timeout.yaml
b/tests/e2e/driver-start-timeout/spark-example-driver-start-timeout.yaml
index 1c03150..9eabf4b 100644
--- a/tests/e2e/driver-start-timeout/spark-example-driver-start-timeout.yaml
+++ b/tests/e2e/driver-start-timeout/spark-example-driver-start-timeout.yaml
@@ -31,4 +31,4 @@ spec:
spark.kubernetes.container.image: "apache/spark:non-existent-image"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/tests/e2e/pi-preview/chainsaw-test.yaml
b/tests/e2e/pi-java25/chainsaw-test.yaml
similarity index 86%
rename from tests/e2e/pi-preview/chainsaw-test.yaml
rename to tests/e2e/pi-java25/chainsaw-test.yaml
index 8b963cf..93dcc43 100644
--- a/tests/e2e/pi-preview/chainsaw-test.yaml
+++ b/tests/e2e/pi-java25/chainsaw-test.yaml
@@ -18,13 +18,13 @@
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
- name: spark-operator-pi-preview-validation
+ name: spark-operator-pi-java25-validation
spec:
namespace: default
steps:
- try:
- apply:
- file: ../../../examples/pi-preview.yaml
+ file: ../../../examples/pi-java25.yaml
- assert:
bindings:
- name: SPARK_APP_NAMESPACE
@@ -37,7 +37,7 @@ spec:
kind: SparkApplication
namespace: default
- podLogs:
- selector: spark-app-name=pi-preview
+ selector: spark-app-name=pi-java25
namespace: default
- podLogs:
selector: app.kubernetes.io/name=spark-kubernetes-operator
@@ -46,4 +46,4 @@ spec:
- script:
timeout: 120s
content: |
- kubectl delete sparkapplication pi-preview --ignore-not-found=true
+ kubectl delete sparkapplication pi-java25 --ignore-not-found=true
diff --git a/tests/e2e/pi-preview/spark-state-transition.yaml
b/tests/e2e/pi-java25/spark-state-transition.yaml
similarity index 98%
rename from tests/e2e/pi-preview/spark-state-transition.yaml
rename to tests/e2e/pi-java25/spark-state-transition.yaml
index b7edc6f..2953a1e 100644
--- a/tests/e2e/pi-preview/spark-state-transition.yaml
+++ b/tests/e2e/pi-java25/spark-state-transition.yaml
@@ -18,7 +18,7 @@
apiVersion: spark.apache.org/v1
kind: SparkApplication
metadata:
- name: pi-preview
+ name: pi-java25
namespace: ($SPARK_APP_NAMESPACE)
status:
stateTransitionHistory:
diff --git a/tests/e2e/python/chainsaw-test.yaml
b/tests/e2e/python/chainsaw-test.yaml
index d56de30..e8be4a5 100644
--- a/tests/e2e/python/chainsaw-test.yaml
+++ b/tests/e2e/python/chainsaw-test.yaml
@@ -23,11 +23,11 @@ spec:
scenarios:
- bindings:
- name: "SPARK_VERSION"
- value: "4.1.2"
+ value: "4.2.0"
- name: "SCALA_VERSION"
value: "2.13"
- name: "IMAGE"
- value: "apache/spark:4.1.2-python3"
+ value: "apache/spark:4.2.0-python3"
steps:
- name: install-spark-application
try:
diff --git
a/tests/e2e/resource-retain-duration/spark-example-retain-duration.yaml
b/tests/e2e/resource-retain-duration/spark-example-retain-duration.yaml
index 6944c15..87fbcf0 100644
--- a/tests/e2e/resource-retain-duration/spark-example-retain-duration.yaml
+++ b/tests/e2e/resource-retain-duration/spark-example-retain-duration.yaml
@@ -32,4 +32,4 @@ spec:
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git
a/tests/e2e/resource-retain-on-failure/spark-example-retain-on-failure.yaml
b/tests/e2e/resource-retain-on-failure/spark-example-retain-on-failure.yaml
index df3fa20..23aeaa1 100644
--- a/tests/e2e/resource-retain-on-failure/spark-example-retain-on-failure.yaml
+++ b/tests/e2e/resource-retain-on-failure/spark-example-retain-on-failure.yaml
@@ -30,4 +30,4 @@ spec:
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/tests/e2e/resource-selector/chainsaw-test.yaml
b/tests/e2e/resource-selector/chainsaw-test.yaml
index adb708e..9782036 100644
--- a/tests/e2e/resource-selector/chainsaw-test.yaml
+++ b/tests/e2e/resource-selector/chainsaw-test.yaml
@@ -27,9 +27,9 @@ spec:
- name: SPARK_APPLICATION_NAME
value: "spark-example-resource-selector"
- name: SPARK_VERSION
- value: "4.1.2"
+ value: "4.2.0"
- name: IMAGE
- value: "apache/spark:4.1.2-scala"
+ value: "apache/spark:4.2.0-scala"
steps:
- name: install-spark-application-and-apply-label
try:
diff --git a/tests/e2e/spark-versions/chainsaw-test.yaml
b/tests/e2e/spark-versions/chainsaw-test.yaml
index 2971706..fc838c0 100644
--- a/tests/e2e/spark-versions/chainsaw-test.yaml
+++ b/tests/e2e/spark-versions/chainsaw-test.yaml
@@ -23,22 +23,22 @@ spec:
scenarios:
- bindings:
- name: "SPARK_VERSION"
- value: "4.1.2"
+ value: "4.2.0"
- name: "SCALA_VERSION"
value: "2.13"
- name: "JAVA_VERSION"
value: "17"
- name: "IMAGE"
- value: "apache/spark:4.1.2-scala-java17"
+ value: "apache/spark:4.2.0-scala-java17"
- bindings:
- name: "SPARK_VERSION"
- value: "4.1.2"
+ value: "4.2.0"
- name: "SCALA_VERSION"
value: "2.13"
- name: "JAVA_VERSION"
value: "21"
- name: "IMAGE"
- value: 'apache/spark:4.1.2-scala'
+ value: 'apache/spark:4.2.0-scala'
steps:
- name: install-spark-application
try:
diff --git a/tests/e2e/state-transition/spark-cluster-example-succeeded.yaml
b/tests/e2e/state-transition/spark-cluster-example-succeeded.yaml
index 6068866..3f1b07e 100644
--- a/tests/e2e/state-transition/spark-cluster-example-succeeded.yaml
+++ b/tests/e2e/state-transition/spark-cluster-example-succeeded.yaml
@@ -19,7 +19,7 @@ metadata:
namespace: default
spec:
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
clusterTolerations:
instanceConfig:
initWorkers: 1
diff --git a/tests/e2e/state-transition/spark-example-succeeded.yaml
b/tests/e2e/state-transition/spark-example-succeeded.yaml
index c0dbdec..cf45b80 100644
--- a/tests/e2e/state-transition/spark-example-succeeded.yaml
+++ b/tests/e2e/state-transition/spark-example-succeeded.yaml
@@ -28,4 +28,4 @@ spec:
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/tests/e2e/watched-namespaces-file/spark-example.yaml
b/tests/e2e/watched-namespaces-file/spark-example.yaml
index 54c798f..9cc2d93 100644
--- a/tests/e2e/watched-namespaces-file/spark-example.yaml
+++ b/tests/e2e/watched-namespaces-file/spark-example.yaml
@@ -28,4 +28,4 @@ spec:
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
diff --git a/tests/e2e/watched-namespaces/spark-example.yaml
b/tests/e2e/watched-namespaces/spark-example.yaml
index 54c798f..9cc2d93 100644
--- a/tests/e2e/watched-namespaces/spark-example.yaml
+++ b/tests/e2e/watched-namespaces/spark-example.yaml
@@ -28,4 +28,4 @@ spec:
spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
runtimeVersions:
- sparkVersion: "4.1.2"
+ sparkVersion: "4.2.0"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
