This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new b187764 [SPARK-49715] Add `Java 21`-based `SparkCluster` example
b187764 is described below
commit b18776458b8f26db8e43f6ddb9275f3cbdbafffe
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Sep 18 22:30:11 2024 -0700
[SPARK-49715] Add `Java 21`-based `SparkCluster` example
### What changes were proposed in this pull request?
This PR aims to add `Java 21`-based `SparkCluster` example.
### Why are the changes needed?
Apache Spark starts to publish Java 21-based image from Today (2024-09-19).
This new example will illustrate how to use it.
- https://github.com/apache/spark-docker/pull/69
### Does this PR introduce _any_ user-facing change?
No, this is a new example.
### How was this patch tested?
Manual review.
```
$ kubectl apply -f cluster-java21.yaml
```
```
$ kubectl get sparkcluster
NAME CURRENT STATE AGE
cluster-java21 RunningHealthy 9s
```
```
$ kubectl describe sparkcluster cluster-java21
Name: cluster-java21
Namespace: default
Labels: <none>
Annotations: <none>
API Version: spark.apache.org/v1alpha1
Kind: SparkCluster
Metadata:
Creation Timestamp: 2024-09-19T04:25:30Z
Finalizers:
sparkclusters.spark.apache.org/finalizer
Generation: 2
Resource Version: 96663
UID: 9421c957-380d-4a26-a266-379304bf83ee
Spec:
Cluster Tolerations:
Instance Config:
Init Workers: 3
Max Workers: 3
Min Workers: 3
Master Spec:
Runtime Versions:
Spark Version: 4.0.0-preview1
Spark Conf:
spark.kubernetes.container.image: apache/spark:4.0.0-preview1-java21
spark.master.rest.enabled: true
spark.master.rest.host: 0.0.0.0
spark.master.ui.title: Prod Spark Cluster (Java 21)
spark.ui.reverseProxy: true
Worker Spec:
Status:
Current Attempt Summary:
Attempt Info:
Id: 0
Current State:
Current State Summary: RunningHealthy
Last Transition Time: 2024-09-19T04:25:30.665095088Z
Message: Cluster has reached ready state.
State Transition History:
0:
Current State Summary: Submitted
Last Transition Time: 2024-09-19T04:25:30.640072963Z
Message: Spark cluster has been submitted to
Kubernetes Cluster.
1:
Current State Summary: RunningHealthy
Last Transition Time: 2024-09-19T04:25:30.665095088Z
Message: Cluster has reached ready state.
Events: <none>
```
```
$ k get pod
NAME READY STATUS RESTARTS
AGE
cluster-java21-master-0 1/1 Running 0
3m20s
cluster-java21-worker-0 1/1 Running 0
3m20s
cluster-java21-worker-1 1/1 Running 0
3m20s
cluster-java21-worker-2 1/1 Running 0
3m20s
spark-kubernetes-operator-778b9bbdc6-fqks9 1/1 Running 0
20m
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #130 from dongjoon-hyun/SPARK-49715.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/cluster-java21.yaml | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/examples/cluster-java21.yaml b/examples/cluster-java21.yaml
new file mode 100644
index 0000000..fefe90c
--- /dev/null
+++ b/examples/cluster-java21.yaml
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: spark.apache.org/v1alpha1
+kind: SparkCluster
+metadata:
+ name: cluster-java21
+spec:
+ runtimeVersions:
+ sparkVersion: "4.0.0-preview1"
+ clusterTolerations:
+ instanceConfig:
+ initWorkers: 3
+ minWorkers: 3
+ maxWorkers: 3
+ sparkConf:
+ spark.kubernetes.container.image: "apache/spark:4.0.0-preview1-java21"
+ spark.master.ui.title: "Prod Spark Cluster (Java 21)"
+ spark.master.rest.enabled: "true"
+ spark.master.rest.host: "0.0.0.0"
+ spark.ui.reverseProxy: "true"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
