This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 45e3ccd [SPARK-58840] Update `word-count-celeborn` example to use
Celeborn 0.7.0 and Spark 4.2.0
45e3ccd is described below
commit 45e3ccdd2e0da5d38af98a9bc8eca09258343896
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Aug 18 06:38:23 2026 -0700
[SPARK-58840] Update `word-count-celeborn` example to use Celeborn 0.7.0
and Spark 4.2.0
### What changes were proposed in this pull request?
This PR aims to update `word-count-celeborn` example to use Celeborn 0.7.0
and Spark 4.2.0.
### Why are the changes needed?
To provide an example with the latest versions:
- [Apache Celeborn
v0.7.0](https://github.com/apache/celeborn/releases/tag/v0.7.0) (2026-08-17)
In addition, the container image now uses the `{{SPARK_VERSION}}`
placeholder consistently like the other examples.
### Does this PR introduce _any_ user-facing change?
No, this is an example change.
### How was this patch tested?
Manual review.
```
$ git clone https://github.com/apache/celeborn.git
$ cd charts/celeborn
$ helm install celeborn . --set image.tag=0.7.0 --set master.replicas=1
--set master.antiAffinity=false --set worker.replicas=1 --set
worker.affinity.podAntiAffinity=null
NAME: celeborn
LAST DEPLOYED: Tue Aug 18 01:56:11 2026
NAMESPACE: default
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
TEST SUITE: None
NOTES:
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
Celeborn
```
```
$ kubectl apply -f examples/word-count-celeborn.yaml
sparkapplication.spark.apache.org/word-count-celeborn created
$ kubectl logs -f word-count-celeborn-0-driver | grep SparkContext
26/08/18 08:57:48 INFO SparkContext: Running Spark version 4.2.0
26/08/18 08:57:48 INFO SparkContext: OS info Linux, 6.18.37-0-virt, aarch64
26/08/18 08:57:48 INFO SparkContext: Java version 21.0.11+10-LTS
26/08/18 08:57:48 INFO SparkContext: Submitted application: JavaWordCount
26/08/18 08:57:48 INFO SparkContext: Use DRIVER_BIND_ADDRESS instead of
DRIVER_HOST_ADDRESS as driver address because
spark.kubernetes.executor.useDriverPodIP is true in K8s mode.
26/08/18 08:57:48 INFO SparkContext: Added JAR
file:/opt/spark/work-dir/spark-examples_2.13-4.2.0.jar at
spark://10.42.0.243:7078/jars/spark-examples_2.13-4.2.0.jar with timestamp
1787043468071
26/08/18 08:57:48 INFO SparkContext: Added JAR
file:/opt/spark/work-dir/org.apache.celeborn_celeborn-client-spark-4-shaded_2.13-0.7.0.jar
at
spark://10.42.0.243:7078/jars/org.apache.celeborn_celeborn-client-spark-4-shaded_2.13-0.7.0.jar
with timestamp 1787043468071
26/08/18 08:57:50 INFO SparkContext: Created broadcast 0 from javaRDD at
JavaWordCount.java:45
26/08/18 08:57:50 INFO SparkContext: Starting job: collect at
JavaWordCount.java:53
26/08/18 08:57:50 INFO SparkContext: Created broadcast 1 from broadcast at
DAGScheduler.scala:1760
26/08/18 08:57:51 INFO SparkContext: Created broadcast 2 from broadcast at
DAGScheduler.scala:1760
26/08/18 08:57:51 INFO SparkContext: SparkContext is stopping with exitCode
0 from stop at JavaWordCount.java:57.
26/08/18 08:57:51 INFO SparkContext: Successfully stopped SparkContext
(Uptime: 3119 ms)
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #797 from dongjoon-hyun/SPARK-58840.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/word-count-celeborn.yaml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/examples/word-count-celeborn.yaml
b/examples/word-count-celeborn.yaml
index 1a0dbd4..3d1c867 100644
--- a/examples/word-count-celeborn.yaml
+++ b/examples/word-count-celeborn.yaml
@@ -23,9 +23,9 @@ spec:
sparkConf:
spark.celeborn.master.endpoints:
"celeborn-master-0.celeborn-master-svc:9097"
spark.jars.ivy: "/tmp/.ivy2.5.2"
- spark.jars.packages:
"org.apache.celeborn:celeborn-client-spark-4-shaded_2.13:0.6.3"
+ spark.jars.packages:
"org.apache.celeborn:celeborn-client-spark-4-shaded_2.13:0.7.0"
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
- spark.kubernetes.container.image: "apache/spark:4.0.4-scala"
+ spark.kubernetes.container.image: "apache/spark:{{SPARK_VERSION}}-scala"
spark.kubernetes.driver.limit.cores: "5"
spark.kubernetes.driver.master: "local[10]"
spark.kubernetes.driver.request.cores: "5"
@@ -34,4 +34,4 @@ spec:
spark.sql.adaptive.localShuffleReader.enabled: "false"
spark.serializer: "org.apache.spark.serializer.KryoSerializer"
runtimeVersions:
- sparkVersion: "4.0.4"
+ sparkVersion: "4.2.0"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
