This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 45e8549 [SPARK-55550] Add `Apache Celeborn` example
45e8549 is described below
commit 45e8549b61e288c37727bd60ec2890a33a1d63e5
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Feb 16 20:56:08 2026 -0800
[SPARK-55550] Add `Apache Celeborn` example
### What changes were proposed in this pull request?
This PR aims to add `Apache Celeborn` example for `Driver`-pod only Spark
jobs.
### Why are the changes needed?
To provide `Apache Celeborn` example via `Apache Spark K8s Operator v0.8.0`.
### Does this PR introduce _any_ user-facing change?
No behavior change.
### How was this patch tested?
Manual review.
**1. Run K8s Operator**
```bash
$ helm install spark spark/spark-kubernetes-operator
```
**2. Install Apache Celeborn v0.6.2 (One master + One worker)**
```bash
$ git clone https://github.com/apache/celeborn.git
$ cd celeborn/charts/celeborn
$ helm install celeborn . --set image.tag=0.6.2 --set master.replicas=1
--set master.antiAffinity=false --set worker.replicas=1 --set
worker.affinity.podAntiAffinity=null
```
**3. Run this PR example and check log**
```bash
$ kubectl apply -f examples/word-count-celeborn.yaml
$ kubectl logs -f word-count-celeborn-0-driver | grep SparkContext
26/02/16 07:01:23 INFO SparkContext: Running Spark version 4.0.2
26/02/16 07:01:23 INFO SparkContext: OS info Linux, 6.6.119-0-virt, aarch64
26/02/16 07:01:23 INFO SparkContext: Java version 17.0.17
26/02/16 07:01:23 INFO SparkContext: Submitted application: JavaWordCount
26/02/16 07:01:23 INFO SparkContext: Added JAR
local:/opt/spark/examples/jars/spark-examples.jar at
file:/opt/spark/examples/jars/spark-examples.jar with timestamp 1771225283452
26/02/16 07:01:23 INFO SparkContext: Added JAR
file:/tmp/.ivy2.5.2/jars/org.apache.celeborn_celeborn-client-spark-4-shaded_2.13-0.6.2.jar
at
spark://word-count-celeborn-0-driver-svc.default.svc:7078/jars/org.apache.celeborn_celeborn-client-spark-4-shaded_2.13-0.6.2.jar
with timestamp 1771225283452
26/02/16 07:01:25 INFO SparkContext: Created broadcast 0 from javaRDD at
JavaWordCount.java:45
26/02/16 07:01:25 INFO SparkContext: Starting job: collect at
JavaWordCount.java:53
26/02/16 07:01:25 INFO SparkContext: Created broadcast 1 from broadcast at
DAGScheduler.scala:1676
26/02/16 07:01:26 INFO SparkContext: Created broadcast 2 from broadcast at
DAGScheduler.scala:1676
26/02/16 07:01:26 INFO SparkContext: SparkContext is stopping with exitCode
0 from stop at JavaWordCount.java:57.
26/02/16 07:01:26 INFO SparkContext: Successfully stopped SparkContext
```
**4. Check `Celeborn` log**
```
$ kubectl logs celeborn-master-0 | grep word-count
Defaulted container "celeborn" out of: celeborn,
chown-celeborn-master-volume (init)
26/02/16 14:51:34,447 INFO [celeborn-dispatcher-3] Master: Successfully
offered slots for 1 reducers of word-count-celeborn-0-0 on 1 workers, offered 3
extra slots.
26/02/16 14:51:57,305 INFO [master-noneager-handler-0] Master: Removed
application word-count-celeborn-0
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #507 from dongjoon-hyun/SPARK-55550.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/word-count-celeborn.yaml | 41 +++++++++++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/examples/word-count-celeborn.yaml
b/examples/word-count-celeborn.yaml
new file mode 100644
index 0000000..86e9956
--- /dev/null
+++ b/examples/word-count-celeborn.yaml
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: spark.apache.org/v1
+kind: SparkApplication
+metadata:
+ name: word-count-celeborn
+spec:
+ mainClass: "org.apache.spark.examples.JavaWordCount"
+ driverArgs: [ "/opt/spark/RELEASE" ]
+ jars: "local:///opt/spark/examples/jars/spark-examples.jar"
+ sparkConf:
+ spark.celeborn.master.endpoints:
"celeborn-master-0.celeborn-master-svc:9097"
+ spark.jars.ivy: "/tmp/.ivy2.5.2"
+ spark.jars.packages:
"org.apache.celeborn:celeborn-client-spark-4-shaded_2.13:0.6.2"
+ spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
+ spark.kubernetes.container.image: "apache/spark:4.0.2-scala"
+ spark.kubernetes.driver.limit.cores: "5"
+ spark.kubernetes.driver.master: "local[10]"
+ spark.kubernetes.driver.pod.excludedFeatureSteps:
"org.apache.spark.deploy.k8s.features.KerberosConfDriverFeatureStep"
+ spark.kubernetes.driver.request.cores: "5"
+ spark.shuffle.manager:
"org.apache.spark.shuffle.celeborn.SparkShuffleManager"
+ spark.shuffle.service.enabled: "false"
+ spark.sql.adaptive.localShuffleReader.enabled: "false"
+ spark.serializer: "org.apache.spark.serializer.KryoSerializer"
+ applicationTolerations:
+ resourceRetainPolicy: OnFailure
+ ttlAfterStopMillis: 10000
+ runtimeVersions:
+ sparkVersion: "4.0.2"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
