dongjoon-hyun opened a new pull request, #507: URL: https://github.com/apache/spark-kubernetes-operator/pull/507
### What changes were proposed in this pull request? This PR aims to add `Apache Celeborn` example for `Driver`-pod only Spark jobs. ### Why are the changes needed? To provide `Apache Celeborn` example via `Apache Spark K8s Operator v0.8.0`. ### Does this PR introduce _any_ user-facing change? No behavior change. ### How was this patch tested? Manual review. **1. Run K8s Operator** ```bash $ helm install spark spark/spark-kubernetes-operator ``` **2. Install Apache Celeborn v0.6.2 (One master + One worker)** ```bash $ git clone https://github.com/apache/celeborn.git $ cd celeborn/charts/celeborn $ helm install celeborn . --set image.tag=0.6.2 --set master.replicas=1 --set master.antiAffinity=false --set worker.replicas=1 --set worker.affinity.podAntiAffinity=null ``` **3. Run this PR example and check log** ```bash $ kubectl apply -f examples/word-count-celeborn.yaml $ kubectl logs -f word-count-celeborn-0-driver | grep SparkContext 26/02/16 07:01:23 INFO SparkContext: Running Spark version 4.0.2 26/02/16 07:01:23 INFO SparkContext: OS info Linux, 6.6.119-0-virt, aarch64 26/02/16 07:01:23 INFO SparkContext: Java version 17.0.17 26/02/16 07:01:23 INFO SparkContext: Submitted application: JavaWordCount 26/02/16 07:01:23 INFO SparkContext: Added JAR local:/opt/spark/examples/jars/spark-examples.jar at file:/opt/spark/examples/jars/spark-examples.jar with timestamp 1771225283452 26/02/16 07:01:23 INFO SparkContext: Added JAR file:/tmp/.ivy2.5.2/jars/org.apache.celeborn_celeborn-client-spark-4-shaded_2.13-0.6.2.jar at spark://word-count-celeborn-0-driver-svc.default.svc:7078/jars/org.apache.celeborn_celeborn-client-spark-4-shaded_2.13-0.6.2.jar with timestamp 1771225283452 26/02/16 07:01:25 INFO SparkContext: Created broadcast 0 from javaRDD at JavaWordCount.java:45 26/02/16 07:01:25 INFO SparkContext: Starting job: collect at JavaWordCount.java:53 26/02/16 07:01:25 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1676 26/02/16 07:01:26 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1676 26/02/16 07:01:26 INFO SparkContext: SparkContext is stopping with exitCode 0 from stop at JavaWordCount.java:57. 26/02/16 07:01:26 INFO SparkContext: Successfully stopped SparkContext ``` **4. Check `Celeborn` log** ``` $ kubectl logs celeborn-master-0 | grep word-count Defaulted container "celeborn" out of: celeborn, chown-celeborn-master-volume (init) 26/02/16 14:51:34,447 INFO [celeborn-dispatcher-3] Master: Successfully offered slots for 1 reducers of word-count-celeborn-0-0 on 1 workers, offered 3 extra slots. 26/02/16 14:51:57,305 INFO [master-noneager-handler-0] Master: Removed application word-count-celeborn-0 ``` ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
