This is an automated email from the ASF dual-hosted git repository.
SteNicholas pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 3aa25932fa [CELEBORN-2383] Support Spark 4.2
3aa25932fa is described below
commit 3aa25932fad50a2193a5f1958f49a82a8da22598
Author: Nicholas Jiang <[email protected]>
AuthorDate: Wed Jul 15 13:37:16 2026 +0800
[CELEBORN-2383] Support Spark 4.2
### What changes were proposed in this pull request?
Support Spark 4.2.
Add a Spark 4.2 build target for the Celeborn Spark client, pinning Spark
4.2.0 on Scala 2.13.18:
- Add the `spark-4.2` Maven profile (`pom.xml`, `tests/spark-it/pom.xml`)
and the `Spark42` sbt project (`project/CelebornBuild.scala`), and register
Scala 2.13.18 in `ALL_SCALA_VERSIONS`.
- Add the `dev/deps/dependencies-client-spark-4.2` dependency manifest.
- Wire `spark-4.2` into `build/make-distribution.sh`, `dev/reformat`, and
the deps/maven/sbt CI matrices, and bump the release deploy profile from
`spark-4.1` to `spark-4.2`.
- Document Spark 4.2 compatibility in `README.md` and
`docs/developers/sbt.md`.
### Why are the changes needed?
Spark 4.2.0 has already released, which release notes refer to [Spark
Release 4.2.0](https://spark.apache.org/releases/spark-release-4-2-0.html).
### Does this PR resolve a correctness bug?
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
- [ ] Yes
### How was this patch tested?
CI.
Closes #3760 from SteNicholas/CELEBORN-2383.
Authored-by: Nicholas Jiang <[email protected]>
Signed-off-by: Nicholas Jiang <[email protected]>
---
.github/workflows/deps.yml | 1 +
.github/workflows/maven.yml | 1 +
.github/workflows/sbt.yml | 3 ++
README.md | 3 +-
build/make-distribution.sh | 2 +
build/release/release.sh | 2 +-
dev/deps/dependencies-client-spark-4.2 | 83 ++++++++++++++++++++++++++++++++++
dev/reformat | 1 +
docs/developers/sbt.md | 1 +
pom.xml | 20 ++++++++
project/CelebornBuild.scala | 22 ++++++++-
tests/spark-it/pom.xml | 18 ++++++++
12 files changed, 154 insertions(+), 3 deletions(-)
diff --git a/.github/workflows/deps.yml b/.github/workflows/deps.yml
index 075eda28e7..9451fc7c5d 100644
--- a/.github/workflows/deps.yml
+++ b/.github/workflows/deps.yml
@@ -125,6 +125,7 @@ jobs:
module:
- 'spark-4.0'
- 'spark-4.1'
+ - 'spark-4.2'
steps:
- uses: actions/checkout@v4
- name: Setup JDK 17
diff --git a/.github/workflows/maven.yml b/.github/workflows/maven.yml
index 8cedf8c968..e7fd0a42ec 100644
--- a/.github/workflows/maven.yml
+++ b/.github/workflows/maven.yml
@@ -126,6 +126,7 @@ jobs:
spark:
- '4.0'
- '4.1'
+ - '4.2'
shuffle-plugin-class:
- 'org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO'
steps:
diff --git a/.github/workflows/sbt.yml b/.github/workflows/sbt.yml
index ec58b11f10..8d9c61c993 100644
--- a/.github/workflows/sbt.yml
+++ b/.github/workflows/sbt.yml
@@ -177,6 +177,7 @@ jobs:
spark:
- '4.0'
- '4.1'
+ - '4.2'
java:
- 17
scala-binary:
@@ -188,6 +189,8 @@ jobs:
scala: '2.13.16'
- spark: '4.1'
scala: '2.13.17'
+ - spark: '4.2'
+ scala: '2.13.18'
steps:
- uses: actions/checkout@v4
- name: Setup JDK ${{ matrix.java }}
diff --git a/README.md b/README.md
index cf349404d6..0d5fcfaf6a 100644
--- a/README.md
+++ b/README.md
@@ -46,7 +46,7 @@ Celeborn worker's slot count decreases when a partition is
allocated and increme
Build Celeborn via `make-distribution.sh`:
```shell
-./build/make-distribution.sh
-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pspark-3.4/-Pspark-3.5/-Pspark-4.0/-Pspark-4.1/-Pflink-1.18/-Pflink-1.19/-Pflink-1.20/-Pflink-2.0/-Pflink-2.1/-Pflink-2.2/-Pflink-2.3/-Pmr
+./build/make-distribution.sh
-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pspark-3.4/-Pspark-3.5/-Pspark-4.0/-Pspark-4.1/-Pspark-4.2/-Pflink-1.18/-Pflink-1.19/-Pflink-1.20/-Pflink-2.0/-Pflink-2.1/-Pflink-2.2/-Pflink-2.3/-Pmr
```
Package `apache-celeborn-${project.version}-bin.tgz` will be generated.
@@ -63,6 +63,7 @@ Package `apache-celeborn-${project.version}-bin.tgz` will be
generated.
| Spark 3.5 | ❌ | ✔ | ✔ |
✔ | ✔ | ✔ | ✔
|
| Spark 4.0 | ❌ | ❌ | ❌ |
❌ | ❌ | ❌ | ✔
|
| Spark 4.1 | ❌ | ❌ | ❌ |
❌ | ❌ | ❌ | ✔
|
+| Spark 4.2 | ❌ | ❌ | ❌ |
❌ | ❌ | ❌ | ✔
|
| Flink 1.18 | ❌ | ✔ | ✔ |
❌ | ❌ | ❌ | ❌
|
| Flink 1.19 | ❌ | ✔ | ✔ |
❌ | ❌ | ❌ | ❌
|
| Flink 1.20 | ❌ | ✔ | ✔ |
❌ | ❌ | ❌ | ❌
|
diff --git a/build/make-distribution.sh b/build/make-distribution.sh
index 5817f7247b..2930346a13 100755
--- a/build/make-distribution.sh
+++ b/build/make-distribution.sh
@@ -379,6 +379,7 @@ if [ "$SBT_ENABLED" == "true" ]; then
export JAVA_HOME=$JAVA17_HOME
sbt_build_client -Pspark-4.0
sbt_build_client -Pspark-4.1
+ sbt_build_client -Pspark-4.2
export JAVA_HOME=$JAVA8_HOME
sbt_build_client -Pflink-1.18
sbt_build_client -Pflink-1.19
@@ -421,6 +422,7 @@ else
export JAVA_HOME=$JAVA17_HOME
build_spark_client -Pspark-4.0
build_spark_client -Pspark-4.1
+ build_spark_client -Pspark-4.2
export JAVA_HOME=$JAVA8_HOME
build_flink_client -Pflink-1.18
build_flink_client -Pflink-1.19
diff --git a/build/release/release.sh b/build/release/release.sh
index 5aff151567..7826dfd94b 100755
--- a/build/release/release.sh
+++ b/build/release/release.sh
@@ -112,7 +112,7 @@ upload_nexus_staging() {
export JAVA_HOME=$JAVA17_HOME
echo "Deploying celeborn-client-spark-4-shaded_2.13"
- ${PROJECT_DIR}/build/sbt -Pspark-4.1
"clean;celeborn-client-spark-4-shaded/publishSigned"
+ ${PROJECT_DIR}/build/sbt -Pspark-4.2
"clean;celeborn-client-spark-4-shaded/publishSigned"
export JAVA_HOME=$JAVA8_HOME
echo "Deploying celeborn-client-flink-1.18-shaded_2.12"
diff --git a/dev/deps/dependencies-client-spark-4.2
b/dev/deps/dependencies-client-spark-4.2
new file mode 100644
index 0000000000..2b62a29fed
--- /dev/null
+++ b/dev/deps/dependencies-client-spark-4.2
@@ -0,0 +1,83 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+RoaringBitmap/1.0.6//RoaringBitmap-1.0.6.jar
+commons-crypto/1.0.0//commons-crypto-1.0.0.jar
+commons-io/2.17.0//commons-io-2.17.0.jar
+commons-lang3/3.17.0//commons-lang3-3.17.0.jar
+commons-logging/1.1.3//commons-logging-1.1.3.jar
+failureaccess/1.0.2//failureaccess-1.0.2.jar
+guava/33.1.0-jre//guava-33.1.0-jre.jar
+hadoop-client-api/3.3.6//hadoop-client-api-3.3.6.jar
+hadoop-client-runtime/3.3.6//hadoop-client-runtime-3.3.6.jar
+jackson-annotations/2.15.3//jackson-annotations-2.15.3.jar
+jackson-core/2.15.3//jackson-core-2.15.3.jar
+jackson-databind/2.15.3//jackson-databind-2.15.3.jar
+jackson-module-scala_2.13/2.15.3//jackson-module-scala_2.13-2.15.3.jar
+jcl-over-slf4j/1.7.36//jcl-over-slf4j-1.7.36.jar
+jsr305/1.3.9//jsr305-1.3.9.jar
+jul-to-slf4j/1.7.36//jul-to-slf4j-1.7.36.jar
+leveldbjni-all/1.8//leveldbjni-all-1.8.jar
+lz4-java/1.11.0//lz4-java-1.11.0.jar
+metrics-core/4.2.25//metrics-core-4.2.25.jar
+metrics-graphite/4.2.25//metrics-graphite-4.2.25.jar
+metrics-jvm/4.2.25//metrics-jvm-4.2.25.jar
+netty-all/4.2.10.Final//netty-all-4.2.10.Final.jar
+netty-buffer/4.2.10.Final//netty-buffer-4.2.10.Final.jar
+netty-codec-base/4.2.10.Final//netty-codec-base-4.2.10.Final.jar
+netty-codec-classes-quic/4.2.10.Final//netty-codec-classes-quic-4.2.10.Final.jar
+netty-codec-compression/4.2.10.Final//netty-codec-compression-4.2.10.Final.jar
+netty-codec-dns/4.2.10.Final//netty-codec-dns-4.2.10.Final.jar
+netty-codec-http/4.2.10.Final//netty-codec-http-4.2.10.Final.jar
+netty-codec-http2/4.2.10.Final//netty-codec-http2-4.2.10.Final.jar
+netty-codec-http3/4.2.10.Final//netty-codec-http3-4.2.10.Final.jar
+netty-codec-marshalling/4.2.10.Final//netty-codec-marshalling-4.2.10.Final.jar
+netty-codec-native-quic/4.2.10.Final/linux-aarch_64/netty-codec-native-quic-4.2.10.Final-linux-aarch_64.jar
+netty-codec-native-quic/4.2.10.Final/linux-x86_64/netty-codec-native-quic-4.2.10.Final-linux-x86_64.jar
+netty-codec-native-quic/4.2.10.Final/osx-aarch_64/netty-codec-native-quic-4.2.10.Final-osx-aarch_64.jar
+netty-codec-native-quic/4.2.10.Final/osx-x86_64/netty-codec-native-quic-4.2.10.Final-osx-x86_64.jar
+netty-codec-native-quic/4.2.10.Final/windows-x86_64/netty-codec-native-quic-4.2.10.Final-windows-x86_64.jar
+netty-codec-protobuf/4.2.10.Final//netty-codec-protobuf-4.2.10.Final.jar
+netty-codec-socks/4.2.10.Final//netty-codec-socks-4.2.10.Final.jar
+netty-codec/4.2.10.Final//netty-codec-4.2.10.Final.jar
+netty-common/4.2.10.Final//netty-common-4.2.10.Final.jar
+netty-handler-proxy/4.2.10.Final//netty-handler-proxy-4.2.10.Final.jar
+netty-handler/4.2.10.Final//netty-handler-4.2.10.Final.jar
+netty-resolver-dns/4.2.10.Final//netty-resolver-dns-4.2.10.Final.jar
+netty-resolver/4.2.10.Final//netty-resolver-4.2.10.Final.jar
+netty-transport-classes-epoll/4.2.10.Final//netty-transport-classes-epoll-4.2.10.Final.jar
+netty-transport-classes-io_uring/4.2.10.Final//netty-transport-classes-io_uring-4.2.10.Final.jar
+netty-transport-classes-kqueue/4.2.10.Final//netty-transport-classes-kqueue-4.2.10.Final.jar
+netty-transport-native-epoll/4.2.10.Final/linux-aarch_64/netty-transport-native-epoll-4.2.10.Final-linux-aarch_64.jar
+netty-transport-native-epoll/4.2.10.Final/linux-riscv64/netty-transport-native-epoll-4.2.10.Final-linux-riscv64.jar
+netty-transport-native-epoll/4.2.10.Final/linux-x86_64/netty-transport-native-epoll-4.2.10.Final-linux-x86_64.jar
+netty-transport-native-io_uring/4.2.10.Final/linux-aarch_64/netty-transport-native-io_uring-4.2.10.Final-linux-aarch_64.jar
+netty-transport-native-io_uring/4.2.10.Final/linux-riscv64/netty-transport-native-io_uring-4.2.10.Final-linux-riscv64.jar
+netty-transport-native-io_uring/4.2.10.Final/linux-x86_64/netty-transport-native-io_uring-4.2.10.Final-linux-x86_64.jar
+netty-transport-native-kqueue/4.2.10.Final/osx-aarch_64/netty-transport-native-kqueue-4.2.10.Final-osx-aarch_64.jar
+netty-transport-native-kqueue/4.2.10.Final/osx-x86_64/netty-transport-native-kqueue-4.2.10.Final-osx-x86_64.jar
+netty-transport-native-unix-common/4.2.10.Final//netty-transport-native-unix-common-4.2.10.Final.jar
+netty-transport/4.2.10.Final//netty-transport-4.2.10.Final.jar
+paranamer/2.8.3//paranamer-2.8.3.jar
+paranamer/2.8//paranamer-2.8.jar
+protobuf-java/3.25.5//protobuf-java-3.25.5.jar
+scala-library/2.13.18//scala-library-2.13.18.jar
+scala-reflect/2.13.18//scala-reflect-2.13.18.jar
+slf4j-api/1.7.36//slf4j-api-1.7.36.jar
+snakeyaml/2.2//snakeyaml-2.2.jar
+snappy-java/1.1.10.5//snappy-java-1.1.10.5.jar
+zstd-jni/1.5.7-7//zstd-jni-1.5.7-7.jar
diff --git a/dev/reformat b/dev/reformat
index 53aee8d347..d89a97130f 100755
--- a/dev/reformat
+++ b/dev/reformat
@@ -34,6 +34,7 @@ else
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-3.5
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-4.0
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-4.1
+ ${PROJECT_DIR}/build/mvn spotless:apply -Pspark-4.2
${PROJECT_DIR}/build/mvn spotless:apply -Paws
${PROJECT_DIR}/build/mvn spotless:apply -Paliyun
${PROJECT_DIR}/build/mvn spotless:apply -Pmr
diff --git a/docs/developers/sbt.md b/docs/developers/sbt.md
index 1c0c77c10d..1529a55918 100644
--- a/docs/developers/sbt.md
+++ b/docs/developers/sbt.md
@@ -36,6 +36,7 @@ The following table indicates the compatibility of Celeborn
Spark and Flink clie
| Spark 3.5 | ❌ | ✔ | ✔ |
✔ | ✔ | ✔ | ✔
|
| Spark 4.0 | ❌ | ❌ | ❌ |
❌ | ❌ | ❌ | ✔
|
| Spark 4.1 | ❌ | ❌ | ❌ |
❌ | ❌ | ❌ | ✔
|
+| Spark 4.2 | ❌ | ❌ | ❌ |
❌ | ❌ | ❌ | ✔
|
| Flink 1.18 | ❌ | ✔ | ✔ |
❌ | ❌ | ❌ | ❌
|
| Flink 1.19 | ❌ | ✔ | ✔ |
❌ | ❌ | ❌ | ❌
|
| Flink 1.20 | ❌ | ✔ | ✔ |
❌ | ❌ | ❌ | ❌
|
diff --git a/pom.xml b/pom.xml
index f8738df863..b7577e991d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1640,6 +1640,26 @@
</properties>
</profile>
+ <profile>
+ <id>spark-4.2</id>
+ <modules>
+ <module>client-spark/common</module>
+ <module>client-spark/spark-3</module>
+ <module>client-spark/spark-3-columnar-common</module>
+ <module>client-spark/spark-4-columnar-shuffle</module>
+ <module>client-spark/spark-4-shaded</module>
+ <module>tests/spark-it</module>
+ </modules>
+ <properties>
+ <lz4-java.group>at.yawk.lz4</lz4-java.group>
+ <lz4-java.version>1.11.0</lz4-java.version>
+ <scala.version>2.13.18</scala.version>
+ <scala.binary.version>2.13</scala.binary.version>
+ <spark.version>4.2.0</spark.version>
+ <zstd-jni.version>1.5.7-7</zstd-jni.version>
+ </properties>
+ </profile>
+
<profile>
<id>jdk-8</id>
<activation>
diff --git a/project/CelebornBuild.scala b/project/CelebornBuild.scala
index 6d497868ce..2d9bb62c67 100644
--- a/project/CelebornBuild.scala
+++ b/project/CelebornBuild.scala
@@ -319,7 +319,8 @@ object CelebornCommonSettings {
val SCALA_2_13_8 = "2.13.8"
val SCALA_2_13_16 = "2.13.16"
val SCALA_2_13_17 = "2.13.17"
- val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15,
SCALA_2_12_17, SCALA_2_12_18, SCALA_2_13_5, SCALA_2_13_8, SCALA_2_13_16,
SCALA_2_13_17)
+ val SCALA_2_13_18 = "2.13.18"
+ val ALL_SCALA_VERSIONS = Seq(SCALA_2_11_12, SCALA_2_12_10, SCALA_2_12_15,
SCALA_2_12_17, SCALA_2_12_18, SCALA_2_13_5, SCALA_2_13_8, SCALA_2_13_16,
SCALA_2_13_17, SCALA_2_13_18)
val DEFAULT_SCALA_VERSION = SCALA_2_12_18
@@ -526,6 +527,7 @@ object Utils {
case Some("spark-3.5") => Some(Spark35)
case Some("spark-4.0") => Some(Spark40)
case Some("spark-4.1") => Some(Spark41)
+ case Some("spark-4.2") => Some(Spark42)
case _ => None
}
@@ -1004,6 +1006,24 @@ object Spark41 extends SparkClientProjects {
override val sparkColumnarShuffleVersion: String = "4"
}
+object Spark42 extends SparkClientProjects {
+
+ val sparkClientProjectPath = "client-spark/spark-3"
+ val sparkClientProjectName = "celeborn-client-spark-4"
+ val sparkClientShadedProjectPath = "client-spark/spark-4-shaded"
+ val sparkClientShadedProjectName = "celeborn-client-spark-4-shaded"
+
+ val lz4JavaVersion = "1.11.0"
+ val sparkProjectScalaVersion = "2.13.18"
+
+ val sparkVersion = "4.2.0"
+ val zstdJniVersion = "1.5.7-7"
+ val scalaBinaryVersion = "2.13"
+
+ override val lz4JavaGroup = "at.yawk.lz4"
+ override val sparkColumnarShuffleVersion: String = "4"
+}
+
trait SparkClientProjects {
val sparkClientProjectPath: String
diff --git a/tests/spark-it/pom.xml b/tests/spark-it/pom.xml
index ca9958cfeb..83c5f9ea33 100644
--- a/tests/spark-it/pom.xml
+++ b/tests/spark-it/pom.xml
@@ -333,5 +333,23 @@
</dependency>
</dependencies>
</profile>
+ <profile>
+ <id>spark-4.2</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.celeborn</groupId>
+
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.celeborn</groupId>
+
<artifactId>celeborn-client-spark-3_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+ </profile>
</profiles>
</project>
