This is an automated email from the ASF dual-hosted git repository.
ethanfeng pushed a commit to branch branch-0.4
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/branch-0.4 by this push:
new 794ca7e1f [CELEBORN-1380][FOLLOWUP] leveldbjni uses
org.openlabtesting.leveldbjni to support linux aarch64 platform for leveldb via
aarch64 profile
794ca7e1f is described below
commit 794ca7e1faa555e3829f67060776970032b15e38
Author: SteNicholas <[email protected]>
AuthorDate: Mon May 27 14:07:02 2024 +0800
[CELEBORN-1380][FOLLOWUP] leveldbjni uses org.openlabtesting.leveldbjni to
support linux aarch64 platform for leveldb via aarch64 profile
Dependency leveldbjni uses `org.openlabtesting.leveldbjni` to support linux
aarch64 platform for leveldb via `aarch64` profile.
Follow up #2476.
Celeborn worker could not start on arm arch devices if db backend is
`LevelDB`, which should support leveldbjni on the aarch64 platform.
aarch64 uses `org.openlabtesting.leveldbjni:leveldbjni-all.1.8`, and other
platforms use `org.fusesource.leveldbjni:leveldbjni-all.1.8`. Meanwhile,
because some hadoop dependencies packages are also depend on
`org.fusesource.leveldbjni:leveldbjni-all`, but hadoop merge the similar change
on trunk, details see
[HADOOP-16614](https://issues.apache.org/jira/browse/HADOOP-16614),
therefore it should exclude the dependency of `org.fusesource.leveldbjni` for
these hadoop packages related.
In addtion, `org.openlabtesting.leveldbjni` requires glibc version 3.4.21.
Otherwise, there will be the following potential runtime risks:
```
--------------- T H R E A D ---------------
Current thread (0x00007f9308001000): JavaThread "leveldb"
[_thread_in_native, id=878, stack(0x00007f9338cf0000,0x00007f93394f0000)]
siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr:
0x00007f97380d2220
```
Backport:
- https://github.com/apache/spark/pull/26636
- https://github.com/apache/spark/pull/31036
No.
No.
Closes #2530 from SteNicholas/CELEBORN-1380.
Authored-by: SteNicholas <[email protected]>
Signed-off-by: mingji <[email protected]>
(cherry picked from commit dd87419044c8a855abd65f7a237f9f8e0d71a657)
Signed-off-by: mingji <[email protected]>
---
LICENSE-binary | 2 +-
README.md | 10 ++++++++++
common/pom.xml | 2 +-
docs/deploy_on_k8s.md | 2 ++
pom.xml | 20 +++++++++++++++++++-
project/CelebornBuild.scala | 46 ++++++++++++++++++++++++++-------------------
worker/pom.xml | 2 +-
7 files changed, 61 insertions(+), 23 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 605a5df49..945016f5b 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -287,7 +287,7 @@ BSD 3-clause
See licenses/LICENSE-protobuf.txt for details.
com.google.protobuf:protobuf-java
See licenses/LICENSE-leveldbjni.txt for details.
-org.openlabtesting.leveldbjni:leveldbjni-all
+org.fusesource.leveldbjni:leveldbjni-all
Common Development and Distribution License (CDDL) 1.0
diff --git a/README.md b/README.md
index ca40effad..26a4c4daa 100644
--- a/README.md
+++ b/README.md
@@ -78,6 +78,16 @@ To compile the client for Spark 2.4 with Scala 2.12, please
use the following co
./build/make-distribution.sh -DskipTests -Pspark-2.4
-Dscala.version=${scala.version} -Dscala.binary.version=2.12
```
+To compile for Spark 3.5 with Java21, please use the following command
+```shell
+./build/make-distribution.sh -Pspark-3.5 -Pjdk-21
+```
+```shell
+./build/make-distribution.sh --sbt-enabled -Pspark-3.5 -Pjdk-21
+```
+
+> **_NOTE:_** Celeborn supports automatic builds on linux aarch64 platform via
`aarch64` profile. `aarch64` profile requires glibc version 3.4.21. There is
potential problematic frame `C [libc.so.6+0x8412a]` for other glibc version
like 2.x etc.
+
### Package Details
Build procedure will create a compressed package.
diff --git a/common/pom.xml b/common/pom.xml
index 936fe4053..7f5e1faed 100644
--- a/common/pom.xml
+++ b/common/pom.xml
@@ -87,7 +87,7 @@
<artifactId>netty-all</artifactId>
</dependency>
<dependency>
- <groupId>org.openlabtesting.leveldbjni</groupId>
+ <groupId>${leveldbjni.group}</groupId>
<artifactId>leveldbjni-all</artifactId>
</dependency>
<dependency>
diff --git a/docs/deploy_on_k8s.md b/docs/deploy_on_k8s.md
index 298979c5c..0d5948bd0 100644
--- a/docs/deploy_on_k8s.md
+++ b/docs/deploy_on_k8s.md
@@ -38,6 +38,8 @@ You can find released version of Celeborn on
https://celeborn.apache.org/downloa
Of course, you can build binary package from master branch or your own branch
by using `./build/make-distribution.sh` in
source code.
+> Notice: Celeborn supports automatic builds on linux aarch64 platform via
`aarch64` profile. `aarch64` profile requires glibc version 3.4.21. There is
potential problematic frame `C [libc.so.6+0x8412a]` for other glibc version
like 2.x etc.
+
Anyway, you should unzip and into binary package.
### 2. Modify Celeborn Configurations
diff --git a/pom.xml b/pom.xml
index 443d93ac1..4e1bfdf0e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -99,6 +99,9 @@
<jackson.version>2.15.3</jackson.version>
<snappy.version>1.1.10.5</snappy.version>
+ <!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
+ <leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
+
<shading.prefix>org.apache.celeborn.shaded</shading.prefix>
<maven.plugin.antrun.version>3.0.0</maven.plugin.antrun.version>
@@ -385,7 +388,7 @@
</exclusions>
</dependency>
<dependency>
- <groupId>org.openlabtesting.leveldbjni</groupId>
+ <groupId>${leveldbjni.group}</groupId>
<artifactId>leveldbjni-all</artifactId>
<version>${leveldb.version}</version>
</dependency>
@@ -1476,5 +1479,20 @@
<module>tests/kubernetes-it</module>
</modules>
</profile>
+
+ <!-- use org.openlabtesting.leveldbjni on aarch64 platform except MacOS -->
+ <profile>
+ <id>aarch64</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>aarch64</arch>
+ </os>
+ </activation>
+ <properties>
+ <!-- org.openlabtesting.leveldbjni requires glibc version 3.4.21 -->
+ <leveldbjni.group>org.openlabtesting.leveldbjni</leveldbjni.group>
+ </properties>
+ </profile>
</profiles>
</project>
diff --git a/project/CelebornBuild.scala b/project/CelebornBuild.scala
index 80b84b13b..767f917df 100644
--- a/project/CelebornBuild.scala
+++ b/project/CelebornBuild.scala
@@ -69,7 +69,7 @@ object Dependencies {
// Versions for proto
val protocVersion = "3.21.7"
val protoVersion = "3.21.7"
-
+
val commonsCompress = "org.apache.commons" % "commons-compress" %
commonsCompressVersion
val commonsCrypto = "org.apache.commons" % "commons-crypto" %
commonsCryptoVersion excludeAll(
ExclusionRule("net.java.dev.jna", "jna"))
@@ -100,7 +100,15 @@ object Dependencies {
val ioNetty = "io.netty" % "netty-all" % nettyVersion excludeAll(
ExclusionRule("io.netty", "netty-handler-ssl-ocsp"))
val javaxServletApi = "javax.servlet" % "javax.servlet-api" %
javaxServletVersion
- val leveldbJniAll = "org.openlabtesting.leveldbjni" % "leveldbjni-all" %
leveldbJniVersion
+ val leveldbJniGroup = if (System.getProperty("os.name").startsWith("Linux")
+ && System.getProperty("os.arch").equals("aarch64")) {
+ // use org.openlabtesting.leveldbjni on aarch64 platform except MacOS
+ // org.openlabtesting.leveldbjni requires glibc version 3.4.21
+ "org.openlabtesting.leveldbjni"
+ } else {
+ "org.fusesource.leveldbjni"
+ }
+ val leveldbJniAll = leveldbJniGroup % "leveldbjni-all" % leveldbJniVersion
val log4j12Api = "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version
val log4jSlf4jImpl = "org.apache.logging.log4j" % "log4j-slf4j-impl" %
log4j2Version
val lz4Java = "org.lz4" % "lz4-java" % lz4JavaVersion
@@ -154,7 +162,7 @@ object CelebornCommonSettings {
scalaVersion := projectScalaVersion
autoScalaLibrary := false
-
+
// crossScalaVersions must be set to Nil on the root project
crossScalaVersions := Nil
@@ -165,7 +173,7 @@ object CelebornCommonSettings {
fork := true,
scalacOptions ++= Seq("-target:jvm-1.8"),
javacOptions ++= Seq("-encoding", UTF_8.name(), "-source", "1.8", "-g"),
-
+
// -target cannot be passed as a parameter to javadoc. See
https://github.com/sbt/sbt/issues/355
Compile / compile / javacOptions ++= Seq("-target", "1.8"),
@@ -196,7 +204,7 @@ object CelebornCommonSettings {
"--add-opens=java.base/sun.util.calendar=ALL-UNNAMED",
"-Dio.netty.tryReflectionSetAccessible=true"
),
-
+
testOptions += Tests.Argument("-oF"),
Test / testOptions += Tests.Argument("-oDF"),
@@ -285,7 +293,7 @@ object CelebornBuild extends sbt.internal.BuildDef {
CelebornWorker.worker,
CelebornMaster.master) ++ maybeSparkClientModules ++
maybeFlinkClientModules ++ maybeMRClientModules
}
-
+
// ThisBuild / parallelExecution := false
// scalaVersion := "2.11.12"
@@ -421,7 +429,7 @@ object CelebornCommon {
Seq(file)
// generate version task depends on PB generate to avoid concurrency
generate source files
}.dependsOn(Compile / PB.generate),
-
+
// a task to show current profiles
printProfiles := {
val message = profiles.mkString("", " ", "")
@@ -680,7 +688,7 @@ trait SparkClientProjects {
) ++ commonUnitTestDependencies
)
}
-
+
def sparkClient: Project = {
Project(sparkClientProjectName, file(sparkClientProjectPath))
.dependsOn(CelebornCommon.common, sparkCommon)
@@ -745,14 +753,14 @@ trait SparkClientProjects {
val extension = artifact.value.extension
s"${moduleName.value}_${scalaBinaryVersion.value}-${version.value}.$extension"
},
-
+
(assembly / test) := { },
-
+
(assembly / logLevel) := Level.Info,
-
+
// Exclude `scala-library` from assembly.
(assembly / assemblyPackageScala / assembleArtifact) := false,
-
+
(assembly / assemblyExcludedJars) := {
val cp = (assembly / fullClasspath).value
cp filter { v =>
@@ -766,7 +774,7 @@ trait SparkClientProjects {
name.startsWith("RoaringBitmap-"))
}
},
-
+
(assembly / assemblyShadeRules) := Seq(
ShadeRule.rename("com.google.protobuf.**" ->
"org.apache.celeborn.shaded.com.google.protobuf.@1").inAll,
ShadeRule.rename("com.google.common.**" ->
"org.apache.celeborn.shaded.com.google.common.@1").inAll,
@@ -774,7 +782,7 @@ trait SparkClientProjects {
ShadeRule.rename("org.apache.commons.**" ->
"org.apache.celeborn.shaded.org.apache.commons.@1").inAll,
ShadeRule.rename("org.roaringbitmap.**" ->
"org.apache.celeborn.shaded.org.roaringbitmap.@1").inAll
),
-
+
(assembly / assemblyMergeStrategy) := {
case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf") =>
MergeStrategy.discard
// the LicenseAndNoticeMergeStrategy always picks the license/notice
file from the current project
@@ -948,12 +956,12 @@ trait FlinkClientProjects {
val artifactValue: Artifact = artifact.value
flinkClientShadeJarName(revision, artifactValue,
scalaBinaryVersion.value)
},
-
+
(assembly / logLevel) := Level.Info,
-
+
// Exclude `scala-library` from assembly.
(assembly / assemblyPackageScala / assembleArtifact) := false,
-
+
(assembly / assemblyExcludedJars) := {
val cp = (assembly / fullClasspath).value
cp filter { v =>
@@ -967,7 +975,7 @@ trait FlinkClientProjects {
name.startsWith("RoaringBitmap-"))
}
},
-
+
(assembly / assemblyShadeRules) := Seq(
ShadeRule.rename("com.google.protobuf.**" ->
"org.apache.celeborn.shaded.com.google.protobuf.@1").inAll,
ShadeRule.rename("com.google.common.**" ->
"org.apache.celeborn.shaded.com.google.common.@1").inAll,
@@ -975,7 +983,7 @@ trait FlinkClientProjects {
ShadeRule.rename("org.apache.commons.**" ->
"org.apache.celeborn.shaded.org.apache.commons.@1").inAll,
ShadeRule.rename("org.roaringbitmap.**" ->
"org.apache.celeborn.shaded.org.roaringbitmap.@1").inAll
),
-
+
(assembly / assemblyMergeStrategy) := {
case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf") =>
MergeStrategy.discard
// the LicenseAndNoticeMergeStrategy always picks the license/notice
file from the current project
diff --git a/worker/pom.xml b/worker/pom.xml
index 8d3620de6..91bedf103 100644
--- a/worker/pom.xml
+++ b/worker/pom.xml
@@ -49,7 +49,7 @@
<version>${project.version}</version>
</dependency>
<dependency>
- <groupId>org.openlabtesting.leveldbjni</groupId>
+ <groupId>${leveldbjni.group}</groupId>
<artifactId>leveldbjni-all</artifactId>
</dependency>
<dependency>
