This is an automated email from the ASF dual-hosted git repository.
ethanfeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new dd8741904 [CELEBORN-1380][FOLLOWUP] leveldbjni uses
org.openlabtesting.leveldbjni to support linux aarch64 platform for leveldb via
aarch64 profile
dd8741904 is described below
commit dd87419044c8a855abd65f7a237f9f8e0d71a657
Author: SteNicholas <[email protected]>
AuthorDate: Mon May 27 14:07:02 2024 +0800
[CELEBORN-1380][FOLLOWUP] leveldbjni uses org.openlabtesting.leveldbjni to
support linux aarch64 platform for leveldb via aarch64 profile
### What changes were proposed in this pull request?
Dependency leveldbjni uses `org.openlabtesting.leveldbjni` to support linux
aarch64 platform for leveldb via `aarch64` profile.
Follow up #2476.
### Why are the changes needed?
Celeborn worker could not start on arm arch devices if db backend is
`LevelDB`, which should support leveldbjni on the aarch64 platform.
aarch64 uses `org.openlabtesting.leveldbjni:leveldbjni-all.1.8`, and other
platforms use `org.fusesource.leveldbjni:leveldbjni-all.1.8`. Meanwhile,
because some hadoop dependencies packages are also depend on
`org.fusesource.leveldbjni:leveldbjni-all`, but hadoop merge the similar change
on trunk, details see
[HADOOP-16614](https://issues.apache.org/jira/browse/HADOOP-16614),
therefore it should exclude the dependency of `org.fusesource.leveldbjni` for
these hadoop packages related.
In addtion, `org.openlabtesting.leveldbjni` requires glibc version 3.4.21.
Otherwise, there will be the following potential runtime risks:
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007fad3630b12a, pid=62, tid=0x00007f93394ef700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build
1.8.0_162-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode
linux-amd64 )
# Problematic frame:
# C [libc.so.6+0x8412a]
#
# Core dump written. Default location: /data/service/celeborn/core or
core.62
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- T H R E A D ---------------
Current thread (0x00007f9308001000): JavaThread "leveldb"
[_thread_in_native, id=878, stack(0x00007f9338cf0000,0x00007f93394f0000)]
siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr:
0x00007f97380d2220
```
Backport:
- https://github.com/apache/spark/pull/26636
- https://github.com/apache/spark/pull/31036
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No.
Closes #2530 from SteNicholas/CELEBORN-1380.
Authored-by: SteNicholas <[email protected]>
Signed-off-by: mingji <[email protected]>
---
LICENSE-binary | 2 +-
README.md | 1 +
common/pom.xml | 2 +-
docs/deploy_on_k8s.md | 2 ++
pom.xml | 20 +++++++++++++++++++-
project/CelebornBuild.scala | 10 +++++++++-
worker/pom.xml | 2 +-
7 files changed, 34 insertions(+), 5 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index 5a35fd16a..034141317 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -302,7 +302,7 @@ BSD 3-clause
See licenses/LICENSE-protobuf.txt for details.
com.google.protobuf:protobuf-java
See licenses/LICENSE-leveldbjni.txt for details.
-org.openlabtesting.leveldbjni:leveldbjni-all
+org.fusesource.leveldbjni:leveldbjni-all
Common Development and Distribution License (CDDL) 1.0
diff --git a/README.md b/README.md
index 525b97894..e375122d5 100644
--- a/README.md
+++ b/README.md
@@ -87,6 +87,7 @@ To compile for Spark 3.5 with Java21, please use the
following command
./build/make-distribution.sh --sbt-enabled -Pspark-3.5 -Pjdk-21
```
+> **_NOTE:_** Celeborn supports automatic builds on linux aarch64 platform via
`aarch64` profile. `aarch64` profile requires glibc version 3.4.21. There is
potential problematic frame `C [libc.so.6+0x8412a]` for other glibc version
like 2.x etc.
### Package Details
Build procedure will create a compressed package.
diff --git a/common/pom.xml b/common/pom.xml
index ab9dba6b1..3273303dd 100644
--- a/common/pom.xml
+++ b/common/pom.xml
@@ -87,7 +87,7 @@
<artifactId>netty-all</artifactId>
</dependency>
<dependency>
- <groupId>org.openlabtesting.leveldbjni</groupId>
+ <groupId>${leveldbjni.group}</groupId>
<artifactId>leveldbjni-all</artifactId>
</dependency>
<dependency>
diff --git a/docs/deploy_on_k8s.md b/docs/deploy_on_k8s.md
index 16597d6ce..3c7edcfb4 100644
--- a/docs/deploy_on_k8s.md
+++ b/docs/deploy_on_k8s.md
@@ -38,6 +38,8 @@ You can find released version of Celeborn on [Downloading
Page](https://celeborn
Of course, you can build binary package from master branch or your own branch
by using `./build/make-distribution.sh` in
source code.
+> Notice: Celeborn supports automatic builds on linux aarch64 platform via
`aarch64` profile. `aarch64` profile requires glibc version 3.4.21. There is
potential problematic frame `C [libc.so.6+0x8412a]` for other glibc version
like 2.x etc.
+
Anyway, you should unzip and into binary package.
### 2. Modify Celeborn Configurations
diff --git a/pom.xml b/pom.xml
index f9a824a2f..307b95227 100644
--- a/pom.xml
+++ b/pom.xml
@@ -113,6 +113,9 @@
<jetty.version>9.4.52.v20230823</jetty.version>
<jakarta.servlet-api.version>4.0.4</jakarta.servlet-api.version>
+ <!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
+ <leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
+
<shading.prefix>org.apache.celeborn.shaded</shading.prefix>
<maven.plugin.antrun.version>3.0.0</maven.plugin.antrun.version>
@@ -394,7 +397,7 @@
</exclusions>
</dependency>
<dependency>
- <groupId>org.openlabtesting.leveldbjni</groupId>
+ <groupId>${leveldbjni.group}</groupId>
<artifactId>leveldbjni-all</artifactId>
<version>${leveldb.version}</version>
</dependency>
@@ -1697,5 +1700,20 @@
<module>tests/kubernetes-it</module>
</modules>
</profile>
+
+ <!-- use org.openlabtesting.leveldbjni on aarch64 platform except MacOS -->
+ <profile>
+ <id>aarch64</id>
+ <activation>
+ <os>
+ <family>linux</family>
+ <arch>aarch64</arch>
+ </os>
+ </activation>
+ <properties>
+ <!-- org.openlabtesting.leveldbjni requires glibc version 3.4.21 -->
+ <leveldbjni.group>org.openlabtesting.leveldbjni</leveldbjni.group>
+ </properties>
+ </profile>
</profiles>
</project>
diff --git a/project/CelebornBuild.scala b/project/CelebornBuild.scala
index d75c6ba46..c8a61cbaa 100644
--- a/project/CelebornBuild.scala
+++ b/project/CelebornBuild.scala
@@ -111,7 +111,15 @@ object Dependencies {
val ioDropwizardMetricsJvm = "io.dropwizard.metrics" % "metrics-jvm" %
metricsVersion
val ioNetty = "io.netty" % "netty-all" % nettyVersion excludeAll(
ExclusionRule("io.netty", "netty-handler-ssl-ocsp"))
- val leveldbJniAll = "org.openlabtesting.leveldbjni" % "leveldbjni-all" %
leveldbJniVersion
+ val leveldbJniGroup = if (System.getProperty("os.name").startsWith("Linux")
+ && System.getProperty("os.arch").equals("aarch64")) {
+ // use org.openlabtesting.leveldbjni on aarch64 platform except MacOS
+ // org.openlabtesting.leveldbjni requires glibc version 3.4.21
+ "org.openlabtesting.leveldbjni"
+ } else {
+ "org.fusesource.leveldbjni"
+ }
+ val leveldbJniAll = leveldbJniGroup % "leveldbjni-all" % leveldbJniVersion
val log4j12Api = "org.apache.logging.log4j" % "log4j-1.2-api" % log4j2Version
val log4jSlf4jImpl = "org.apache.logging.log4j" % "log4j-slf4j-impl" %
log4j2Version
val lz4Java = "org.lz4" % "lz4-java" % lz4JavaVersion
diff --git a/worker/pom.xml b/worker/pom.xml
index ef32e97aa..e0be5e904 100644
--- a/worker/pom.xml
+++ b/worker/pom.xml
@@ -49,7 +49,7 @@
<version>${project.version}</version>
</dependency>
<dependency>
- <groupId>org.openlabtesting.leveldbjni</groupId>
+ <groupId>${leveldbjni.group}</groupId>
<artifactId>leveldbjni-all</artifactId>
</dependency>
<dependency>
