This is an automated email from the ASF dual-hosted git repository.
zaynt4606 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new ad5b99f248 [CELEBORN-2473] Do not relocate zstd-jni in the Tez shaded
jar
ad5b99f248 is described below
commit ad5b99f24808e89f9273e33ce79f7b9b8df30d32
Author: senthh <[email protected]>
AuthorDate: Mon Oct 12 11:34:11 2026 +0800
[CELEBORN-2473] Do not relocate zstd-jni in the Tez shaded jar
## [CELEBORN-2473] Do not relocate zstd-jni in the Tez shaded jar
**Problem:** the Maven shade plugin in `client-tez/tez-shaded/pom.xml` and
the sbt assembly rules in `project/CelebornBuild.scala` relocate
`com.github.luben` (zstd-jni) to the shading prefix. zstd-jni is a JNI library
whose bundled native `.so`/`.dll` exports symbols using the original package
names (e.g. `Java_com_github_luben_zstd_Zstd_compressBound`). After relocation
the JVM binds those symbols from the remapped class names, so the native load
fails with `UnsatisfiedLinkError`/`N [...]
**Fix:** keep zstd-jni bundled (it stays in the `artifactSet` includes) but
**stop relocating** `com.github.luben`. This matches `client-mr/mr-shaded`,
which already bundles zstd-jni without remapping it. (Spark/Flink shaded
clients don't bundle zstd-jni at all — those engines ship their own — so
`mr-shaded` is the direct precedent.)
**Files:**
- `client-tez/tez-shaded/pom.xml` (−4)
- `project/CelebornBuild.scala` (−1)
**Validation:** pom remains well-formed XML; zstd-jni still present in the
artifact set. cf. HADOOP-17891 / HBASE-23007: never relocate JNI libraries.
Closes #3865 from senthh/CELEBORN-2473.
Authored-by: senthh <[email protected]>
Signed-off-by: zhengtao <[email protected]>
AI-Contributed/Feature: 0/5
AI-Contributed/UT: 0/0
---
client-tez/tez-shaded/pom.xml | 4 ----
project/CelebornBuild.scala | 1 -
2 files changed, 5 deletions(-)
diff --git a/client-tez/tez-shaded/pom.xml b/client-tez/tez-shaded/pom.xml
index 73bb78371a..f73f9f4b7a 100644
--- a/client-tez/tez-shaded/pom.xml
+++ b/client-tez/tez-shaded/pom.xml
@@ -78,10 +78,6 @@
<pattern>com.codahale.metrics</pattern>
<shadedPattern>${shading.prefix}.com.codahale.metrics</shadedPattern>
</relocation>
- <relocation>
- <pattern>com.github.luben</pattern>
- <shadedPattern>${shading.prefix}.com.github.luben</shadedPattern>
- </relocation>
</relocations>
<artifactSet>
<includes>
diff --git a/project/CelebornBuild.scala b/project/CelebornBuild.scala
index 1a00e98626..cfaba0ec51 100644
--- a/project/CelebornBuild.scala
+++ b/project/CelebornBuild.scala
@@ -1983,7 +1983,6 @@ object TezClientProjects {
ShadeRule.rename("org.roaringbitmap.**" ->
"org.apache.celeborn.shaded.org.roaringbitmap.@1").inAll,
ShadeRule.rename("io.dropwizard.metrics.**" ->
"org.apache.celeborn.shaded.io.dropwizard.metrics.@1").inAll,
ShadeRule.rename("com.codahale.metrics.**" ->
"org.apache.celeborn.shaded.com.codahale.metrics.@1").inAll,
- ShadeRule.rename("com.github.luben.**" ->
"org.apache.celeborn.shaded.com.github.luben.@1").inAll,
),
(assembly / assemblyMergeStrategy) := {
