This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a48e2e824641 [SPARK-54636][BUILD][YARN] Correctly relocate Netty
native libs for YARN ESS
a48e2e824641 is described below
commit a48e2e824641062db4b19492ed8093b9c6706e55
Author: Cheng Pan <[email protected]>
AuthorDate: Tue Dec 9 13:31:23 2025 +0800
[SPARK-54636][BUILD][YARN] Correctly relocate Netty native libs for YARN ESS
### What changes were proposed in this pull request?
Spark upgrade to Netty 4.2 since SPARK-53849, which brings several
additional native libs, YARN ESS should correctly relocate them as it did for
other existing Netty native libs.
### Why are the changes needed?
Fix potential classpath conflicts issues for YARN ESS.
### Does this PR introduce _any_ user-facing change?
No, Spark upgrades to Netty 4.2 in 4.1.0, which has not been released yet.
### How was this patch tested?
Spark 4.0.1
```
$ jar tf spark-4.0.1-yarn-shuffle.jar | grep META-INF/native/ | grep netty
META-INF/native/liborg_sparkproject_netty_tcnative_linux_aarch_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_linux_x86_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_osx_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_tcnative_osx_x86_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_riscv64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_x86_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
META-INF/native/netty_tcnative_windows_x86_64.dll
```
master branch
```
$ jar tf spark-4.2.0-SNAPSHOT-yarn-shuffle.jar | grep META-INF/native/ |
grep netty
META-INF/native/libnetty_quiche42_linux_aarch_64.so
META-INF/native/libnetty_quiche42_linux_x86_64.so
META-INF/native/libnetty_quiche42_osx_aarch_64.jnilib
META-INF/native/libnetty_quiche42_osx_x86_64.jnilib
META-INF/native/libnetty_transport_native_io_uring42_aarch_64.so
META-INF/native/libnetty_transport_native_io_uring42_riscv64.so
META-INF/native/libnetty_transport_native_io_uring42_x86_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_linux_aarch_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_linux_x86_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_osx_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_tcnative_osx_x86_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_riscv64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_x86_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
META-INF/native/netty_quiche42_windows_x86_64.dll
META-INF/native/netty_tcnative_windows_x86_64.dll
```
this PR
```
$ jar tf spark-4.2.0-SNAPSHOT-yarn-shuffle.jar | grep META-INF/native/ |
grep netty
META-INF/native/liborg_sparkproject_netty_quiche42_linux_aarch_64.so
META-INF/native/liborg_sparkproject_netty_quiche42_linux_x86_64.so
META-INF/native/liborg_sparkproject_netty_quiche42_osx_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_quiche42_osx_x86_64.jnilib
META-INF/native/liborg_sparkproject_netty_tcnative_linux_aarch_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_linux_x86_64.so
META-INF/native/liborg_sparkproject_netty_tcnative_osx_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_tcnative_osx_x86_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_riscv64.so
META-INF/native/liborg_sparkproject_netty_transport_native_epoll_x86_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_io_uring42_aarch_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_io_uring42_riscv64.so
META-INF/native/liborg_sparkproject_netty_transport_native_io_uring42_x86_64.so
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib
META-INF/native/liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
META-INF/native/netty_quiche42_windows_x86_64.dll
META-INF/native/netty_tcnative_windows_x86_64.dll
```
Manually tested with Hadoop YARN (v3.4.2), with ESS enabled.
<img width="1585" height="983" alt="image"
src="https://github.com/user-attachments/assets/e7cfc0c9-05e5-4480-bd35-29946d1c17e9";
/>
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #53382 from pan3793/SPARK-54636.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
common/network-yarn/pom.xml | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 8d7cb2eb2109..7ff63f8430d6 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -171,18 +171,25 @@
<phase>package</phase>
<configuration>
<target>
+ <!-- Follow
https://github.com/netty/netty/blob/netty-4.2.7.Final/testsuite-shading/pom.xml
to skip processing MS Windows DLLs -->
<echo message="Shade netty native libraries to
${spark.shade.native.packageName}" />
<unzip src="${shuffle.jar}"
dest="${project.build.directory}/exploded/" />
<move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_x86_64.so"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so"
/>
+ <move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_io_uring42_x86_64.so"
+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_io_uring42_x86_64.so"
/>
<move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_x86_64.jnilib"
/>
<move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_aarch_64.so"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_aarch_64.so"
/>
+ <move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_io_uring42_aarch_64.so"
+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_io_uring42_aarch_64.so"
/>
<move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib"
/>
<move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_riscv64.so"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_riscv64.so"
/>
+ <move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_io_uring42_riscv64.so"
+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_io_uring42_riscv64.so"
/>
<move
file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_linux_x86_64.so"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_linux_x86_64.so"
/>
<move
file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_linux_aarch_64.so"
@@ -191,6 +198,14 @@
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_osx_x86_64.jnilib"
/>
<move
file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_osx_aarch_64.jnilib"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_osx_aarch_64.jnilib"
/>
+ <move
file="${project.build.directory}/exploded/META-INF/native/libnetty_quiche42_linux_x86_64.so"
+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_quiche42_linux_x86_64.so"
/>
+ <move
file="${project.build.directory}/exploded/META-INF/native/libnetty_quiche42_linux_aarch_64.so"
+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_quiche42_linux_aarch_64.so"
/>
+ <move
file="${project.build.directory}/exploded/META-INF/native/libnetty_quiche42_osx_x86_64.jnilib"
+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_quiche42_osx_x86_64.jnilib"
/>
+ <move
file="${project.build.directory}/exploded/META-INF/native/libnetty_quiche42_osx_aarch_64.jnilib"
+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_quiche42_osx_aarch_64.jnilib"
/>
<jar destfile="${shuffle.jar}"
basedir="${project.build.directory}/exploded" />
</target>
</configuration>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
