This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7154fb989de [SPARK-38885][BUILD][FOLLOWUP] Fix compile error on
`Appleslicon/MacOs` and ensure `./dev/test-dependencies.sh` produce the same
results on Linux and `Appleslicon/MacOs`
7154fb989de is described below
commit 7154fb989de9b0dfefb5dc8953ee63ab43dc414b
Author: yangjie01 <[email protected]>
AuthorDate: Sun Apr 24 11:02:02 2022 -0500
[SPARK-38885][BUILD][FOLLOWUP] Fix compile error on `Appleslicon/MacOs` and
ensure `./dev/test-dependencies.sh` produce the same results on Linux and
`Appleslicon/MacOs`
### What changes were proposed in this pull request?
After SPARK-38885 upgrading `Netty` to 4.1.76.Final, there are 2 problems:
1. `spark-network-yarn` module compile failed on `Appleslicon/MacOs`
2. The result file generated by `./dev/test-dependencies.sh
--replace-manifest ` on Linux is different from that generated on
`Appleslicon/MacOS`.
This pr add Netty native dependencies explicitly to fix above 2 problems.
### Why are the changes needed?
ix compile error on Appleslicon/MacOs and ensure
`./dev/test-dependencies.sh --replace-manifest ` produce the same results on
Linux and `Appleslicon/MacOS`.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GA
- Manual test 1：
Run `mvn clean install -DskipTests -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn
-Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive`
**Before**
```
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.8:run (unpack) on project
spark-network-yarn_2.12: An Ant BuildException has occured: Warning: Could not
find file
/spark-source/common/network-yarn/target/exploded/META-INF/native/libnetty_transport_native_epoll_x86_64.so
to copy.
[ERROR] around Ant part ...<move
file="/spark-source/common/network-yarn/target/exploded/META-INF/native/libnetty_transport_native_epoll_x86_64.so"
tofile="/spark-source/common/network-yarn/target/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so"/>...
6:340 in /spark-source/common/network-yarn/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR] mvn <args> -rf :spark-network-yarn_2.12
```
**After**
```
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 20:03 min
[INFO] Finished at: 2022-04-24T19:31:23+08:00
[INFO]
------------------------------------------------------------------------
```
- Manual test 2：
Run `./dev/test-dependencies.sh --replace-manifest ` on `Appleslicon/MacOS`
**Before**
```
diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3
b/dev/deps/spark-deps-hadoop-2-hive-2.3
index b6df3ea5ce..acf3d22600 100644
--- a/dev/deps/spark-deps-hadoop-2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2-hive-2.3
-137,6 +137,7 javolution/5.5.1//javolution-5.5.1.jar
jaxb-api/2.2.11//jaxb-api-2.2.11.jar
jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
jcl-over-slf4j/1.7.32//jcl-over-slf4j-1.7.32.jar
+jctools-core/3.1.0//jctools-core-3.1.0.jar
jdo-api/3.0.1//jdo-api-3.0.1.jar
jersey-client/2.35//jersey-client-2.35.jar
jersey-common/2.35//jersey-common-2.35.jar
-209,10 +210,6
netty-resolver/4.1.76.Final//netty-resolver-4.1.76.Final.jar
netty-tcnative-classes/2.0.51.Final//netty-tcnative-classes-2.0.51.Final.jar
netty-transport-classes-epoll/4.1.76.Final//netty-transport-classes-epoll-4.1.76.Final.jar
netty-transport-classes-kqueue/4.1.76.Final//netty-transport-classes-kqueue-4.1.76.Final.jar
-netty-transport-native-epoll/4.1.76.Final/linux-aarch_64/netty-transport-native-epoll-4.1.76.Final-linux-aarch_64.jar
-netty-transport-native-epoll/4.1.76.Final/linux-x86_64/netty-transport-native-epoll-4.1.76.Final-linux-x86_64.jar
-netty-transport-native-kqueue/4.1.76.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.76.Final-osx-aarch_64.jar
-netty-transport-native-kqueue/4.1.76.Final/osx-x86_64/netty-transport-native-kqueue-4.1.76.Final-osx-x86_64.jar
netty-transport-native-unix-common/4.1.76.Final//netty-transport-native-unix-common-4.1.76.Final.jar
netty-transport/4.1.76.Final//netty-transport-4.1.76.Final.jar
objenesis/3.2//objenesis-3.2.jar
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index ad0700db8c..a91129fcb0 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
-124,6 +124,7 javolution/5.5.1//javolution-5.5.1.jar
jaxb-api/2.2.11//jaxb-api-2.2.11.jar
jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar
jcl-over-slf4j/1.7.32//jcl-over-slf4j-1.7.32.jar
+jctools-core/3.1.0//jctools-core-3.1.0.jar
jdo-api/3.0.1//jdo-api-3.0.1.jar
jdom2/2.0.6//jdom2-2.0.6.jar
jersey-client/2.35//jersey-client-2.35.jar
-195,10 +196,6
netty-resolver/4.1.76.Final//netty-resolver-4.1.76.Final.jar
netty-tcnative-classes/2.0.51.Final//netty-tcnative-classes-2.0.51.Final.jar
netty-transport-classes-epoll/4.1.76.Final//netty-transport-classes-epoll-4.1.76.Final.jar
netty-transport-classes-kqueue/4.1.76.Final//netty-transport-classes-kqueue-4.1.76.Final.jar
-netty-transport-native-epoll/4.1.76.Final/linux-aarch_64/netty-transport-native-epoll-4.1.76.Final-linux-aarch_64.jar
-netty-transport-native-epoll/4.1.76.Final/linux-x86_64/netty-transport-native-epoll-4.1.76.Final-linux-x86_64.jar
-netty-transport-native-kqueue/4.1.76.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.76.Final-osx-aarch_64.jar
-netty-transport-native-kqueue/4.1.76.Final/osx-x86_64/netty-transport-native-kqueue-4.1.76.Final-osx-x86_64.jar
netty-transport-native-unix-common/4.1.76.Final//netty-transport-native-unix-common-4.1.76.Final.jar
netty-transport/4.1.76.Final//netty-transport-4.1.76.Final.jar
objenesis/3.2//objenesis-3.2.jar
```
**After**
Consistent with the master branch replica
Closes #36321 from LuciferYang/SPARK-38885-FOLLOWUP.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
---
common/network-common/pom.xml | 24 ++++++++++++++++++++++-
core/pom.xml | 22 ++++++++++++++++++++++
pom.xml | 44 +++++++++++++++++++++++++++++++++++++++++--
3 files changed, 87 insertions(+), 3 deletions(-)
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 1be72c67b04..ef7f9824e13 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -42,19 +42,41 @@
</dependency>
<!-- Core dependencies -->
+ <!-- Netty Begin -->
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-epoll</artifactId>
+ <classifier>linux-x86_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-epoll</artifactId>
+ <classifier>linux-aarch_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-kqueue</artifactId>
+ <classifier>osx-aarch_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-kqueue</artifactId>
+ <classifier>osx-x86_64</classifier>
+ </dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-classes</artifactId>
</dependency>
+ <!-- Netty End -->
+
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
-
<dependency>
<groupId>${leveldbjni.group}</groupId>
<artifactId>leveldbjni-all</artifactId>
diff --git a/core/pom.xml b/core/pom.xml
index 9ba9aa6566c..7d4bab55659 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -299,14 +299,36 @@
<artifactId>jersey-test-framework-provider-simple</artifactId>
<scope>test</scope>
</dependency>
+ <!-- Netty Begin -->
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-epoll</artifactId>
+ <classifier>linux-x86_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-epoll</artifactId>
+ <classifier>linux-aarch_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-kqueue</artifactId>
+ <classifier>osx-aarch_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-kqueue</artifactId>
+ <classifier>osx-x86_64</classifier>
+ </dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-classes</artifactId>
</dependency>
+ <!-- Netty End -->
<dependency>
<groupId>com.clearspring.analytics</groupId>
<artifactId>stream</artifactId>
diff --git a/pom.xml b/pom.xml
index 18a63c5867f..9a43b038426 100644
--- a/pom.xml
+++ b/pom.xml
@@ -207,6 +207,11 @@
<commons-cli.version>1.5.0</commons-cli.version>
<bouncycastle.version>1.60</bouncycastle.version>
<tink.version>1.6.1</tink.version>
+ <!-- When upgrading `netty.version`, need to check whether
+ the version of `netty-tcnative-classes.version` also needs to be
upgraded
+ -->
+ <netty.version>4.1.76.Final</netty.version>
+
<netty-tcnative-classes.version>2.0.51.Final</netty-tcnative-classes.version>
<!--
If you are changing Arrow version specification, please check
./python/pyspark/sql/pandas/utils.py, and ./python/setup.py too.
@@ -816,10 +821,12 @@
<artifactId>RoaringBitmap</artifactId>
<version>0.9.25</version>
</dependency>
+
+ <!-- Netty Begin -->
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
- <version>4.1.76.Final</version>
+ <version>${netty.version}</version>
<exclusions>
<exclusion>
<groupId>io.netty</groupId>
@@ -893,8 +900,39 @@
<groupId>io.netty</groupId>
<artifactId>netty-transport-udt</artifactId>
</exclusion>
+ <exclusion>
+ <groupId>org.jctools</groupId>
+ <artifactId>jctools-core</artifactId>
+ </exclusion>
</exclusions>
</dependency>
+ <!-- SPARK-38885: After Netty 4.1.76, add the following `Netty`
dependencies explicitly
+ to ensure `./dev/test-dependencies.sh` produce the same results on
Linux and MacOS.
+ -->
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-epoll</artifactId>
+ <version>${netty.version}</version>
+ <classifier>linux-x86_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-epoll</artifactId>
+ <version>${netty.version}</version>
+ <classifier>linux-aarch_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-kqueue</artifactId>
+ <version>${netty.version}</version>
+ <classifier>osx-aarch_64</classifier>
+ </dependency>
+ <dependency>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-transport-native-kqueue</artifactId>
+ <version>${netty.version}</version>
+ <classifier>osx-x86_64</classifier>
+ </dependency>
<!-- SPARK-38541: Add the explicit dependency on
`netty-tcnative-classes` because the dependency on
`netty-tcnative-classes` is changed to `Optional` in
[netty](https://github.com/netty/netty/pull/12146).
`netty-tcnative-classes` module is mainly used by SSL package in
`netty-handler`.
@@ -903,8 +941,10 @@
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-classes</artifactId>
- <version>2.0.51.Final</version>
+ <version>${netty-tcnative-classes.version}</version>
</dependency>
+ <!-- Netty End -->
+
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
