This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 7154fb989de [SPARK-38885][BUILD][FOLLOWUP] Fix compile error on `Appleslicon/MacOs` and ensure `./dev/test-dependencies.sh` produce the same results on Linux and `Appleslicon/MacOs` 7154fb989de is described below commit 7154fb989de9b0dfefb5dc8953ee63ab43dc414b Author: yangjie01 <yangji...@baidu.com> AuthorDate: Sun Apr 24 11:02:02 2022 -0500 [SPARK-38885][BUILD][FOLLOWUP] Fix compile error on `Appleslicon/MacOs` and ensure `./dev/test-dependencies.sh` produce the same results on Linux and `Appleslicon/MacOs` ### What changes were proposed in this pull request? After SPARK-38885 upgrading `Netty` to 4.1.76.Final, there are 2 problems: 1. `spark-network-yarn` module compile failed on `Appleslicon/MacOs` 2. The result file generated by `./dev/test-dependencies.sh --replace-manifest ` on Linux is different from that generated on `Appleslicon/MacOS`. This pr add Netty native dependencies explicitly to fix above 2 problems. ### Why are the changes needed? ix compile error on Appleslicon/MacOs and ensure `./dev/test-dependencies.sh --replace-manifest ` produce the same results on Linux and `Appleslicon/MacOS`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GA - Manual test 1: Run `mvn clean install -DskipTests -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive` **Before** ``` [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (unpack) on project spark-network-yarn_2.12: An Ant BuildException has occured: Warning: Could not find file /spark-source/common/network-yarn/target/exploded/META-INF/native/libnetty_transport_native_epoll_x86_64.so to copy. [ERROR] around Ant part ...<move file="/spark-source/common/network-yarn/target/exploded/META-INF/native/libnetty_transport_native_epoll_x86_64.so" tofile="/spark-source/common/network-yarn/target/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so"/>... 6:340 in /spark-source/common/network-yarn/target/antrun/build-main.xml [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :spark-network-yarn_2.12 ``` **After** ``` [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 20:03 min [INFO] Finished at: 2022-04-24T19:31:23+08:00 [INFO] ------------------------------------------------------------------------ ``` - Manual test 2: Run `./dev/test-dependencies.sh --replace-manifest ` on `Appleslicon/MacOS` **Before** ``` diff --git a/dev/deps/spark-deps-hadoop-2-hive-2.3 b/dev/deps/spark-deps-hadoop-2-hive-2.3 index b6df3ea5ce..acf3d22600 100644 --- a/dev/deps/spark-deps-hadoop-2-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-2-hive-2.3 -137,6 +137,7 javolution/5.5.1//javolution-5.5.1.jar jaxb-api/2.2.11//jaxb-api-2.2.11.jar jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar jcl-over-slf4j/1.7.32//jcl-over-slf4j-1.7.32.jar +jctools-core/3.1.0//jctools-core-3.1.0.jar jdo-api/3.0.1//jdo-api-3.0.1.jar jersey-client/2.35//jersey-client-2.35.jar jersey-common/2.35//jersey-common-2.35.jar -209,10 +210,6 netty-resolver/4.1.76.Final//netty-resolver-4.1.76.Final.jar netty-tcnative-classes/2.0.51.Final//netty-tcnative-classes-2.0.51.Final.jar netty-transport-classes-epoll/4.1.76.Final//netty-transport-classes-epoll-4.1.76.Final.jar netty-transport-classes-kqueue/4.1.76.Final//netty-transport-classes-kqueue-4.1.76.Final.jar -netty-transport-native-epoll/4.1.76.Final/linux-aarch_64/netty-transport-native-epoll-4.1.76.Final-linux-aarch_64.jar -netty-transport-native-epoll/4.1.76.Final/linux-x86_64/netty-transport-native-epoll-4.1.76.Final-linux-x86_64.jar -netty-transport-native-kqueue/4.1.76.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.76.Final-osx-aarch_64.jar -netty-transport-native-kqueue/4.1.76.Final/osx-x86_64/netty-transport-native-kqueue-4.1.76.Final-osx-x86_64.jar netty-transport-native-unix-common/4.1.76.Final//netty-transport-native-unix-common-4.1.76.Final.jar netty-transport/4.1.76.Final//netty-transport-4.1.76.Final.jar objenesis/3.2//objenesis-3.2.jar diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index ad0700db8c..a91129fcb0 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 -124,6 +124,7 javolution/5.5.1//javolution-5.5.1.jar jaxb-api/2.2.11//jaxb-api-2.2.11.jar jaxb-runtime/2.3.2//jaxb-runtime-2.3.2.jar jcl-over-slf4j/1.7.32//jcl-over-slf4j-1.7.32.jar +jctools-core/3.1.0//jctools-core-3.1.0.jar jdo-api/3.0.1//jdo-api-3.0.1.jar jdom2/2.0.6//jdom2-2.0.6.jar jersey-client/2.35//jersey-client-2.35.jar -195,10 +196,6 netty-resolver/4.1.76.Final//netty-resolver-4.1.76.Final.jar netty-tcnative-classes/2.0.51.Final//netty-tcnative-classes-2.0.51.Final.jar netty-transport-classes-epoll/4.1.76.Final//netty-transport-classes-epoll-4.1.76.Final.jar netty-transport-classes-kqueue/4.1.76.Final//netty-transport-classes-kqueue-4.1.76.Final.jar -netty-transport-native-epoll/4.1.76.Final/linux-aarch_64/netty-transport-native-epoll-4.1.76.Final-linux-aarch_64.jar -netty-transport-native-epoll/4.1.76.Final/linux-x86_64/netty-transport-native-epoll-4.1.76.Final-linux-x86_64.jar -netty-transport-native-kqueue/4.1.76.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.76.Final-osx-aarch_64.jar -netty-transport-native-kqueue/4.1.76.Final/osx-x86_64/netty-transport-native-kqueue-4.1.76.Final-osx-x86_64.jar netty-transport-native-unix-common/4.1.76.Final//netty-transport-native-unix-common-4.1.76.Final.jar netty-transport/4.1.76.Final//netty-transport-4.1.76.Final.jar objenesis/3.2//objenesis-3.2.jar ``` **After** Consistent with the master branch replica Closes #36321 from LuciferYang/SPARK-38885-FOLLOWUP. Authored-by: yangjie01 <yangji...@baidu.com> Signed-off-by: Sean Owen <sro...@gmail.com> --- common/network-common/pom.xml | 24 ++++++++++++++++++++++- core/pom.xml | 22 ++++++++++++++++++++++ pom.xml | 44 +++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 87 insertions(+), 3 deletions(-) diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml index 1be72c67b04..ef7f9824e13 100644 --- a/common/network-common/pom.xml +++ b/common/network-common/pom.xml @@ -42,19 +42,41 @@ </dependency> <!-- Core dependencies --> + <!-- Netty Begin --> <dependency> <groupId>io.netty</groupId> <artifactId>netty-all</artifactId> </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-epoll</artifactId> + <classifier>linux-x86_64</classifier> + </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-epoll</artifactId> + <classifier>linux-aarch_64</classifier> + </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-kqueue</artifactId> + <classifier>osx-aarch_64</classifier> + </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-kqueue</artifactId> + <classifier>osx-x86_64</classifier> + </dependency> <dependency> <groupId>io.netty</groupId> <artifactId>netty-tcnative-classes</artifactId> </dependency> + <!-- Netty End --> + <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> </dependency> - <dependency> <groupId>${leveldbjni.group}</groupId> <artifactId>leveldbjni-all</artifactId> diff --git a/core/pom.xml b/core/pom.xml index 9ba9aa6566c..7d4bab55659 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -299,14 +299,36 @@ <artifactId>jersey-test-framework-provider-simple</artifactId> <scope>test</scope> </dependency> + <!-- Netty Begin --> <dependency> <groupId>io.netty</groupId> <artifactId>netty-all</artifactId> </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-epoll</artifactId> + <classifier>linux-x86_64</classifier> + </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-epoll</artifactId> + <classifier>linux-aarch_64</classifier> + </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-kqueue</artifactId> + <classifier>osx-aarch_64</classifier> + </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-kqueue</artifactId> + <classifier>osx-x86_64</classifier> + </dependency> <dependency> <groupId>io.netty</groupId> <artifactId>netty-tcnative-classes</artifactId> </dependency> + <!-- Netty End --> <dependency> <groupId>com.clearspring.analytics</groupId> <artifactId>stream</artifactId> diff --git a/pom.xml b/pom.xml index 18a63c5867f..9a43b038426 100644 --- a/pom.xml +++ b/pom.xml @@ -207,6 +207,11 @@ <commons-cli.version>1.5.0</commons-cli.version> <bouncycastle.version>1.60</bouncycastle.version> <tink.version>1.6.1</tink.version> + <!-- When upgrading `netty.version`, need to check whether + the version of `netty-tcnative-classes.version` also needs to be upgraded + --> + <netty.version>4.1.76.Final</netty.version> + <netty-tcnative-classes.version>2.0.51.Final</netty-tcnative-classes.version> <!-- If you are changing Arrow version specification, please check ./python/pyspark/sql/pandas/utils.py, and ./python/setup.py too. @@ -816,10 +821,12 @@ <artifactId>RoaringBitmap</artifactId> <version>0.9.25</version> </dependency> + + <!-- Netty Begin --> <dependency> <groupId>io.netty</groupId> <artifactId>netty-all</artifactId> - <version>4.1.76.Final</version> + <version>${netty.version}</version> <exclusions> <exclusion> <groupId>io.netty</groupId> @@ -893,8 +900,39 @@ <groupId>io.netty</groupId> <artifactId>netty-transport-udt</artifactId> </exclusion> + <exclusion> + <groupId>org.jctools</groupId> + <artifactId>jctools-core</artifactId> + </exclusion> </exclusions> </dependency> + <!-- SPARK-38885: After Netty 4.1.76, add the following `Netty` dependencies explicitly + to ensure `./dev/test-dependencies.sh` produce the same results on Linux and MacOS. + --> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-epoll</artifactId> + <version>${netty.version}</version> + <classifier>linux-x86_64</classifier> + </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-epoll</artifactId> + <version>${netty.version}</version> + <classifier>linux-aarch_64</classifier> + </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-kqueue</artifactId> + <version>${netty.version}</version> + <classifier>osx-aarch_64</classifier> + </dependency> + <dependency> + <groupId>io.netty</groupId> + <artifactId>netty-transport-native-kqueue</artifactId> + <version>${netty.version}</version> + <classifier>osx-x86_64</classifier> + </dependency> <!-- SPARK-38541: Add the explicit dependency on `netty-tcnative-classes` because the dependency on `netty-tcnative-classes` is changed to `Optional` in [netty](https://github.com/netty/netty/pull/12146). `netty-tcnative-classes` module is mainly used by SSL package in `netty-handler`. @@ -903,8 +941,10 @@ <dependency> <groupId>io.netty</groupId> <artifactId>netty-tcnative-classes</artifactId> - <version>2.0.51.Final</version> + <version>${netty-tcnative-classes.version}</version> </dependency> + <!-- Netty End --> + <dependency> <groupId>org.apache.derby</groupId> <artifactId>derby</artifactId> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org