LuciferYang edited a comment on pull request #34881: URL: https://github.com/apache/spark/pull/34881#issuecomment-994261211
> May I ask you to run a maven build as well? Just to be in the safe side... I am not 100% sure but as you changed the exclusion in `SparkBuild.scala` but not in a pom.xml we might find something here. > And agree, I think you need Maven exclusions to match SBT @attilapiros @srowen In SPARK-35134 (https://github.com/apache/spark/pull/32230), there is a netty version conflict issue during sbt test process, the phenomenon that triggers the bug is that even if we use `netty-4.1.51-all.jar`, there will be `copying deps/jars/netty-handler-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars` and `copying deps/jars/netty-transport-native-epoll-4.1.50.Final.jar -> pyspark-3.2.0.dev0/deps/jars` behavior during sbt testing, this makes classpath conflicts possible So I add 2 new `ExclusionRule` in SPARK-35134 to fix this sbt test bug. ``` ExclusionRule("io.netty", "netty-handler"), ExclusionRule("io.netty", "netty-transport-native-epoll") ``` We didn't find this problem during Maven testing, so I didn't make corresponding change for Maven In SPARK-35134. Then after Netty-4.1.69 ([Netty#11732 netty-all should not re-package jars](https://github.com/netty/netty/pull/11732)) , netty-all replaces a fat jar with separate modules, keeping `ExclusionRule("io.netty", "netty-handler")` and `ExclusionRule("io.netty", "netty-transport-native-epoll")` will result missing `netty-transport-native-epoll-4.1.71.Final.jar` and `netty-handler-4.1.71.Final.jar` problem during sbt test because `netty-all-4.1.71.jar` no longer contains them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
