[ https://issues.apache.org/jira/browse/SPARK-23509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376198#comment-16376198 ]
Kazuaki Ishizaki commented on SPARK-23509: ------------------------------------------ During the build using sbt, we are seeing the following message, too. {code} [warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible: [warn] [warn] * com.google.code.findbugs:jsr305:3.0.2 is selected over 1.3.9 [warn] +- org.apache.arrow:arrow-vector:0.8.0 (depends on 3.0.2) [warn] +- org.apache.arrow:arrow-memory:0.8.0 (depends on 3.0.2) [warn] +- org.apache.spark:spark-unsafe_2.11:2.4.0-SNAPSHOT (depends on 1.3.9) [warn] +- org.apache.hadoop:hadoop-common:2.6.5 (depends on 1.3.9) [warn] +- org.apache.spark:spark-core_2.11:2.4.0-SNAPSHOT (depends on 1.3.9) [warn] +- org.apache.spark:spark-network-common_2.11:2.4.0-SNAPSHOT (depends on 1.3.9) [warn] [warn] * io.netty:netty:3.9.9.Final is selected over {3.6.2.Final, 3.7.0.Final} [warn] +- org.apache.spark:spark-core_2.11:2.4.0-SNAPSHOT (depends on 3.9.9.Final) [warn] +- org.apache.zookeeper:zookeeper:3.4.6 (depends on 3.6.2.Final) [warn] +- org.apache.hadoop:hadoop-hdfs:2.6.5 (depends on 3.6.2.Final) [warn] [warn] * commons-net:commons-net:3.1 is selected over 2.2 [warn] +- org.apache.hadoop:hadoop-common:2.6.5 (depends on 3.1) [warn] +- org.apache.spark:spark-core_2.11:2.4.0-SNAPSHOT (depends on 2.2) {code} > Upgrade commons-net from 2.2 to 3.1 > ----------------------------------- > > Key: SPARK-23509 > URL: https://issues.apache.org/jira/browse/SPARK-23509 > Project: Spark > Issue Type: Dependency upgrade > Components: Spark Core > Affects Versions: 2.4.0 > Reporter: PandaMonkey > Priority: Major > Fix For: 2.4.0 > > Attachments: spark.txt > > > Hi, after analyzing spark-master\core\pom.xml, we found that Spark-core > depends on org.apache.hadoop:hadoop-client:2.6.5, which transitivity > introduced commons-net:3.1. At the same time, Spark-core directly depends on > a older version of commons-net:2.2. By further look into the source code, > these two versions of commons-net have many different features. The > dependency conflict problem brings high risks of "NotClassDefFoundError:" or > "NoSuchMethodError" issues at runtime. Please notice this problem. Maybe > upgrading commons-net from 2.2 to 3.1 is a good choice. Hope this report can > help you. Thanks! > > Regards, > Panda -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org