[
https://issues.apache.org/jira/browse/SPARK-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345128#comment-14345128
]
Sean Owen commented on SPARK-5143:
----------------------------------
Yes, the same thing appears for 1.2.1:
http://mvnrepository.com/artifact/org.apache.spark/spark-network-yarn_2.11/1.2.1
I tried building for Scala 2.11 and got the correct dependencies:
{code}
[INFO] org.apache.spark:spark-network-yarn_2.11:jar:1.3.0-SNAPSHOT
[INFO] +- org.apache.spark:spark-network-shuffle_2.11:jar:1.3.0-SNAPSHOT:compile
[INFO] | \-
org.apache.spark:spark-network-common_2.11:jar:1.3.0-SNAPSHOT:compile
[INFO] | +- io.netty:netty-all:jar:4.0.23.Final:compile
[INFO] | \- com.google.guava:guava:jar:14.0.1:provided
{code}
But I added a dep to my project:
{code}
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-network-yarn_2.11</artifactId>
<version>1.2.1</version>
</dependency>
{code}
and do indeed see
{code}
[INFO] \- org.apache.spark:spark-network-yarn_2.11:jar:1.2.1:compile
[INFO] +- (org.apache.spark:spark-network-shuffle_2.10:jar:1.2.1:compile)
[INFO] \- (org.spark-project.spark:unused:jar:1.0.0:compile)
{code}
I think the problem here is that the "2.11" must be hard-coded elsewhere into
the POMs for this to work. In particular, scala.binary.version has to be set to
2.11 in the POM in order for the artifact names to be correct, since the
dependency is expressed as {{spark-network-shuffle_${scala.binary.version}}}
Let me try a PR
> spark-network-yarn 2.11 depends on spark-network-shuffle 2.10
> -------------------------------------------------------------
>
> Key: SPARK-5143
> URL: https://issues.apache.org/jira/browse/SPARK-5143
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.2.0
> Reporter: Aniket Bhatnagar
> Priority: Critical
>
> It seems that spark-network-yarn compiled for scala 2.11 depends on
> spark-network-shuffle compiled for scala 2.10. This causes builds with SBT
> 0.13.7 to fail with the error "Conflicting cross-version suffixes".
> Screenshot of dependency:
> http://www.uploady.com/#!/download/6Yn95UZA0DR/3taAJFjCJjrsSXOR
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
