[ https://issues.apache.org/jira/browse/SPARK-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400167#comment-15400167 ]
Apache Spark commented on SPARK-1121: ------------------------------------- User 'pwendell' has created a pull request for this issue: https://github.com/apache/spark/pull/37 > Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set > --------------------------------------------------------------------- > > Key: SPARK-1121 > URL: https://issues.apache.org/jira/browse/SPARK-1121 > Project: Spark > Issue Type: Improvement > Components: Build > Reporter: Patrick Cogan > Assignee: prashant > Fix For: 1.0.0 > > > The reason why this is needed is that in the 0.23.X versions of hadoop-client > the avro dependency is fully excluded: > http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/0.23.10/hadoop-client-0.23.10.pom > In later versions 2.2.X the avro dependency is correctly inherited from > hadoop-common: > http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.2.0/hadoop-client-2.2.0.pom > So as a workaround Spark currently depends on Avro directly in the sbt and > scala builds. This is a bit ugly so I'd like to propose the following: > 1. In the Maven build just remove avro and make a note on the > building-with-maven page that they will need to manually add avro for this > build. > 2. On sbt only add the avro dependency if the version is 0.23.X and > SPARK_YARN is true. Also we only need to add avro not both {avro, avro-ipc} > like is there now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org