[
https://issues.apache.org/jira/browse/SPARK-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14608002#comment-14608002
]
Josiah Samuel Sathiadass commented on SPARK-8410:
-------------------------------------------------
Great Burak. Your fix works.
I have to recreate my m2 and ivy2 repo's by completely deleting these
directories from my home and then applied your commit, kick started the
compilation and it works.
For our testing, we were in spark 1.4.0-rc4 and I cherry picked the following
commits to get this working,
218be70 [SPARK-8410] [SPARK-8475] remove previous ivy resolution when using
spark-submit
06788cc [SPARK-8095] Resolve dependencies of --packages in local ivy cache
01accf0 [SPARK-8126] [BUILD] Use custom temp directory during build.
Now we are moving to spark 1.4.1.
Thanks,
Joe.
> Hive VersionsSuite RuntimeException
> -----------------------------------
>
> Key: SPARK-8410
> URL: https://issues.apache.org/jira/browse/SPARK-8410
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.3.1, 1.4.0
> Environment: IBM Power system - P7
> running Ubuntu 14.04LE
> Reporter: Josiah Samuel Sathiadass
> Assignee: Burak Yavuz
> Priority: Minor
> Fix For: 1.5.0, 1.4.2
>
>
> While testing Spark Project Hive, there are RuntimeExceptions as follows,
> VersionsSuite:
> - success sanity check *** FAILED ***
> java.lang.RuntimeException: [download failed:
> org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle), download failed:
> org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar, download failed:
> asm#asm;3.2!asm.jar]
> at
> org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:978)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62)
> at org.apache.spark.sql.catalyst.util.package$.quietly(package.scala:38)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$.org$apache$spark$sql$hive$client$IsolatedClientLoader$$downloadVersion(IsolatedClientLoader.scala:61)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44)
> at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
> at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$.forVersion(IsolatedClientLoader.scala:44)
> ...
> The tests are executed with the following set of options,
> build/mvn --pl sql/hive --fail-never -Pyarn -Phadoop-2.4
> -Dhadoop.version=2.6.0 test
> Adding the following dependencies in the "spark/sql/hive/pom.xml" file
> solves this issue,
> < <dependency>
> < <groupId>org.jboss.netty</groupId>
> < <artifactId>netty</artifactId>
> < <version>3.2.2.Final</version>
> < <scope>test</scope>
> < </dependency>
> < <dependency>
> < <groupId>org.codehaus.groovy</groupId>
> < <artifactId>groovy-all</artifactId>
> < <version>2.1.6</version>
> < <scope>test</scope>
> < </dependency>
> <
> < <dependency>
> < <groupId>asm</groupId>
> < <artifactId>asm</artifactId>
> < <version>3.2</version>
> < <scope>test</scope>
> < </dependency>
> <
> The question is, Is this the correct way to fix this runtimeException ?
> If yes, Can a pull request fix this issue permanently ?
> If not, suggestions please.
> Updates:
> The above mentioned quick fix is not working with the latest 1.4 because of
> this pull commits :
> [SPARK-8095] Resolve dependencies of --packages in local ivy cache #6788
> https://github.com/apache/spark/pull/6788
> Due to this above commit, now the lookup directories during testing phase
> has changed as follows,
> :: problems summary ::
> :::: WARNINGS
> [NOT FOUND ]
> org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle) (2ms)
> ==== local-m2-cache: tried
>
> file:/home/joe/sparkibmsoe/spark/sql/hive/dummy/.m2/repository/org/jboss/netty/netty/3.2.2.Final/netty-3.2.2.Final.jar
> [NOT FOUND ]
> org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar (0ms)
> ==== local-m2-cache: tried
>
> file:/home/joe/sparkibmsoe/spark/sql/hive/dummy/.m2/repository/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.jar
> [NOT FOUND ] asm#asm;3.2!asm.jar (0ms)
> ==== local-m2-cache: tried
>
> file:/home/joe/sparkibmsoe/spark/sql/hive/dummy/.m2/repository/asm/asm/3.2/asm-3.2.jar
> ::::::::::::::::::::::::::::::::::::::::::::::
> :: FAILED DOWNLOADS ::
> :: ^ see resolution messages for details ^ ::
> ::::::::::::::::::::::::::::::::::::::::::::::
> :: org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle)
> :: org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar
> :: asm#asm;3.2!asm.jar
> ::::::::::::::::::::::::::::::::::::::::::::::
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
