[
https://issues.apache.org/jira/browse/SPARK-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591382#comment-14591382
]
Josiah Samuel Sathiadass commented on SPARK-8410:
-------------------------------------------------
Thanks for looking into this query.
Yes. I did install the artifacts before proceeding with testing the sql/hive
submodule.
I tried with -Phive-thriftserver for both build & test and still no success.
The jars are not getting downloaded automatically into the m2 repository.
I did notice that the respective m2 repository directories are getting created
w/o the jar file during the install stage as follows,
joe@soe10-vm7:~/.m2/repository/org/codehaus/groovy/groovy-all/2.1.6$ ls -lt
total 28
-rw-rw-r-- 1 joe joe 40 Jun 18 02:38 groovy-all-2.1.6.pom.sha1
-rw-rw-r-- 1 joe joe 156 Jun 18 02:38 _maven.repositories
-rw-rw-r-- 1 joe joe 18038 Jun 18 02:38 groovy-all-2.1.6.pom
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Hive 1.4.0
[INFO] ------------------------------------------------------------------------
Downloading:
https://repo1.maven.org/maven2/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.pom
4/18 KB ^M8/18 KB ^M12/18 KB ^M16/18 KB ^M18/18 KB ^M ^M
Downloaded:
https://repo1.maven.org/maven2/org/codehaus/groovy/groovy-all/2.1.6/groovy-all-2.1.6.pom
(18 KB at 9.8 KB/sec)
Its the same case for the other 2 dependencies i.e. asm#asm 3.2 &
org.jboss.netty#netty 3.2.2.Final
Do we need to explicitly add these dependencies in the
"spark/sql/hive/pom.xml" for the respective jars to be downloaded (or)
The jars will get downloaded by the
SparkSubmit.scala(resolveMavenCoordinates) ?
Around 29 testcases are impacted due to the missing jars.
> Hive VersionsSuite RuntimeException
> -----------------------------------
>
> Key: SPARK-8410
> URL: https://issues.apache.org/jira/browse/SPARK-8410
> Project: Spark
> Issue Type: Question
> Components: SQL
> Affects Versions: 1.3.1, 1.4.0
> Environment: IBM Power system - P7
> running Ubuntu 14.04LE
> with IBM JDK version 1.7.0
> Reporter: Josiah Samuel Sathiadass
> Priority: Minor
>
> While testing Spark Project Hive, there are RuntimeExceptions as follows,
> VersionsSuite:
> - success sanity check *** FAILED ***
> java.lang.RuntimeException: [download failed:
> org.jboss.netty#netty;3.2.2.Final!netty.jar(bundle), download failed:
> org.codehaus.groovy#groovy-all;2.1.6!groovy-all.jar, download failed:
> asm#asm;3.2!asm.jar]
> at
> org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:978)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$3.apply(IsolatedClientLoader.scala:62)
> at org.apache.spark.sql.catalyst.util.package$.quietly(package.scala:38)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$.org$apache$spark$sql$hive$client$IsolatedClientLoader$$downloadVersion(IsolatedClientLoader.scala:61)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$$anonfun$1.apply(IsolatedClientLoader.scala:44)
> at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
> at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
> at
> org.apache.spark.sql.hive.client.IsolatedClientLoader$.forVersion(IsolatedClientLoader.scala:44)
> ...
> The tests are executed with the following set of options,
> build/mvn --pl sql/hive --fail-never -Pyarn -Phadoop-2.4
> -Dhadoop.version=2.6.0 test
> Adding the following dependencies in the "spark/sql/hive/pom.xml" file
> solves this issue,
> < <dependency>
> < <groupId>org.jboss.netty</groupId>
> < <artifactId>netty</artifactId>
> < <version>3.2.2.Final</version>
> < <scope>test</scope>
> < </dependency>
> < <dependency>
> < <groupId>org.codehaus.groovy</groupId>
> < <artifactId>groovy-all</artifactId>
> < <version>2.1.6</version>
> < <scope>test</scope>
> < </dependency>
> <
> < <dependency>
> < <groupId>asm</groupId>
> < <artifactId>asm</artifactId>
> < <version>3.2</version>
> < <scope>test</scope>
> < </dependency>
> <
> The question is, Is this the correct way to fix this runtimeException ?
> If yes, Can a pull request fix this issue permanently ?
> If not, suggestions please.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
