I want to add spark-hive as a dependence to submit my job, but it seems that spark-submit can not resolve it.
$ ./bin/spark-submit \ → --packages org.apache.spark:spark-hive_2.10:1.4.0,org.postgresql:postgresql:9.3-1103-jdbc3,joda-time:joda-time:2.8.1 \ → --class fr.leboncoin.etl.jobs.dwh.AdStateTraceDWHTransform \ → --master spark://localhost:7077 \ Ivy Default Cache set to: /home/invkrh/.ivy2/cache The jars for the packages stored in: /home/invkrh/.ivy2/jars https://repository.jboss.org/nexus/content/repositories/releases/ added as a remote repository with the name: repo-1 :: loading settings :: url = jar:file:/home/invkrh/workspace/scala/spark/assembly/target/scala-2.10/spark-assembly-1.4.0-SNAPSHOT-hadoop2.2.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.apache.spark#spark-hive_2.10 added as a dependency org.postgresql#postgresql added as a dependency joda-time#joda-time added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.postgresql#postgresql;9.3-1103-jdbc3 in local-m2-cache found joda-time#joda-time;2.8.1 in central :: resolution report :: resolve 139ms :: artifacts dl 3ms :: modules in use: joda-time#joda-time;2.8.1 from central in [default] org.postgresql#postgresql;9.3-1103-jdbc3 from local-m2-cache in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 2 | 0 | 0 | 0 || 2 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 2 already retrieved (0kB/6ms) Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/HiveContext at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:633) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveContext at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/07/07 16:57:59 INFO Utils: Shutdown hook called Any help is appreciated. Thank you. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-can-not-resolve-spark-hive-2-10-tp23695.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
