Hi folks, I have been using Spark against an external Metastore service which runs Hive with Cdh 4.6
In Spark 1.2, I was able to successfully connect by building with the following: ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -Phive-thriftserver -Phive-0.12.0 I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor of spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars When I tried to use this setup spark-shell fails for me with the following error: 15/06/23 18:18:07 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: file:/hive/lib/guava-11.0.2.jar, file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar, file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar, file:/hive/lib/commons-logging-1.0.4.jar I don't know why it's not seeing the class -- it's in the guava jar. If anyone has had success with 0.12 version please let me know what jars need to be on the classpath. I think my Hive version might be too outdated but I don't control the metastore and I had success with Spark1.2 so I'm hoping...
