Re: Can Spark1.4 work with CDH4.6
You can look into spark.driver.userClassPathFirst flag. spark.driver.userClassPathFirstfalse(Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in the the driver. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. This is used in cluster mode only. Thanks Best Regards On Thu, Jun 25, 2015 at 1:08 AM, Yana Kadiyska yana.kadiy...@gmail.com wrote: Thanks, that did seem to make a difference. I am a bit scared of this approach as spark itself has a different guava dependency but the error does go away this way On Wed, Jun 24, 2015 at 10:04 AM, Akhil Das ak...@sigmoidanalytics.com wrote: Can you try to add those jars in the SPARK_CLASSPATH and give it a try? Thanks Best Regards On Wed, Jun 24, 2015 at 12:07 AM, Yana Kadiyska yana.kadiy...@gmail.com wrote: Hi folks, I have been using Spark against an external Metastore service which runs Hive with Cdh 4.6 In Spark 1.2, I was able to successfully connect by building with the following: ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -Phive-thriftserver -Phive-0.12.0 I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor of spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars When I tried to use this setup spark-shell fails for me with the following error: 15/06/23 18:18:07 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: file:/hive/lib/guava-11.0.2.jar, file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar, file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar, file:/hive/lib/commons-logging-1.0.4.jar I don't know why it's not seeing the class -- it's in the guava jar. If anyone has had success with 0.12 version please let me know what jars need to be on the classpath. I think my Hive version might be too outdated but I don't control the metastore and I had success with Spark1.2 so I'm hoping...
Re: Can Spark1.4 work with CDH4.6
Can you try to add those jars in the SPARK_CLASSPATH and give it a try? Thanks Best Regards On Wed, Jun 24, 2015 at 12:07 AM, Yana Kadiyska yana.kadiy...@gmail.com wrote: Hi folks, I have been using Spark against an external Metastore service which runs Hive with Cdh 4.6 In Spark 1.2, I was able to successfully connect by building with the following: ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -Phive-thriftserver -Phive-0.12.0 I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor of spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars When I tried to use this setup spark-shell fails for me with the following error: 15/06/23 18:18:07 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: file:/hive/lib/guava-11.0.2.jar, file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar, file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar, file:/hive/lib/commons-logging-1.0.4.jar I don't know why it's not seeing the class -- it's in the guava jar. If anyone has had success with 0.12 version please let me know what jars need to be on the classpath. I think my Hive version might be too outdated but I don't control the metastore and I had success with Spark1.2 so I'm hoping...
Re: Can Spark1.4 work with CDH4.6
Thanks, that did seem to make a difference. I am a bit scared of this approach as spark itself has a different guava dependency but the error does go away this way On Wed, Jun 24, 2015 at 10:04 AM, Akhil Das ak...@sigmoidanalytics.com wrote: Can you try to add those jars in the SPARK_CLASSPATH and give it a try? Thanks Best Regards On Wed, Jun 24, 2015 at 12:07 AM, Yana Kadiyska yana.kadiy...@gmail.com wrote: Hi folks, I have been using Spark against an external Metastore service which runs Hive with Cdh 4.6 In Spark 1.2, I was able to successfully connect by building with the following: ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -Phive-thriftserver -Phive-0.12.0 I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor of spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars When I tried to use this setup spark-shell fails for me with the following error: 15/06/23 18:18:07 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: file:/hive/lib/guava-11.0.2.jar, file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar, file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar, file:/hive/lib/commons-logging-1.0.4.jar I don't know why it's not seeing the class -- it's in the guava jar. If anyone has had success with 0.12 version please let me know what jars need to be on the classpath. I think my Hive version might be too outdated but I don't control the metastore and I had success with Spark1.2 so I'm hoping...
Can Spark1.4 work with CDH4.6
Hi folks, I have been using Spark against an external Metastore service which runs Hive with Cdh 4.6 In Spark 1.2, I was able to successfully connect by building with the following: ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -Phive-thriftserver -Phive-0.12.0 I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor of spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars When I tried to use this setup spark-shell fails for me with the following error: 15/06/23 18:18:07 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: file:/hive/lib/guava-11.0.2.jar, file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar, file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar, file:/hive/lib/commons-logging-1.0.4.jar I don't know why it's not seeing the class -- it's in the guava jar. If anyone has had success with 0.12 version please let me know what jars need to be on the classpath. I think my Hive version might be too outdated but I don't control the metastore and I had success with Spark1.2 so I'm hoping...
