Joey Paskhay commented on SPARK-11702:

Apologies for the super late response, Sabs. In case you or anyone else is 
still having issues with the work-around, the guava JAR needs to be in both the 
spark.driver.extraClassPath and spark.executor.extraClassPath properties.

So our spark-defaults.conf ended up containing something like the following:


Hope that helps,

> Guava ClassLoading Issue When Using Different Hive Metastore Version
> --------------------------------------------------------------------
>                 Key: SPARK-11702
>                 URL: https://issues.apache.org/jira/browse/SPARK-11702
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.5.1
>            Reporter: Joey Paskhay
> A Guava classloading error can occur when using a different version of the 
> Hive metastore.
> Running the latest version of Spark at this time (1.5.1) and patched versions 
> of Hadoop 2.2.0 and Hive 1.0.0. We set "spark.sql.hive.metastore.version" to 
> "1.0.0" and "spark.sql.hive.metastore.jars" to 
> "<path_to_hive>/lib/*:<output_of_hadoop_classpath_cmd>". When trying to 
> launch the spark-shell, the sqlContext would fail to initialize with:
> {code}
> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: 
> com/google/common/base/Predicate when creating Hive client using classpath: 
> <all the jars>
> Please make sure that jars for your version of hive and hadoop are included 
> in the paths passed to SQLConfEntry(key = spark.sql.hive.metastore.jars, 
> defaultValue=builtin, doc=...
> {code}
> We verified the Guava libraries are in the huge list of the included jars, 
> but we saw that in the 
> org.apache.spark.sql.hive.client.IsolatedClientLoader.isSharedClass method it 
> seems to assume that *all* "com.google" (excluding "com.google.cloud") 
> classes should be loaded from the base class loader. The Spark libraries seem 
> to have *some* "com.google.common.base" classes shaded in but not all.
> See 
> [https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAB51Vx4ipV34e=eishlg7bzldm0uefd_mpyqfe4dodbnbv9...@mail.gmail.com%3E]
>  and its replies.
> The work-around is to add the guava JAR to the "spark.driver.extraClassPath" 
> property.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to