[ 
https://issues.apache.org/jira/browse/SPARK-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joey Paskhay updated SPARK-11702:
---------------------------------
    Description: 
A Guava classloading error can occur when using a different version of the Hive 
metastore.

Running the latest version of Spark at this time (1.5.1) and patched versions 
of Hadoop 2.2.0 and Hive 1.0.0. We set "spark.sql.hive.metastore.version" to 
"1.0.0" and "spark.sql.hive.metastore.jars" to 
"<path_to_hive>/lib/*:<output_of_hadoop_classpath_cmd>". When trying to launch 
the spark-shell, the sqlContext would fail to initialize with:

{code}
java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: 
com/google/common/base/Predicate when creating Hive client using classpath: 
<all the jars>
Please make sure that jars for your version of hive and hadoop are included in 
the paths passed to SQLConfEntry(key = spark.sql.hive.metastore.jars, 
defaultValue=builtin, doc=...
{code}

We verified the Guava libraries are in the huge list of the included jars, but 
we saw that in the 
org.apache.spark.sql.hive.client.IsolatedClientLoader.isSharedClass method it 
seems to assume that *all* "com.google" (excluding "com.google.cloud") classes 
should be loaded from the base class loader. The Spark libraries seem to have 
*some* "com.google.common.base" classes shaded in but not all.

See 
[https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAB51Vx4ipV34e=eishlg7bzldm0uefd_mpyqfe4dodbnbv9...@mail.gmail.com%3E]
 and its replies.

The work-around is to add the guava JAR to the "spark.driver.extraClassPath" 
and "spark.executor.extraClassPath" properties.

  was:
A Guava classloading error can occur when using a different version of the Hive 
metastore.

Running the latest version of Spark at this time (1.5.1) and patched versions 
of Hadoop 2.2.0 and Hive 1.0.0. We set "spark.sql.hive.metastore.version" to 
"1.0.0" and "spark.sql.hive.metastore.jars" to 
"<path_to_hive>/lib/*:<output_of_hadoop_classpath_cmd>". When trying to launch 
the spark-shell, the sqlContext would fail to initialize with:

{code}
java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: 
com/google/common/base/Predicate when creating Hive client using classpath: 
<all the jars>
Please make sure that jars for your version of hive and hadoop are included in 
the paths passed to SQLConfEntry(key = spark.sql.hive.metastore.jars, 
defaultValue=builtin, doc=...
{code}

We verified the Guava libraries are in the huge list of the included jars, but 
we saw that in the 
org.apache.spark.sql.hive.client.IsolatedClientLoader.isSharedClass method it 
seems to assume that *all* "com.google" (excluding "com.google.cloud") classes 
should be loaded from the base class loader. The Spark libraries seem to have 
*some* "com.google.common.base" classes shaded in but not all.

See 
[https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAB51Vx4ipV34e=eishlg7bzldm0uefd_mpyqfe4dodbnbv9...@mail.gmail.com%3E]
 and its replies.

The work-around is to add the guava JAR to the "spark.driver.extraClassPath" 
property.


> Guava ClassLoading Issue When Using Different Hive Metastore Version
> --------------------------------------------------------------------
>
>                 Key: SPARK-11702
>                 URL: https://issues.apache.org/jira/browse/SPARK-11702
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.5.1
>            Reporter: Joey Paskhay
>
> A Guava classloading error can occur when using a different version of the 
> Hive metastore.
> Running the latest version of Spark at this time (1.5.1) and patched versions 
> of Hadoop 2.2.0 and Hive 1.0.0. We set "spark.sql.hive.metastore.version" to 
> "1.0.0" and "spark.sql.hive.metastore.jars" to 
> "<path_to_hive>/lib/*:<output_of_hadoop_classpath_cmd>". When trying to 
> launch the spark-shell, the sqlContext would fail to initialize with:
> {code}
> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: 
> com/google/common/base/Predicate when creating Hive client using classpath: 
> <all the jars>
> Please make sure that jars for your version of hive and hadoop are included 
> in the paths passed to SQLConfEntry(key = spark.sql.hive.metastore.jars, 
> defaultValue=builtin, doc=...
> {code}
> We verified the Guava libraries are in the huge list of the included jars, 
> but we saw that in the 
> org.apache.spark.sql.hive.client.IsolatedClientLoader.isSharedClass method it 
> seems to assume that *all* "com.google" (excluding "com.google.cloud") 
> classes should be loaded from the base class loader. The Spark libraries seem 
> to have *some* "com.google.common.base" classes shaded in but not all.
> See 
> [https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAB51Vx4ipV34e=eishlg7bzldm0uefd_mpyqfe4dodbnbv9...@mail.gmail.com%3E]
>  and its replies.
> The work-around is to add the guava JAR to the "spark.driver.extraClassPath" 
> and "spark.executor.extraClassPath" properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to