[ https://issues.apache.org/jira/browse/SPARK-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510748#comment-15510748 ]
Joey Paskhay commented on SPARK-11702: -------------------------------------- Apologies for the super late response, Sabs. In case you or anyone else is still having issues with the work-around, the guava JAR needs to be in both the spark.driver.extraClassPath and spark.executor.extraClassPath properties. So our spark-defaults.conf ended up containing something like the following: ... spark.driver.extraClassPath=/usr/lib/hive/lib/guava-15.0.jar:<any_other_custom_jars> spark.executor.extraClassPath=/usr/lib/hive/lib/guava-15.0.jar:<any_other_custom_jars> ... Hope that helps, Joey > Guava ClassLoading Issue When Using Different Hive Metastore Version > -------------------------------------------------------------------- > > Key: SPARK-11702 > URL: https://issues.apache.org/jira/browse/SPARK-11702 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.5.1 > Reporter: Joey Paskhay > > A Guava classloading error can occur when using a different version of the > Hive metastore. > Running the latest version of Spark at this time (1.5.1) and patched versions > of Hadoop 2.2.0 and Hive 1.0.0. We set "spark.sql.hive.metastore.version" to > "1.0.0" and "spark.sql.hive.metastore.jars" to > "<path_to_hive>/lib/*:<output_of_hadoop_classpath_cmd>". When trying to > launch the spark-shell, the sqlContext would fail to initialize with: > {code} > java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: > com/google/common/base/Predicate when creating Hive client using classpath: > <all the jars> > Please make sure that jars for your version of hive and hadoop are included > in the paths passed to SQLConfEntry(key = spark.sql.hive.metastore.jars, > defaultValue=builtin, doc=... > {code} > We verified the Guava libraries are in the huge list of the included jars, > but we saw that in the > org.apache.spark.sql.hive.client.IsolatedClientLoader.isSharedClass method it > seems to assume that *all* "com.google" (excluding "com.google.cloud") > classes should be loaded from the base class loader. The Spark libraries seem > to have *some* "com.google.common.base" classes shaded in but not all. > See > [https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAB51Vx4ipV34e=eishlg7bzldm0uefd_mpyqfe4dodbnbv9...@mail.gmail.com%3E] > and its replies. > The work-around is to add the guava JAR to the "spark.driver.extraClassPath" > property. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org