[ https://issues.apache.org/jira/browse/SPARK-33376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon reassigned SPARK-33376: ------------------------------------ Assignee: Chao Sun (was: Apache Spark) > Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader > ----------------------------------------------------------------------- > > Key: SPARK-33376 > URL: https://issues.apache.org/jira/browse/SPARK-33376 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.1 > Reporter: Chao Sun > Assignee: Chao Sun > Priority: Major > Fix For: 3.1.0 > > > Currently, when initializing {{IsolatedClientLoader}}, ppl can specify to > either share Hadoop classes from Spark or not. In the latter case it's > supposed to only loads the Hadoop classes from the Hive jars themselves. > However this feature is currently used in two cases: 1) unit tests, 2) when > the Hadoop version defined in Maven can not be found when > {{spark.sql.hive.metastore.jars == "maven"}}. Also when > {{sharesHadoopClasses}} is false, it isn't really only using Hadoop classes > from Hive jars: Spark also download {{hadoop-client}} jar and put it together > with the Hive jars, and the Hadoop version used by {{hadoop-client}} is the > same version used by Spark itself. This could potentially cause issues > because we are mixing two versions of Hadoop jars in the classpath. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org