[
https://issues.apache.org/jira/browse/SPARK-33376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-33376:
------------------------------------
Assignee: Chao Sun (was: Apache Spark)
> Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
> -----------------------------------------------------------------------
>
> Key: SPARK-33376
> URL: https://issues.apache.org/jira/browse/SPARK-33376
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.1
> Reporter: Chao Sun
> Assignee: Chao Sun
> Priority: Major
> Fix For: 3.1.0
>
>
> Currently, when initializing {{IsolatedClientLoader}}, ppl can specify to
> either share Hadoop classes from Spark or not. In the latter case it's
> supposed to only loads the Hadoop classes from the Hive jars themselves.
> However this feature is currently used in two cases: 1) unit tests, 2) when
> the Hadoop version defined in Maven can not be found when
> {{spark.sql.hive.metastore.jars == "maven"}}. Also when
> {{sharesHadoopClasses}} is false, it isn't really only using Hadoop classes
> from Hive jars: Spark also download {{hadoop-client}} jar and put it together
> with the Hive jars, and the Hadoop version used by {{hadoop-client}} is the
> same version used by Spark itself. This could potentially cause issues
> because we are mixing two versions of Hadoop jars in the classpath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
