Chao Sun created SPARK-33376:
--------------------------------
Summary: Remove the option of "sharesHadoopClasses" in Hive
IsolatedClientLoader
Key: SPARK-33376
URL: https://issues.apache.org/jira/browse/SPARK-33376
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.1
Reporter: Chao Sun
Currently, when initializing {{IsolatedClientLoader}}, ppl can specify to
either share Hadoop classes from Spark or not. In the latter case it's supposed
to only loads the Hadoop classes from the Hive jars themselves.
However this feature is currently used in two cases: 1) unit tests, 2) when the
Hadoop version defined in Maven can not be found when
{{spark.sql.hive.metastore.jars == "maven"}}. Also when {{sharesHadoopClasses}}
is false, it isn't really only using Hadoop classes from Hive jars: Spark also
download {{hadoop-client}} jar and put it together with the Hive jars, and the
Hadoop version used by {{hadoop-client}} is the same version used by Spark
itself. This could potentially cause issues because we are mixing two versions
of Hadoop jars in the classpath.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
