sunchao commented on a change in pull request #29843:
URL: https://github.com/apache/spark/pull/29843#discussion_r509704948
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -118,19 +118,32 @@ private[hive] object IsolatedClientLoader extends Logging
{
hadoopVersion: String,
ivyPath: Option[String],
remoteRepos: String): Seq[URL] = {
+ val hadoopJarNames = if (hadoopVersion.startsWith("3")) {
+ Seq(s"org.apache.hadoop:hadoop-client-api:$hadoopVersion",
+ s"org.apache.hadoop:hadoop-client-runtime:$hadoopVersion")
+ } else {
+ Seq(s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+ }
val hiveArtifacts = version.extraDeps ++
Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
.map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
- Seq("com.google.guava:guava:14.0.1",
- s"org.apache.hadoop:hadoop-client:$hadoopVersion")
+ Seq("com.google.guava:guava:14.0.1") ++ hadoopJarNames
+
+ val extraExclusions = if (hadoopVersion.startsWith("3")) {
+ // this introduced from lower version of Hive could conflict with jars
in Hadoop 3.2+, so
+ // exclude here in favor of the ones in Hadoop 3.2+
+ Seq("org.apache.hadoop:hadoop-auth")
Review comment:
I only excluded this one because it caused issues, but yes we could
exclude more.
A bit more context, I think currently the way it works is a bit messed up:
it pulls hadoop dependencies from both Hive and the explicit `hadoop-client`
dependency, while the latter uses the Hadoop version in Spark (which is 3.x in
default, as opposed to 2.x in Hive). As result, one can end up with two
versions of various Hadoop jars in the class path and conflicts can happen
unexpectedly. This only happens when we are not sharing Hadoop classes though
(which is rare): in the other path Spark always loads its own Hadoop classes
first, then Hive's.
This was amplified when we replace hadoop-client with hadoop-client-api,
since latter does not include other jars such as hadoop-common: it just put all
classes in a single jar.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
