Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/7929#discussion_r36217952
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -62,6 +63,39 @@ private[hive] class ClientWrapper(
extends ClientInterface
with Logging {
+ overrideHadoopShims()
+
+ // !! HACK ALERT !!
+ //
+ // This method is used to workaround CDH Hadoop versions like
2.0.0-mr1-cdh4.1.1.
+ //
+ // Internally, Hive `ShimLoader` tries to load different versions of
Hadoop shims by checking
+ // version information gathered from Hadoop jar files. If the major
version number is 1,
+ // `Hadoop20SShims` will be loaded. Otherwise, if the major version
number is 2, `Hadoop23Shims`
+ // will be chosen. However, CDH Hadoop versions like 2.0.0-mr1-cdh4.1.1
have 2 as major version
+ // number, but contain Hadoop 1 code. This confuses Hive `ShimLoader`
and loads wrong version of
--- End diff --
@srowen Thanks for the detailed explanation! And, wow, I expected this to
be confusing, but not THAT confusing...
Then I think the key problem is what shims should Hive use for various
Hadoop versions. The problematic code is [here] [1]. I found there are also a
bunch of `mr1` variants go [from Hadoop 2.0.0 to 2.6.0] [2]. Is it OK to
assume that all these versions should be used together with `Hadoop20SShims`,
just like all the Hadoop 1.x.x versions?
It seems that the rules for choosing Hadoop shims should be:
1. For 1.x.x versions, use `Hadoop20SShims`
2. For any versions with `mr1` tag, use `Hadoop20SShims`
3. For everything else, use `Hadoop23Shims`
If this is correct, then the remaining question is how to obtain the
knowledge of the `mr1` tag in the version string. Currently Hive `ShimLoader`
is using `o.a.hadoop.util.VersionInfo`, which looks for
`common-version-info.properties` in `hadoop-common`, which knows nothing about
Hadoop 1 (I guess?). That's why the `ShimLoader` receives `2.0.0-cdh4.1.1`
instead of `2.0.0-mr1-cdh4.1.1` in the test failure this PR tries to fix.
[1]:
https://github.com/apache/hive/blob/release-1.2.1/shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java#L159-L176
[2]:
https://repository.cloudera.com/artifactory/public/org/apache/hadoop/hadoop-core/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
