Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/7929#discussion_r36273191
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -62,6 +64,52 @@ private[hive] class ClientWrapper(
extends ClientInterface
with Logging {
+ overrideHadoopShims()
+
+ // !! HACK ALERT !!
+ //
+ // This method is a surgical fix for Hadoop version 2.0.0-mr1-cdh4.1.1,
which is used by Spark EC2
+ // scripts. We should remove this after upgrading Spark EC2 scripts to
some more recent Hadoop
+ // version in the future.
+ //
+ // Internally, Hive `ShimLoader` tries to load different versions of
Hadoop shims by checking
+ // version information gathered from Hadoop jar files. If the major
version number is 1,
+ // `Hadoop20SShims` will be loaded. Otherwise, if the major version
number is 2, `Hadoop23Shims`
+ // will be chosen.
+ //
+ // However, part of APIs in Hadoop 2.0.x and 2.1.x versions were in flux
due to historical
+ // reasons. So 2.0.0-mr1-cdh4.1.1 is actually more Hadoop-1-like and
should be used together with
--- End diff --
Yeah I agree the comment is slightly wrong. I think CDH4 named the release
with "mr1" because they took the upstream 2.0.X release but then packaged with
the older (pre-yarn) version of MR. So this comment could be improved or just
made shorter.
In terms of covering other Hadoop 2.0.x distributions - as far as I know no
one other than cloudera ever really distributed this. I am pretty hesitant to
make any assumptions about what other Hadoop 2.0.x distributions might contain,
because that in general was not a time of API stability for Hadoop and there
generally variance around API's. So my feeling was to just cover the one case
we do distribute binary builds for (the chd4 distribution).
My main feeling was, we should make this work for the cdh4 version that we
do provide binary builds for, but not go crazy trying to hypothesize about
other one-off hadoop versions that were packaged around that time, if any exist.
I do agree though the comment could be made more succinct and accurate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
