Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7929#discussion_r36272840
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -62,6 +64,52 @@ private[hive] class ClientWrapper(
extends ClientInterface
with Logging {
+ overrideHadoopShims()
+
+ // !! HACK ALERT !!
+ //
+ // This method is a surgical fix for Hadoop version 2.0.0-mr1-cdh4.1.1,
which is used by Spark EC2
+ // scripts. We should remove this after upgrading Spark EC2 scripts to
some more recent Hadoop
+ // version in the future.
+ //
+ // Internally, Hive `ShimLoader` tries to load different versions of
Hadoop shims by checking
+ // version information gathered from Hadoop jar files. If the major
version number is 1,
+ // `Hadoop20SShims` will be loaded. Otherwise, if the major version
number is 2, `Hadoop23Shims`
+ // will be chosen.
+ //
+ // However, part of APIs in Hadoop 2.0.x and 2.1.x versions were in flux
due to historical
+ // reasons. So 2.0.0-mr1-cdh4.1.1 is actually more Hadoop-1-like and
should be used together with
--- End diff --
I still think this comment doesn't make sense. My "more Hadoop 1-like"
comment refers to the MapReduce part, which is not relevant here.
`2.0.0-mr1-cdh4.1.1` is correctly a 2.0.x Hadoop build. The next line has a
typo one way or the other.
Right now, the logic is: if Hadoop version = 1.x, then use Hadoop 2.0
shims. Else use the Hadoop 2.3 shims. That's the problem.
The desired logic seems to be: if Hadoop version <= 2.0.x, use Hadoop 2.0
shims. Else use the Hadoop 2.3 shims. That's much better, even if the "2.3
shims" name isn't the most accurate.
Why is the logic not "Hadoop version <= 2.0.x"? why is this suggesting CDH4
is a special case -- let alone mr1? Right now this is still not going to work
for other Hadoop 2.0.x distributions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
