Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7929#discussion_r36232467
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -62,6 +63,43 @@ private[hive] class ClientWrapper(
extends ClientInterface
with Logging {
+ overrideHadoopShims()
+
+ // !! HACK ALERT !!
+ //
+ // This method is used to workaround CDH Hadoop versions like
2.0.0-mr1-cdh4.1.1.
+ //
+ // Internally, Hive `ShimLoader` tries to load different versions of
Hadoop shims by checking
+ // version information gathered from Hadoop jar files. If the major
version number is 1,
+ // `Hadoop20SShims` will be loaded. Otherwise, if the major version
number is 2, `Hadoop23Shims`
+ // will be chosen.
+ //
+ // However, part of APIs in Hadoop 2.0.x and 2.1.x versions were in flux
due to historical reasons.
+ // So CDH Hadoop versions like 2.0.0-mr1-cdh4.1.1 are more
Hadoop-1-like, but have a major version
+ // of 2. This confuses Hive `ShimLoader` and loads wrong version of
shims.
+ //
+ // Here we check for existence of the
`Path.getPathWithoutSchemeAndAuthority` method, which
+ // doesn't exist in Hadoop 1 and 2.0.x (it's also the method that
reveals this shims loading
--- End diff --
I think this is better, but, the method you're checking for shouldn't be in
any 2.0.x-based Hadoop distribution. So I don't think CDH or the `mr1` flavor
is a special case here. The comment here doesn't quite make sense to me since
you're saying you're trying to detect whether Hadoop >= 2.1 or not, but then
say that somehow 2.0-based versions are a special case? that soudns like it
works as intended.
So, are you trying to use "major version = 2" to distinguish between Hadoop
2.3+ and Hadoop 2.2 or earlier? That's the problem, of course: Hadoop 2.0-2.2
are not Hadoop 2.3 but have a major version of 2.
Do you mean to just check "major version = 2 and minor version = 3" or
something? why not do that? or check for some method that the shim for 2.3
expects?
Or, are the shims misnamed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
