GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/6627
[SPARK-8065] [hive] Add support for Hive 0.14 metastores
This change has two parts.
The first one gets rid of "ReflectionMagic". That worked well for the
differences between 0.12 and
0.13, but breaks in 0.14, since some of the APIs that need to be used have
primitive types. I could
not figure out a way to make that class work with primitive types. So
instead I wrote some shims
(I can already hear the collective sigh) that find the appropriate methods
via reflection. This should
be faster since the method instances are cached, and the code is not much
uglier than before,
with the advantage that all the ugliness is local to one file (instead of
multiple switch statements on
the version being used scattered in ClientWrapper).
The second part is simple: add code to handle Hive 0.14. A few new methods
had to be added
to the new shims.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-8065
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6627.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6627
----
commit ae98c876da5679347a6878f92d48ea2298107eca
Author: Marcelo Vanzin <[email protected]>
Date: 2015-06-03T18:06:02Z
PART I: Get rid of reflection magic.
The reflection magic stuff in sql/hive does not work with primitive
Java types. Lots of APIs added in Hive 0.14 use primitive types in
their signature, so there's no way to make them work with that code.
Instead, have explicit shim-like classes that use reflection to call
into actual Hive code. Each version has its own version-specific
implementations of each needed method.
commit a21cad8ede9d37cb93d765efc2c8c02c9fa1e3e8
Author: Marcelo Vanzin <[email protected]>
Date: 2015-06-03T23:10:37Z
Part II: Add shims / version for Hive 0.14.
Some dependency mess needed to be sorted out; in the process I removed the
unused constants for v10 and v11 (which, if desired, can be restored later
when someone decides to write shims for them).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
