GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/4107
[SPARK-5325][SQL][WIP] Shrink the Hive shim layer
This PR shrinks the Hive shim layer significantly by the following means:
1. Re-implement some functions in a backwards compatible way, e.g. constant
object inspector methods in `WritableConstantObjectInspectors`, and the
`FileSinkDescWrapper` class
2. For some simple functions that are not performance critical,
re-implemented them with reflection tricks.
Most functions are moved from `HiveShim` to `HiveUtils` (maybe `HiveCompat`
is a better name). Constant object inspector related methods are moved to
`o.a.h.h.s.o.p.WritableConstantObjectInspectors` because we need to call
package private constructors.
Reflection utility function `callWithAlternativies` is introduced to
simplify reflection tricks. It tries to invoke all provided reflected
constructor calls, method calls, and static method calls one by one until one
of them succeeds. For example, `HiveDecimal` objects are created via normal
constructor in Hive 0.12.0, while `HiveDecimal.create` must be used in Hive
0.13.1. We can fix this by:
```scala
def createDecimal(bd: JBigDecimal) = callWithAlternatives(
// For Hive 0.12.0
InvokeStatic[HiveDecimal](classOf[HiveDecimal], "create",
classOf[JBigDecimal] -> bd),
// For Hive 0.13.1
Construct(classOf[HiveDecimal],
classOf[JBigDecimal] -> bd))
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark thin-hive-shim
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4107
----
commit 1a0a6cfdfd07135e8c42c36c77212aa9addbda6f
Author: Cheng Lian <[email protected]>
Date: 2015-01-19T21:28:17Z
Shrink the Hive shim layer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
