GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/935
Avoid dynamic dispatching when unwrapping Hive data.
This is a follow up of PR #758.
The `unwrapHiveData` function is now composed statically before actual rows
are scanned according to the field object inspector to avoid dynamic
dispatching cost.
According to the same micro benchmark used in PR #758, this simple change
brings slight performance boost: 2.5% for CSV table and 1% for RCFile table.
```
Optimized version:
CSV: 6870 ms, RCFile: 5687 ms
CSV: 6832 ms, RCFile: 5800 ms
CSV: 6822 ms, RCFile: 5679 ms
CSV: 6704 ms, RCFile: 5758 ms
CSV: 6819 ms, RCFile: 5725 ms
Original version:
CSV: 7042 ms, RCFile: 5667 ms
CSV: 6883 ms, RCFile: 5703 ms
CSV: 7115 ms, RCFile: 5665 ms
CSV: 7020 ms, RCFile: 5981 ms
CSV: 6871 ms, RCFile: 5906 ms
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark staticUnwrapping
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/935.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #935
----
commit c49c70ca1465bb1cf755f03d55ee5eba27325a5b
Author: Cheng Lian <[email protected]>
Date: 2014-06-01T08:58:52Z
Avoid dynamic dispatching when unwrapping Hive data.
Conflicts:
sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveOperators.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
