GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/374
[WIP][BUGFIX] In-memory columnar storage bug fixes
Fixed several bugs in in-memory columnar storage to make the in-memory
version of `HiveCompatibilitySuite` pass (not accomplished yet, 6 failed and 4
wrong cases left being fixed). Compression is also disabled by default and can
be enabled by setting `spark.sql.inMemCompression.enabled` to `true`.
@rxin @marmbrus It is reasonable to include `InMemHiveCompatibilitySuite`
in this PR, but I didn't, since it significantly increases test execution time.
What do you think?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark inMemBugFix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/374.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #374
----
commit 8949e756c6f21cb210078a0a04ce93b6afa84113
Author: Cheng Lian <[email protected]>
Date: 2014-04-08T15:00:03Z
Fixed bugs in IntegralDelta
commit 140bf0b0087ac008ad43dfe462445aef9f75c601
Author: Cheng Lian <[email protected]>
Date: 2014-04-08T15:48:19Z
Bug fix: should only gather compressibility info for non-null values
commit 3982ef06c84ca86b0fdd3926883ef4a578bf6680
Author: Cheng Lian <[email protected]>
Date: 2014-04-09T10:57:26Z
Bug fix: NullableColumnAccessor.hasNext must take nulls into account
commit 6591440adde9a46a5dc6735f175840656e7e76a3
Author: Cheng Lian <[email protected]>
Date: 2014-04-09T10:59:06Z
Clean up unused imports
commit 15edf98d88aba91567afbcaea60a8e15a9e00076
Author: Cheng Lian <[email protected]>
Date: 2014-04-09T11:01:58Z
Bug fix: InMemoryColumnarTableScan should cache columns specified by the
attributes argument
commit 511c9f4d5aa0af8a53cf4384cda0b97a38c3169d
Author: Cheng Lian <[email protected]>
Date: 2014-04-09T11:16:58Z
Bug fix: incorrect byte order in CompressionScheme.columnHeaderSize
commit ac3a44af9822bd646d58f5605f337b5010801f6d
Author: Cheng Lian <[email protected]>
Date: 2014-04-09T22:01:56Z
Disable compression by default
commit 6a1117cc80c1ad376f1828d9e19bb8c093c44bab
Author: Cheng Lian <[email protected]>
Date: 2014-04-09T22:03:44Z
Let InsertIntoTable support InMemoryColumnarTableScan
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
