GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/9131
[SPARK-11119][SQL] cleanup for unsafe array and map
The purpose of this PR is to keep the unsafe format detail only inside the
unsafe class itself, so when we use them(like use unsafe array in unsafe map,
use unsafe array and map in columnar cache), we don't need to understand the
format before use them.
change list:
* unsafe array's 4-bytes numElements header is now required(was optional),
and become a part of unsafe array format.
* w.r.t the previous changing, the `sizeInBytes` of unsafe array now counts
the 4-bytes header.
* unsafe map's format was `[numElements] [key array numBytes] [key array
content(without numElements header)] [value array content(without numElements
header)]` before, which is a little hacky as it makes unsafe array's header
optional. I think saving 4 bytes is not a big deal, so the format is now: `[key
array numBytes] [unsafe key array] [unsafe value array]`.
* w.r.t the previous changing, the `sizeInBytes` of unsafe map now counts
both map's header and array's header.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark unsafe
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9131.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9131
----
commit 4ada41e31b61e16aceeb1b5baf58c4b74a1d21c4
Author: Wenchen Fan <[email protected]>
Date: 2015-10-14T23:40:06Z
cleanup for unsafe array and map
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
