GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/14927
[SPARK-16922] [SPARK-17211] [SQL] make the address of values portable in
LongToUnsafeRowMap
## What changes were proposed in this pull request?
In LongToUnsafeRowMap, we use offset of a value as pointer, stored in a
array also in the page for chained values. The offset is not portable, because
Platform.LONG_ARRAY_OFFSET will be different with different JVM Heap size, then
the deserialized LongToUnsafeRowMap will be corrupt.
This PR will change to use portable address (without
Platform.LONG_ARRAY_OFFSET).
## How was this patch tested?
Added a test case with random generated keys, to improve the coverage. But
this test is not a regression test, that could require a Spark cluster that
have at least 32G heap in driver or executor.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark longmap
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14927.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14927
----
commit 0c8450cee37af734a9b6d45c7dc7962e5436cba4
Author: Davies Liu <[email protected]>
Date: 2016-09-01T20:50:48Z
make the address of values portable
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
