GitHub user mallman opened a pull request:
https://github.com/apache/spark/pull/16499
[SPARK-17204][CORE] Fix replicated off heap storage
(Jira: https://issues.apache.org/jira/browse/SPARK-17204)
## What changes were proposed in this pull request?
There are a couple of bugs in the `BlockManager` with respect to support
for replicated off-heap storage. First, the locally-stored off-heap byte buffer
is disposed of when it is replicated. It should not be. Second, the replica
byte buffers are stored as heap byte buffers instead of direct byte buffers
even when the storage level memory mode is off-heap. This PR addresses both of
these problems.
## How was this patch tested?
`BlockManagerReplicationSuite` was enhanced to fill in the coverage gaps.
It now fails if either of the bugs in this PR exist.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/VideoAmp/spark-public
spark-17204-replicated_off_heap_storage
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16499.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16499
----
commit e49aeca23ff463fbd9a9cc4db99078c466bfbd56
Author: Michael Allman <[email protected]>
Date: 2017-01-02T01:28:12Z
Fix a couple of bugs in replicated off-heap storage
commit 40b6b97ca9013544702433dc5dc388c054daf41a
Author: Michael Allman <[email protected]>
Date: 2017-01-07T21:17:33Z
Shore-up BlockManagerReplicationSuite to identify a couple of bugs with
off-heap storage replication
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
