GitHub user mallman opened a pull request:
https://github.com/apache/spark/pull/17390
[SPARK-17204][CORE] Fix replicated off heap storage
(Jira: https://issues.apache.org/jira/browse/SPARK-17204)
There are a couple of bugs in the `BlockManager` with respect to support
for replicated off-heap storage. First, the locally-stored off-heap byte buffer
is disposed of when it is replicated. It should not be. Second, the replica
byte buffers are stored as heap byte buffers instead of direct byte buffers
even when the storage level memory mode is off-heap. This PR addresses both of
these problems.
`BlockManagerReplicationSuite` was enhanced to fill in the coverage gaps.
It now fails if either of the bugs in this PR exist.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/VideoAmp/spark-public
spark-17204-replicated_off_heap_storage-2.0_backport
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17390.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17390
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
