GitHub user aarondav opened a pull request:
https://github.com/apache/spark/pull/2742
[SPARK-3889] Attempt to avoid SIGBUS by not mmapping files in
ConnectionManager
In general, individual shuffle blocks are frequently small, so mmapping
them often creates a lot of waste. It may not be bad to mmap the larger ones,
but it is pretty inconvenient to get configuration into ManagedBuffer, and
besides it is unlikely to help all that much.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aarondav/spark mmap
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2742.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2742
----
commit 52b6cd2c2038c56014e053751090085aa79c6a74
Author: Aaron Davidson <[email protected]>
Date: 2014-10-10T00:40:04Z
[SPARK-3889] Attempt to avoid SIGBUS by not mmapping files in
ConnectionManager
In general, individual shuffle blocks are frequently small, so mmapping them
often creates a lot of waste. It may not be bad to mmap the larger ones, but
it is pretty inconvenient to get configuration into ManagedBuffer, and
besides
it is unlikely to help all that much.
Note that user of ManagedBuffer#nioByteBuffer() seems generally bad
practice,
and would ideally never be used for data that may be large. Users of such
data
would ideally stream the data instead.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
