GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/4827
[SPARK-6076][Block Manager] Fix a potential OOM issue when StorageLevel is
MEMORY_AND_DISK_SER
In
https://github.com/apache/spark/blob/dcd1e42d6b6ac08d2c0736bf61a15f515a1f222b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L538
, when StorageLevel is `MEMORY_AND_DISK_SER`, it will copy the content from
file into memory, then put it into MemoryStore.
```
scala
val copyForMemory = ByteBuffer.allocate(bytes.limit)
copyForMemory.put(bytes)
memoryStore.putBytes(blockId, copyForMemory, level)
bytes.rewind()
```
However, if the file is bigger than the free memory, OOM will happen. A
better approach is testing if there is enough memory. If not, copyForMemory
should not be created, since this is an option operation.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark SPARK-6076
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4827.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4827
----
commit 0cc0257769070843e05ec010220a78827001f519
Author: zsxwing <[email protected]>
Date: 2015-02-28T06:49:32Z
Fix a potential OOM issue when StorageLevel is MEMORY_AND_DISK_SER
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
