GitHub user patmcdonough reopened a pull request:
https://github.com/apache/spark/pull/377
[WIP] Spark-1392: Add parameter to reserve minimum memory for the system
and increase default executor memory
* create a new property that sets the minimum amount of heap reserved for
the system/application: _spark.system.memoryReservedSize_
* account for that value prior to calculating memory available for storage
and shuffle
* set the new property at 300m by default (based on what we are seeing in a
local spark-shell running JDK7 with the spark-0.9.0-hadoop-2 binary
distribution)
* increase the default spark.executor.memory beyond 512m since we are going
to reserve over half of that for spark itself
This is marked as WIP as testing is still in progress. I was still able to
cause an OOM by running a count distinct on the dataset linked to in the JIRA,
but bumping _spark.system.memoryReservedSize_ to 350m prevented it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/patmcdonough/spark reserved-memory
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/377.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #377
----
commit 89a7f8b910565dca34d0099cbf9f5f90d66afa91
Author: Pat McDonough <[email protected]>
Date: 2014-04-10T05:14:25Z
Addeded a new parameter, spark.system.reservedMemorySize to reserve a
portion of the heap for system objects, especilaly in the case of smaller heaps
(like the out of the box conifuration for the spark shell).
commit e8e0dd7c9e9563a6edd4a9df6e709aad05921795
Author: Pat McDonough <[email protected]>
Date: 2014-04-10T06:08:33Z
Increase the default for spark.executor.memory to 1g
commit d5c33650a5b94c4983cd9dedc15a886ebc846d7d
Author: Pat McDonough <[email protected]>
Date: 2014-04-11T00:16:50Z
Updating styles and docs per suggestions in the PR
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
