GitHub user mateiz opened a pull request:
https://github.com/apache/spark/pull/7586
[SPARK-9244] Increase some memory defaults
There are a few memory limits that people hit often and that we could
make higher, especially now that memory sizes have grown.
- spark.akka.frameSize: This defaults at 10 but is often hit for map
output statuses in large shuffles. This memory is not fully allocated
up-front, so we can just make this larger and still not affect jobs
that never sent a status that large. We increase it to 128.
- spark.executor.memory: Defaults at 512m, which is really small. We
increase it to 1g.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mateiz/spark configs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7586.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7586
----
commit ce0038acc0527495b94a19534ebd5c2b83a9ee19
Author: Matei Zaharia <[email protected]>
Date: 2015-07-22T06:13:54Z
[SPARK-9244] Increase some memory defaults
There are a few memory limits that people hit often and that we could
make higher, especially now that memory sizes have grown.
- spark.akka.frameSize: This defaults at 10 but is often hit for map
output statuses in large shuffles. This memory is not fully allocated
up-front, so we can just make this larger and still not affect jobs
that never sent a status that large. We increase it to 128.
- spark.executor.memory: Defaults at 512m, which is really small. We
increase it to 1g.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
