GitHub user Devian-ua opened a pull request:
https://github.com/apache/spark/pull/15906
[YARN][DOC] Increasing NodeManager's heap size with External Shuffle Service
## What changes were proposed in this pull request?
Suggest users to increase `NodeManager's` heap size if `External Shuffle
Service` is enabled as
`NM` can spend a lot of time doing GC resulting in shuffle operations
being a bottleneck due to `Shuffle Read blocked time` bumped up.
Also because of GC `NodeManager` can use an enormous amount of CPU and
cluster performance will suffer.
I have seen NodeManager using 5-13G RAM and up to 2700% CPU with
`spark_shuffle` service on.
## How was this patch tested?
#### Added step 5:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Devian-ua/spark nmHeapSize
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15906.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15906
----
commit 85f9fbbc11b7573fe1170e394ea6f70b39e7fdd6
Author: Artur Sukhenko <[email protected]>
Date: 2016-11-16T16:08:56Z
[YARN][DOC] Suggest increasing NodeManager's heap size with External
Shuffle Service
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
