Sure Reynold, Here is pull request - [YARN][DOC] Increasing NodeManager's heap size with External Shuffle Service <https://github.com/apache/spark/pull/15906>
On Wed, Nov 16, 2016, 04:07 Reynold Xin <r...@databricks.com> wrote: Can you submit a pull request to add that to the documentation? On November 15, 2016 at 12:45:57 PM, Artur Sukhenko ( artur.sukhe...@gmail.com) wrote: Hello guys, When you enable ExternalShuffleService (spark-shuffle) in NodeManager, there are no suggestions of increasing NM heap size in Spark docs or anywhere else, shouldn't we include this in spark's documentation? I have seen NM take a lot of memory 5+ gb with default 1g, and in case of its GC pauses spark can become very slow when tasks are doing shuffle. I don't think users are aware of NM becoming bottleneck. Sincerely, Artur Sukhenko -- -- Artur Sukhenko -- -- Artur Sukhenko
