[
https://issues.apache.org/jira/browse/SPARK-2444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14058909#comment-14058909
]
Sean Owen commented on SPARK-2444:
----------------------------------
In SPARK-2398, YARN was allocating 32.5GB for a 32GB executor and that's not
nearly enough headroom. This param is one mechanism that builds in some
padding. But it's fairly small.
I'm used to having to set Xmx to some fraction of the YARN container size when
running Java processes in YARN, because the physical memory required by the
heap is bigger than the Java heap size by some ratio, not just some fixed
overhead. A 32GB heap can eat significantly more than 32GB of physical memory,
more than the ~384MB extra buffer built in by default.
So this is me being ignorant, but, was that not already factored in somewhere
by Spark-on-YARN, as a multiplier? or am I mixing this up with some other
system?
> Make spark.yarn.executor.memoryOverhead a first class citizen
> -------------------------------------------------------------
>
> Key: SPARK-2444
> URL: https://issues.apache.org/jira/browse/SPARK-2444
> Project: Spark
> Issue Type: Improvement
> Components: Documentation
> Affects Versions: 1.0.0
> Reporter: Nishkam Ravi
>
> Higher value of spark.yarn.executor.memoryOverhead is critical to running
> Spark applications on Yarn (https://issues.apache.org/jira/browse/SPARK-2398)
> at least for 1.0. It would be great to have this parameter highlighted in the
> docs/usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
