[
https://issues.apache.org/jira/browse/SPARK-2444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14058930#comment-14058930
]
Thomas Graves commented on SPARK-2444:
--------------------------------------
Maybe I'm missing something in what you are asking. The default is 384MB and
you can set the overhead to be larger then that with the config
spark.yarn.executor.memoryOverhead. Then you will get Xmx as what you specify
(32GB) and a container size of what you specify +
spark.yarn.executor.memoryOverhead. Different applications could have very
different values for this that is why we decided to let the user set it.
In other things on Yarn you have to specify both of these. On MapReduce for
instance, you actually specify the container size and then specify the Xmx
separately. Here with Spark on Yarn it is the inverse based on what the
community thought was easier.
> Make spark.yarn.executor.memoryOverhead a first class citizen
> -------------------------------------------------------------
>
> Key: SPARK-2444
> URL: https://issues.apache.org/jira/browse/SPARK-2444
> Project: Spark
> Issue Type: Improvement
> Components: Documentation
> Affects Versions: 1.0.0
> Reporter: Nishkam Ravi
>
> Higher value of spark.yarn.executor.memoryOverhead is critical to running
> Spark applications on Yarn (https://issues.apache.org/jira/browse/SPARK-2398)
> at least for 1.0. It would be great to have this parameter highlighted in the
> docs/usage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
