ruiliang created SPARK-51435:
--------------------------------
Summary: spark submit executor jvm Xmx and yarn container
container_memory Whether it can be set separately
Key: SPARK-51435
URL: https://issues.apache.org/jira/browse/SPARK-51435
Project: Spark
Issue Type: Question
Components: Spark Submit
Affects Versions: 3.2.1
Reporter: ruiliang
spark submit executor jvm Xmx and yarn container container_memory Whether it
can be set separately
yarn submit doc
[YarnApplications submit
args|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html#:~:text=//%20Set%20Xmx%20based%20on%20am%20memory%20size%0Avargs.add(%22%2DXmx%22%20%2B%20amMemory%20%2B%20%22m%22)%3B%0A//%20Set%20class%20name%0Avargs.add(appMasterMainClass)%3B%0A//%20Set%20params%20for%20Application%20Master%0Avargs.add(%22%2D%2Dcontainer_memory%20%22%20%2B%20String.valueOf(containerMemory))%3B]
h2. [YARN Elastic Memory Feature
|https://hadoop.apache.org/docs/r3.3.0/hadoop-yarn/hadoop-yarn-site/NodeManagerCGroupsMemory.html#:~:text=Elastic%20Memory%20Feature,the%20OOM%20condition]
container Elastic Memory is required but spark submit executor defaults to
set-Xmx resulting in jvm oom
My needs
https://issues.apache.org/jira/projects/YARN/issues/YARN-11789?filter=reportedbyme
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
