wang-zhun opened a new pull request #28069: [SPARK-31265][YARN] Add -XX:MaxDirectMemorySize jvm options in yarn mode URL: https://github.com/apache/spark/pull/28069 ### What changes were proposed in this pull request? Current memory composition `amMemory` + `amMemoryOverhead` ``` val capability = Records.newRecord(classOf[Resource]) capability.setMemory(amMemory + amMemoryOverhead) capability.setVirtualCores(amCores) if (amResources.nonEmpty) { ResourceRequestHelper.setResourceRequests(amResources, capability) } logDebug(s"Created resource capability for AM request: $capability") ... // Add Xmx for AM memory javaOpts += "-Xmx" + amMemory + "m" ``` It is possible that the physical memory of the container exceeds the limit and is killed by yarn. I suggest setting `-XX:MaxDirectMemorySize` here ``` // Add Xmx for AM memory javaOpts += s"-Xmx${amMemory}m" if (isClusterMode) { javaOpts += s"-XX:MaxDirectMemorySize=${amMemoryOverhead}m" } ``` ### How was this patch tested? Tested by verifying JVM parameters in the following cases: - client and cluster mode
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
