Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/894#discussion_r13751088
--- Diff:
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
---
@@ -88,6 +88,18 @@ private[yarn] class YarnAllocationHandler(
// Containers to be released in next request to RM
private val pendingReleaseContainers = new
ConcurrentHashMap[ContainerId, Boolean]
+ // Additional memory overhead - in mb.
+ private def memoryOverhead: Int = {
+ var defaultMemoryOverhead = YarnAllocationHandler.MEMORY_OVERHEAD
+ sparkConf.getOption("spark.yarn.container.memoryOverhead").foreach { s
=>
--- End diff --
Docs are pretty good, we may add a little bit to explain this memory
doesn't go into the heap. If you want I can help you with this.
And actually it looks like it doesn't properly add it as non heap memory
for yarn 2.4 and >. Would you mind fixing that here also. Look in the Client
for yarn stable at calculateAMMemory. We could also file a separate jira to
fix this.
As far as the 3 configs, I am actually thinking we don't really need the
general spark.yarn.container.memoryOverhead config though. We can do the same
thing with the other 2 (have them default to 384). Using just the 2 configs
follows the general pattern that other spark configs take they just have the
individual configs (like executor/driver configs). thoughts or objections?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
