LucaCanali commented on a change in pull request #29477:
URL: https://github.com/apache/spark/pull/29477#discussion_r475437215
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
##########
@@ -59,7 +59,8 @@ private[spark] class BasicExecutorFeatureStep(
.getOrElse(math.max(
(kubernetesConf.get(MEMORY_OVERHEAD_FACTOR) * executorMemoryMiB).toInt,
MEMORY_OVERHEAD_MIN_MIB))
- private val executorMemoryWithOverhead = executorMemoryMiB +
memoryOverheadMiB
+ private val memoryOffHeapMiB =
KubernetesUtils.executorOffHeapMemorySizeAsMb(kubernetesConf)
+ private val executorMemoryWithOverhead = executorMemoryMiB +
memoryOverheadMiB + memoryOffHeapMiB
Review comment:
YARN:
The resource manager for YARN adds the offheap memory to the total memory
request. Which is consistent with what one finds in the documentation, in
particular the documentation for `spark.executor.memoryOverhead` states:
"The maximum memory size of container to running executor is determined by
the sum of `spark.executor.memoryOverhead`, `spark.executor.memory`,
`spark.memory.offHeap.size` and `spark.executor.pyspark.memory`. "
Mesos:
I don't work much with Mesos, but from what I can see:
- the documentation for `spark.executor.memoryOverhead` states that "This
option is currently supported on YARN and Kubernetes."
- From the code it looks like the memory overhead is implemeted, but just
the "standard part", without the off-heap and pyspark memory parts. It should
be easy to bring this up to speed with K8S and YARN, in case. Any thoughts on
this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
