Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22298#discussion_r214405499
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
---
@@ -58,6 +58,15 @@ private[spark] class BasicExecutorFeatureStep(
(kubernetesConf.get(MEMORY_OVERHEAD_FACTOR) *
executorMemoryMiB).toInt,
MEMORY_OVERHEAD_MIN_MIB))
private val executorMemoryWithOverhead = executorMemoryMiB +
memoryOverheadMiB
+ // TODO: Have memory limit checks on executorMemory
+ private val executorMemoryTotal = kubernetesConf.sparkConf
+ .getOption(APP_RESOURCE_TYPE.key).map{ res =>
+ val additionalPySparkMemory = if (res == "python") {
--- End diff --
So this means that we couldn't turn this on in a mixed language pipeline
even if the pipeline author did some fun hacks (since we might need to support
Scala/Python/R all at the same time). Is there a reason we did this instead of
the YARN approach of isPython ?
Also I'd write this as a case statement instead perhaps style wise, what do
you think?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
