Github user ifilonenko commented on a diff in the pull request:
https://github.com/apache/spark/pull/22298#discussion_r214411255
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
---
@@ -58,6 +58,15 @@ private[spark] class BasicExecutorFeatureStep(
(kubernetesConf.get(MEMORY_OVERHEAD_FACTOR) *
executorMemoryMiB).toInt,
MEMORY_OVERHEAD_MIN_MIB))
private val executorMemoryWithOverhead = executorMemoryMiB +
memoryOverheadMiB
+ // TODO: Have memory limit checks on executorMemory
+ private val executorMemoryTotal = kubernetesConf.sparkConf
+ .getOption(APP_RESOURCE_TYPE.key).map{ res =>
+ val additionalPySparkMemory = if (res == "python") {
--- End diff --
Well, if there is a mixed pipeline, the binding steps would need to be
reconfigured to include mixed pipelines. But currently, language is only
determined based on the `appResource`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
