Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/22897#discussion_r230115469
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala
---
@@ -58,16 +58,13 @@ private[spark] class BasicExecutorFeatureStep(
(kubernetesConf.get(MEMORY_OVERHEAD_FACTOR) *
executorMemoryMiB).toInt,
MEMORY_OVERHEAD_MIN_MIB))
private val executorMemoryWithOverhead = executorMemoryMiB +
memoryOverheadMiB
- private val executorMemoryTotal = kubernetesConf.sparkConf
- .getOption(APP_RESOURCE_TYPE.key).map{ res =>
- val additionalPySparkMemory = res match {
- case "python" =>
- kubernetesConf.sparkConf
- .get(PYSPARK_EXECUTOR_MEMORY).map(_.toInt).getOrElse(0)
- case _ => 0
- }
- executorMemoryWithOverhead + additionalPySparkMemory
- }.getOrElse(executorMemoryWithOverhead)
+ private val executorMemoryTotal = kubernetesConf.get(APP_RESOURCE_TYPE)
match {
+ case Some("python") =>
--- End diff --
I was under the impression that generally we don' want to match against
option types - instead we should be using `option.map.getOrElse`? More just my
impression of the Scala idiomatic style than anything.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
