Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20460#discussion_r165221502
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -267,7 +267,11 @@ private[deploy] class SparkSubmitArguments(args:
Seq[String], env: Map[String, S
&& Try(JavaUtils.byteStringAsBytes(executorMemory)).getOrElse(-1L)
<= 0) {
SparkSubmit.printErrorAndExit("Executor Memory cores must be a
positive number")
}
- if (executorCores != null && Try(executorCores.toInt).getOrElse(-1) <=
0) {
+ // Kubernetes mode allows fractional values for spark.executor.cores
so bypass this check in
+ // the Kubernetes mode.
+ if (!master.startsWith("k8s")
+ && executorCores != null
+ && Try(executorCores.toInt).getOrElse(-1) <= 0) {
--- End diff --
If the intent here is just to check that it's positive, then just try to
get it as a double and check it. This doesn't require making the logic
dependent on a particular resource manager. Of course it relaxes the condition
from what it is now, which would reject input like "1.5". But it would reject
it not with a graceful exit but an exception, so I kind of think it's not what
this code intends to catch anyway right now. (It would cause an error pretty
quickly in YARN et al anyway)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
