Repository: spark Updated Branches: refs/heads/master 1d95fb678 -> 496d2a2b4
[SPARK-13889][YARN] Fix integer overflow when calculating the max number of executor failure ## What changes were proposed in this pull request? The max number of executor failure before failing the application is default to twice the maximum number of executors if dynamic allocation is enabled. The default value for "spark.dynamicAllocation.maxExecutors" is Int.MaxValue. So this causes an integer overflow and a wrong result. The calculated value of the default max number of executor failure is 3. This PR adds a check to avoid the overflow. ## How was this patch tested? It tests if the value is greater that Int.MaxValue / 2 to avoid the overflow when it multiplies 2. Author: Carson Wang <carson.w...@intel.com> Closes #11713 from carsonwang/IntOverflow. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/496d2a2b Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/496d2a2b Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/496d2a2b Branch: refs/heads/master Commit: 496d2a2b403ac83b390a90519217dd310b0013a4 Parents: 1d95fb6 Author: Carson Wang <carson.w...@intel.com> Authored: Wed Mar 16 10:56:01 2016 +0000 Committer: Sean Owen <so...@cloudera.com> Committed: Wed Mar 16 10:56:01 2016 +0000 ---------------------------------------------------------------------- .../scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/496d2a2b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---------------------------------------------------------------------- diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala index cd179cf..a06e677 100644 --- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala +++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala @@ -73,7 +73,10 @@ private[spark] class ApplicationMaster( } else { sparkConf.get(EXECUTOR_INSTANCES).getOrElse(0) } - val defaultMaxNumExecutorFailures = math.max(3, 2 * effectiveNumExecutors) + // By default, effectiveNumExecutors is Int.MaxValue if dynamic allocation is enabled. We need + // avoid the integer overflow here. + val defaultMaxNumExecutorFailures = math.max(3, + if (effectiveNumExecutors > Int.MaxValue / 2) Int.MaxValue else (2 * effectiveNumExecutors)) sparkConf.get(MAX_EXECUTOR_FAILURES).getOrElse(defaultMaxNumExecutorFailures) } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
