Repository: spark Updated Branches: refs/heads/branch-2.3 ca27d9cb5 -> 801ffd799
[SPARK-22870][CORE] Dynamic allocation should allow 0 idle time ## What changes were proposed in this pull request? This pr to make `0` as a valid value for `spark.dynamicAllocation.executorIdleTimeout`. For details, see the jira description: https://issues.apache.org/jira/browse/SPARK-22870. ## How was this patch tested? N/A Author: Yuming Wang <yumw...@ebay.com> Author: Yuming Wang <wgy...@gmail.com> Closes #20080 from wangyum/SPARK-22870. (cherry picked from commit fc6fe8a1d0f161c4788f3db94de49a8669ba3bcc) Signed-off-by: Sean Owen <so...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/801ffd79 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/801ffd79 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/801ffd79 Branch: refs/heads/branch-2.3 Commit: 801ffd799922e1c2751d3331874b88a67da8cf01 Parents: ca27d9c Author: Yuming Wang <yumw...@ebay.com> Authored: Sat Jan 13 10:01:44 2018 -0600 Committer: Sean Owen <so...@cloudera.com> Committed: Sat Jan 13 10:01:53 2018 -0600 ---------------------------------------------------------------------- .../scala/org/apache/spark/ExecutorAllocationManager.scala | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/801ffd79/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala index 2e00dc8..6c59038 100644 --- a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala +++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala @@ -195,8 +195,11 @@ private[spark] class ExecutorAllocationManager( throw new SparkException( "spark.dynamicAllocation.sustainedSchedulerBacklogTimeout must be > 0!") } - if (executorIdleTimeoutS <= 0) { - throw new SparkException("spark.dynamicAllocation.executorIdleTimeout must be > 0!") + if (executorIdleTimeoutS < 0) { + throw new SparkException("spark.dynamicAllocation.executorIdleTimeout must be >= 0!") + } + if (cachedExecutorIdleTimeoutS < 0) { + throw new SparkException("spark.dynamicAllocation.cachedExecutorIdleTimeout must be >= 0!") } // Require external shuffle service for dynamic allocation // Otherwise, we may lose shuffle files when killing executors --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
