This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f2adb61 [SPARK-26446][CORE] Add cachedExecutorIdleTimeout docs at
ExecutorAllocationManager
f2adb61 is described below
commit f2adb610680f9addec51bf470acf64f3849073e9
Author: wuqingxin <[email protected]>
AuthorDate: Fri Dec 28 00:15:57 2018 -0800
[SPARK-26446][CORE] Add cachedExecutorIdleTimeout docs at
ExecutorAllocationManager
## What changes were proposed in this pull request?
Add docs to describe how remove policy act while considering the property
`spark.dynamicAllocation.cachedExecutorIdleTimeout` in ExecutorAllocationManager
## How was this patch tested?
comment-only PR.
Closes #23386 from TopGunViper/SPARK-26446.
Authored-by: wuqingxin <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../scala/org/apache/spark/ExecutorAllocationManager.scala | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
index c3e5b96..3f0b71b 100644
--- a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
+++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
@@ -57,7 +57,8 @@ import org.apache.spark.util.{Clock, SystemClock,
ThreadUtils, Utils}
* a long time to ramp up under heavy workloads.
*
* The remove policy is simpler: If an executor has been idle for K seconds,
meaning it has not
- * been scheduled to run any tasks, then it is removed.
+ * been scheduled to run any tasks, then it is removed. Note that an executor
caching any data
+ * blocks will be removed if it has been idle for more than L seconds.
*
* There is no retry logic in either case because we make the assumption that
the cluster manager
* will eventually fulfill all requests it receives asynchronously.
@@ -81,7 +82,12 @@ import org.apache.spark.util.{Clock, SystemClock,
ThreadUtils, Utils}
* This is used only after the initial backlog timeout is exceeded
*
* spark.dynamicAllocation.executorIdleTimeout (K) -
- * If an executor has been idle for this duration, remove it
+ * If an executor without caching any data blocks has been idle for this
duration, remove it
+ *
+ * spark.dynamicAllocation.cachedExecutorIdleTimeout (L) -
+ * If an executor with caching data blocks has been idle for more than
this duration,
+ * the executor will be removed
+ *
*/
private[spark] class ExecutorAllocationManager(
client: ExecutorAllocationClient,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
