mridulm commented on a change in pull request #27313: [SPARK-29148][CORE] Add
stage level scheduling dynamic allocation and scheduler backend changes
URL: https://github.com/apache/spark/pull/27313#discussion_r375094408
##########
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##########
@@ -57,14 +59,17 @@ import org.apache.spark.util.{Clock, SystemClock,
ThreadUtils, Utils}
* quickly over time in case the maximum number of executors is very high.
Otherwise, it will take
* a long time to ramp up under heavy workloads.
*
- * The remove policy is simpler: If an executor has been idle for K seconds,
meaning it has not
- * been scheduled to run any tasks, then it is removed. Note that an executor
caching any data
+ * The remove policy is simpler: If an executor has been idle for K seconds
and the number of
+ * executors is more then what is needed, meaning there are not enough tasks
that could use
+ * the executor, then it is removed. Note that an executor caching any data
Review comment:
nit: This is per resource profile (and not global) right ? Can we add that
into the language ? The 'there are not enough tasks that could use the
executor' does imply it - but would be nice if we made it explicit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
