dongjoon-hyun commented on code in PR #59126:
URL: https://github.com/apache/spark/pull/59126#discussion_r4130150292
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorRollPlugin.scala:
##########
@@ -135,6 +135,8 @@ class ExecutorRollDriverPlugin extends DriverPlugin with
Logging {
listWithoutDriver.sortBy(_.totalShuffleWrite).reverse
case ExecutorRollPolicy.DISK_USED =>
listWithoutDriver.sortBy(_.diskUsed).reverse
+ case ExecutorRollPolicy.ACTIVE_TASKS =>
Review Comment:
Thank you, @sarutak. I added a comment about the tie-breaker.
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala:
##########
@@ -261,6 +261,8 @@ private[spark] object Config extends Logging {
"off-heap memory. " +
"TOTAL_SHUFFLE_WRITE policy chooses an executor with the biggest total
shuffle write. " +
"DISK_USED policy chooses an executor with the biggest used disk size.
" +
+ "ACTIVE_TASKS policy chooses an executor with the smallest number of
active tasks. " +
Review Comment:
Thank you. Since a newly started executor usually has no active tasks, I
updated the `ACTIVE_TASKS` description to recommend using this policy together
with `spark.kubernetes.executor.minTasksPerExecutorBeforeRolling`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
