dongjoon-hyun commented on code in PR #44906:
URL: https://github.com/apache/spark/pull/44906#discussion_r1468233367
##########
core/src/main/scala/org/apache/spark/deploy/master/Master.scala:
##########
@@ -824,9 +827,15 @@ private[deploy] class Master(
// If the cores left is less than the coresPerExecutor,the cores left
will not be allocated
if (app.coresLeft >= coresPerExecutor) {
// Filter out workers that don't have enough resources to launch an
executor
- val usableWorkers = workers.toArray.filter(_.state ==
WorkerState.ALIVE)
+ val aliveWorkers = workers.toArray.filter(_.state ==
WorkerState.ALIVE)
.filter(canLaunchExecutor(_, resourceDesc))
- .sortBy(_.coresFree).reverse
+ val usableWorkers = workerSelectionPolicy match {
+ case CORES_FREE_ASC => aliveWorkers.sortBy(w => (w.coresFree,
w.id))
+ case CORES_FREE_DESC => aliveWorkers.sortBy(w => (w.coresFree,
w.id)).reverse
Review Comment:
Ya, the ID is added to be deterministic. Previously, it's based on HashSet
and indeterministic among `coresFree`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
