FMX commented on code in PR #3297:
URL: https://github.com/apache/celeborn/pull/3297#discussion_r2115031577
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -974,11 +976,14 @@ private[celeborn] class Master(
.asScala.map { case (worker, slots) => worker.toUniqueId -> slots
}.asJava,
requestSlots.requestId)
- logInfo(s"Offer slots successfully for $numReducers reducers of
$shuffleKey" +
- s" on ${slots.size()} workers.")
-
+ var offerSlotsMsg = s"Successfully offered slots for $numReducers reducers
of $shuffleKey" +
+ s" on ${slots.size()} workers"
val workersNotSelected =
availableWorkers.asScala.filter(!slots.containsKey(_))
- val offerSlotsExtraSize = Math.min(conf.masterSlotAssignExtraSlots,
workersNotSelected.size)
+ val offerSlotsExtraSize = Math.min(
+ Math.max(
+ slotsAssignExtraSlots,
+ slots.size() - slotsAssignMinWorkers),
Review Comment:
This line contains a logic error. You should use `slotsAssignMinWorkers -
slots.size()` to make sure that a shuffle will involve adequate workers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
