zaynt4606 commented on code in PR #3775:
URL: https://github.com/apache/celeborn/pull/3775#discussion_r3726533606
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -1062,34 +1068,116 @@ private[celeborn] class Master(
s"extraSlots=$offerSlotsExtraSize"))
if (authEnabled) {
- pushApplicationMetaToWorkers(requestSlots, slots)
+ pushApplicationMetaToWorkers(requestSlots.applicationId, slots.keySet())
}
context.reply(RequestSlotsResponse(
StatusCode.SUCCESS,
slots.asInstanceOf[WorkerResource],
requestSlots.packed))
}
+ private def selectWorkersForRequest(
+ requestWorkers: PbRequestWorkers,
+ availableWorkers: util.List[WorkerInfo]): util.List[WorkerInfo] = {
+ val maxWorkers =
+ if (requestWorkers.getMaxWorkers <= 0) splitSlotAssignMaxWorkers
+ else Math.min(splitSlotAssignMaxWorkers, requestWorkers.getMaxWorkers)
+ val storageType = StorageInfo.typesMap.get(requestWorkers.getStorageType)
+ if (storageType == null) {
+ return Collections.emptyList()
+ }
+ val eligibleWorkers = new util.ArrayList[WorkerInfo]()
+ availableWorkers.asScala
+ .filter { worker =>
+ (storageType != StorageInfo.Type.HDD && storageType !=
StorageInfo.Type.SSD) ||
+ worker.haveDisk
+ }
+ .foreach(eligibleWorkers.add)
+ if (eligibleWorkers.isEmpty) {
+ return Collections.emptyList()
+ }
+
+ val minWorkers = if (requestWorkers.getShouldReplicate) 2 else 1
+ val selectedWorkerCount =
+ Math.min(Math.max(minWorkers, maxWorkers), eligibleWorkers.size)
+ val startIndex = Random.nextInt(eligibleWorkers.size)
Review Comment:
Minor: `scala.util.Random` is thread-safe but uses internal synchronization,
which can cause contention under concurrent Master requests. Since
`handleRequestWorkers` is throttled to 30s per client the practical impact is
low, but if the number of active applications grows, consider using
`ThreadLocalRandom.current().nextInt(eligibleWorkers.size)` for better
concurrency. The rest of the file already uses `Random.nextInt` (lines 972,
1051) so this is consistent with existing code — not blocking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
