FMX commented on code in PR #2071:
URL:
https://github.com/apache/incubator-celeborn/pull/2071#discussion_r1382919284
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -121,15 +121,7 @@ private[celeborn] class Master(
private def shutdownWorkerSnapshot: util.List[WorkerInfo] =
statusSystem.workers.synchronized(new
util.ArrayList[WorkerInfo](statusSystem.shutdownWorkers))
- private def diskReserveSize = conf.workerDiskReserveSize
-
private val slotsAssignMaxWorkers = conf.masterSlotAssignMaxWorkers
- private val slotsAssignLoadAwareDiskGroupNum =
conf.masterSlotAssignLoadAwareDiskGroupNum
- private val slotsAssignLoadAwareDiskGroupGradient =
- conf.masterSlotAssignLoadAwareDiskGroupGradient
- private val loadAwareFlushTimeWeight =
conf.masterSlotAssignLoadAwareFlushTimeWeight
- private val loadAwareFetchTimeWeight =
conf.masterSlotAssignLoadAwareFetchTimeWeight
Review Comment:
Access celeborn conf only once will have better performance.
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java:
##########
@@ -135,13 +137,19 @@ static class UsableDiskInfo {
workers, partitionIds, shouldReplicate, shouldRackAware,
availableStorageTypes);
}
+ int diskGroupCount = conf.masterSlotAssignLoadAwareDiskGroupNum();
+
if (!initialized) {
- initLoadAwareAlgorithm(diskGroupCount, diskGroupGradient);
+ initLoadAwareAlgorithm(diskGroupCount,
conf.masterSlotAssignLoadAwareDiskGroupGradient());
}
Map<WorkerInfo, List<UsableDiskInfo>> restrictions =
getRestriction(
- placeDisksToGroups(usableDisks, diskGroupCount, flushTimeWeight,
fetchTimeWeight),
+ placeDisksToGroups(
+ usableDisks,
+ diskGroupCount,
+ conf.masterSlotAssignLoadAwareFlushTimeWeight(),
+ conf.masterSlotAssignLoadAwareFetchTimeWeight()),
Review Comment:
Getting a config value from Celeborn Conf is expensive. Save the value and
reuse would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
