waitinfuture commented on code in PR #1840:
URL:
https://github.com/apache/incubator-celeborn/pull/1840#discussion_r1306341813
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java:
##########
@@ -161,7 +162,8 @@ public void updateWorkerHeartbeatMeta(
}
appDiskUsageMetric.update(estimatedAppDiskUsage);
// If using HDFSONLY mode, workers with empty disks should not be put into
excluded worker list.
- if (!excludedWorkers.contains(worker) && (disks.isEmpty() &&
!conf.hasHDFSStorage())) {
+ if (!excludedWorkers.contains(worker)
+ && ((disks.isEmpty() && !conf.hasHDFSStorage()) || highWorkload)) {
Review Comment:
> this may cause HDFS-only workers to never back?
Nice catch!
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -2693,6 +2694,16 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(false)
+ val WORKER_ACTIVE_CONNECTION_MAX: OptionalConfigEntry[Long] =
+ buildConf("celeborn.worker.activeConnection.max")
+ .categories("worker")
+ .doc("If the number of active connections on a worker exceeds this
configuration value, " +
+ "the worker will be marked as high-load in the heartbeat report, " +
+ "and the master will not include that node in the response of
RequestSlots.")
+ .version("0.3.0")
Review Comment:
Should be 0.3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
