yew1eb commented on code in PR #3839:
URL: https://github.com/apache/celeborn/pull/3839#discussion_r4236694162
##########
docs/deploy.md:
##########
@@ -112,6 +112,12 @@ celeborn.worker.replicate.fastFail.duration 240s
celeborn.worker.monitor.disk.enabled false
```
+Note: In an HA cluster, each master runs a Ratis JvmPauseMonitor. If the JVM
or the host is paused
+longer than `celeborn.master.ha.ratis.raft.server.close.threshold` (default
60s), the local raft
+server is closed and the master leaves the raft group; the process must be
restarted to recover.
+Avoid operations that suspend the JVM for a long time on production masters,
e.g. `jmap -F` or
+heap dumps, and prefer low-intrusion diagnostics like `jcmd <pid>
GC.class_histogram`.
Review Comment:
Fixed in c979047ce. deploy.md no longer recommends `jcmd GC.class_histogram`
as low-intrusion: it now states the command forces a full GC by default (the
`all=true` form skips the forced GC but still walks the whole heap) and
recommends `jstat -gcutil <pid> <interval>` for GC observation instead.
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/HARaftServer.java:
##########
@@ -374,6 +374,10 @@ private RaftProperties newRaftProperties(CelebornConf
conf, RpcType rpc) {
TimeDuration.valueOf(conf.haMasterRatisRpcRequestTimeout(),
TimeUnit.SECONDS);
RaftServerConfigKeys.Rpc.setRequestTimeout(properties,
serverRequestTimeout);
+ // Set the threshold for JvmPauseMonitor to close the local raft server
+ RaftServerConfigKeys.setCloseThreshold(
Review Comment:
Fixed in c979047ce. The dedicated config is now re-applied after the
`haRatisCustomConfigs()` passthrough loop, so it takes precedence when
explicitly set, and a warning is logged when both keys are configured. When
only the passthrough is set it still applies, so existing deployments keep
their behavior. The precedence is documented in the config doc (and regenerated
ha.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
