Kalvin2077 commented on code in PR #3858:
URL: https://github.com/apache/celeborn/pull/3858#discussion_r4438882209
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -843,6 +852,7 @@ private[celeborn] class Master(
s" for WorkerLost handler!")
} else {
statusSystem.handleWorkerLost(host, rpcPort, pushPort, fetchPort,
replicatePort, requestId)
+ workerStatsStore.remove(worker)
Review Comment:
@SteNicholas
Fixed through the metadata apply path. Worker loss now removes local
observations on every Master, including former leaders replaying the event as
followers. Snapshot restoration clears the local cache and creates fresh
references for restored members. Metrics remain outside Raft logs and metadata
snapshots. Added a three-node Ratis regression that transfers leadership from A
to B, removes the worker under B, then transfers leadership back to A. It
verifies cleanup on the follower and that updating the old reference cannot
restore the removed worker's observations.
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -759,6 +764,10 @@ private[celeborn] class Master(
highWorkload,
workerStatus,
requestId)
+ workerStats match {
+ case Some(stats) => workerStatsStore.update(targetWorker, stats)
Review Comment:
@SteNicholas
Thx for the reproduction. Each registration now creates its own
AtomicReference<WorkerStats> in the metadata manager. A heartbeat captures that
reference before metadata processing and only updates that reference afterward.
Removal or re-registration detaches the old reference from the map, so an
in-flight heartbeat cannot recreate a removed entry or overwrite or clear a
later registration's observations. Heartbeat updates no longer insert map
entries. Added deterministic tests that pause heartbeats at the membership lock
and exercise worker loss and re-registration, including heartbeats without
metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
