yew1eb opened a new pull request, #3810:
URL: https://github.com/apache/celeborn/pull/3810
### What changes were proposed in this pull request?
In HA mode, aggregate worker/application heartbeats on the master leader
over a short time window (default 1s) and replicate them as ONE
`BatchHeartbeat` raft log entry, instead of one entry per heartbeat.
- New `HeartbeatAggregator` on the leader: pending heartbeats are kept in
per-worker/per-app maps (newest wins, buffer intrinsically bounded), a
scheduled thread flushes one batch per window; heartbeat RPCs are enqueued
asynchronously and replied immediately from leader-local memory.
- New internal protobuf messages `BatchHeartbeatRequest` /
`PbBatchHeartbeatRequest` (`Type.BatchHeartbeat = 31`); `MetaHandler` expands
the batch and applies each child heartbeat through the same
`handleWorkerHeartbeat` / `handleAppHeartbeat` helpers as the single-entry path.
- Fully additive and off by default:
`celeborn.master.ha.heartbeat.batch.enabled` (false),
`celeborn.master.ha.heartbeat.batch.interval` (1s).
### Why are the changes needed?
Heartbeats are the most frequent metadata writes: A/10 + W/30 raft entries/s
for A apps and W workers. On one of our production clusters at peak (900+
running apps + 200 workers, ~97 heartbeat entries/s), follower apply lag
(`RatisApplyCompletedIndexDiff`) grows in lockstep with
`RunningApplicationCount` (~50 -> 1000). A follower flame graph shows the cost
is the per-entry raft pipeline (appendEntries ~35%, log flush/fsync ~20%), not
apply compute (~7.7%).
Heartbeats are periodic, self-healing, timeout-tolerant, and their replies
are built from leader-local memory — they do not need a synchronous raft commit
each. Batching cuts raft write volume by ~100x at peak.
### Does this PR resolve a correctness bug?
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
- [ ] Yes
### How was this patch tested?
- New `HeartbeatAggregatorSuiteJ` (single-node raft, full offer → submit →
replicate → apply chain): N offers produce far fewer entries; empty windows
produce none; duplicate keys within one window collapse to the newest heartbeat.
- `MasterStateMachineSuiteJ#testBatchHeartbeat`: batched entry applies
correctly and survives the `ResourceRequest` → `PbMetaRequest` wire round-trip.
- Existing master suites pass.
Design doc (CIP-22):
<https://docs.google.com/document/d/1YXDCjk_kR_5jyRGmUz1DqImjnAtsr7oFEwpYKK1NxAE/edit?tab=t.0#heading=h.998gtkpbnes6>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
