Hi all, I' d like to bring CIP-23 to the mailing list for wider discussion.
At peak, every worker/application heartbeat becomes an individual raft log entry on the master, and raft write volume becomes the bottleneck. CIP-23 aggregates heartbeats on the leader over a short window (default 1s) into a single BatchHeartbeat raft entry, cutting raft write volume by ~100x at peak. The change is additive and saft to roll out: a new BatchHeartbeatRequest proto, gated by `celeborn.master.ha.heartbeat.batch.enabled (off by default). The PR is up and has received some initial review: https://github.com/apache/celeborn/pull/3810 Design doc: https://docs.google.com/document/d/1YXDCjk_kR_5jyRGmUz1DqImjnAtsr7oFEwpYKK1NxAE/edit?usp=sharing Any feedback on the design is welcome -- here or on the PR. Thanks! Best regards, Hai Zhou
