Thanks, that's a clear and helpful breakdown. The push-for-liveness / pull-for-stats split makes sense: keeping a continuous in-memory pulse per worker in Nimbus (HeartbeatCache) while serving heavy stats on demand from ZK keeps Nimbus lightweight, and the Supervisor Thrift relay keeps those frequent liveness writes off ZK.
One thing I left out of my original question: the *supervisor liveness* heartbeat — which is also liveness, but it still goes to ZooKeeper (ephemeral znode at /supervisors/<id>, via SupervisorHeartbeat), consumed by Nimbus under nimbus.supervisor.timeout.secs. So "liveness" isn't uniformly on the Thrift path. My working reconciliation is that the deciding factor is *volume and semantics, not liveness-vs-stats*: - Worker/executor liveness is high-volume (potentially thousands of heartbeats per cluster, every ~1s) → moved off ZK onto the Supervisor Thrift relay to avoid write pressure. - Supervisor liveness is low-volume (one per node, every ~5s) → cheap on ZK, and the *ephemeral* znode gives automatic crash detection when the session dies, which the Thrift path wouldn't provide for free. Does that match the design intent — i.e. supervisor heartbeats stayed on ZK deliberately because per-node volume is low and ephemeral-node semantics are valuable, whereas per-worker heartbeats were the actual ZK scaling problem? Also noted on 3.0.0's ZK read/serialization improvements — thanks for the pointer, will look into it. On Thu, Jul 23, 2026 at 4:53 PM Karthick <[email protected]> wrote: > Hi all, > > I'm studying the Storm 2.0 heartbeat/liveness paths and want to confirm my > understanding of a design decision. > > As I read the 2.0 code, there are two distinct worker-originated > heartbeats: > > 1. *Liveness* — the worker writes an LSWorkerHeartbeat to local disk ( > Worker.doHeartBeat); the supervisor reads those files and relays a > batch to the leader Nimbus over *Thrift* (ReportWorkerHeartbeats → > Nimbus.sendSupervisorWorkerHeartbeats → HeartbeatCache), governed by > nimbus.task.timeout.secs. > 2. *Stats* — Worker.doExecutorHeartbeats writes a heartbeat object ( > time-secs + uptime + executor stats) to *ZooKeeper*, which the > UI/metrics consume (and which HeartbeatCache.updateFromZkHeartbeat can > still use for liveness on the ZK strategy). > > My understanding is that liveness was moved off ZooKeeper (the 1.x model, > and later Pacemaker) because high-volume per-worker heartbeat writes made > ZK a scaling bottleneck, and since heartbeats are ephemeral they don't need > ZK's persistence/consistency — so the supervisor-relay-over-Thrift model > removes those writes from ZK entirely. > > A few questions: > > - Is that the correct/primary motivation for the Thrift > supervisor-relay path, or were there other drivers (connection count, watch > load, Nimbus HA, recovery on leader change)? > - Why do executor *stats* still go through ZooKeeper rather than > riding the same Thrift path — is it purely that stats are lower-frequency > and UI-oriented, or is there a stronger reason? > - Is there a JIRA / design doc that captures this transition (beyond > docs/Pacemaker.md) that I could read? > > Thanks for any pointers — trying to make sure I document this accurately. >
