Hi, I dug through the JIRA history and your reconciliation holds, though with two key refinements: the bottleneck in STORM-2693 was actually per-round read-and-recompute overhead on Nimbus rather than Zookeeper write pressure, which is why the fix prioritized caching and supervisor reporting over a faster store. SupervisorInfo stayed in ZK not for liveness monitoring, but because it is shared cluster-state metadata that any elected Nimbus leader needs to access for scheduling (content of the serialized https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/generated/SupervisorInfo.java )
Best, Gianluca Il giorno ven 24 lug 2026 alle ore 12:25 Karthick < [email protected]> ha scritto: > Thanks, that's a clear and helpful breakdown. The push-for-liveness / > pull-for-stats split makes sense: keeping a continuous in-memory pulse per > worker in Nimbus (HeartbeatCache) while serving heavy stats on demand > from ZK keeps Nimbus lightweight, and the Supervisor Thrift relay keeps > those frequent liveness writes off ZK. > > One thing I left out of my original question: the *supervisor liveness* > heartbeat — which is also liveness, but it still goes to ZooKeeper > (ephemeral znode at /supervisors/<id>, via SupervisorHeartbeat), consumed > by Nimbus under nimbus.supervisor.timeout.secs. So "liveness" isn't > uniformly on the Thrift path. > > My working reconciliation is that the deciding factor is *volume and > semantics, not liveness-vs-stats*: > > - Worker/executor liveness is high-volume (potentially thousands of > heartbeats per cluster, every ~1s) → moved off ZK onto the Supervisor > Thrift relay to avoid write pressure. > - Supervisor liveness is low-volume (one per node, every ~5s) → cheap > on ZK, and the *ephemeral* znode gives automatic crash detection when > the session dies, which the Thrift path wouldn't provide for free. > > Does that match the design intent — i.e. supervisor heartbeats stayed on > ZK deliberately because per-node volume is low and ephemeral-node semantics > are valuable, whereas per-worker heartbeats were the actual ZK scaling > problem? > > Also noted on 3.0.0's ZK read/serialization improvements — thanks for the > pointer, will look into it. > > > On Thu, Jul 23, 2026 at 4:53 PM Karthick <[email protected]> > wrote: > >> Hi all, >> >> I'm studying the Storm 2.0 heartbeat/liveness paths and want to confirm >> my understanding of a design decision. >> >> As I read the 2.0 code, there are two distinct worker-originated >> heartbeats: >> >> 1. *Liveness* — the worker writes an LSWorkerHeartbeat to local disk ( >> Worker.doHeartBeat); the supervisor reads those files and relays a >> batch to the leader Nimbus over *Thrift* (ReportWorkerHeartbeats → >> Nimbus.sendSupervisorWorkerHeartbeats → HeartbeatCache), governed by >> nimbus.task.timeout.secs. >> 2. *Stats* — Worker.doExecutorHeartbeats writes a heartbeat object ( >> time-secs + uptime + executor stats) to *ZooKeeper*, which the >> UI/metrics consume (and which HeartbeatCache.updateFromZkHeartbeat >> can still use for liveness on the ZK strategy). >> >> My understanding is that liveness was moved off ZooKeeper (the 1.x model, >> and later Pacemaker) because high-volume per-worker heartbeat writes made >> ZK a scaling bottleneck, and since heartbeats are ephemeral they don't need >> ZK's persistence/consistency — so the supervisor-relay-over-Thrift model >> removes those writes from ZK entirely. >> >> A few questions: >> >> - Is that the correct/primary motivation for the Thrift >> supervisor-relay path, or were there other drivers (connection count, >> watch >> load, Nimbus HA, recovery on leader change)? >> - Why do executor *stats* still go through ZooKeeper rather than >> riding the same Thrift path — is it purely that stats are lower-frequency >> and UI-oriented, or is there a stronger reason? >> - Is there a JIRA / design doc that captures this transition (beyond >> docs/Pacemaker.md) that I could read? >> >> Thanks for any pointers — trying to make sure I document this accurately. >> >
