Hi, Thanks for bringing this up! It's a great question. To answer your point about why stats and liveness follow different paths: it comes down to push vs. pull mechanics and memory overhead:
Liveness (Push): Must be kept in-memory and constantly updated on Nimbus (HeartbeatCache), as the scheduler and failure detector require a continuous pulse for every worker. Sending light heartbeat signals via the Supervisor Thrift relay avoids clogging ZK while guaranteeing low-latency updates where Nimbus needs them. Stats (Pull at request): Unlike liveness, executor stats carry a heavy payload. Keeping all of that cached in Nimbus memory at all times-regardless of whether anyone is viewing the UI-would be wasteful. Instead, stats are read on-demand directly from ZK only when UI/API calls occur. ZK's storage model fits this on-demand access pattern much better. Next steps depending on your goal: - If you are hitting ZK performance issues: Consider upgrading to 3.0.0. It addresses ZK bottlenecks at the root level by optimizing read semantics and compressing serialized objects. - If you are looking for architectural symmetry: The current split is a trade-off to keep Nimbus lightweight while serving heavy UI metrics on demand. Cheers, Gianluca Il giorno gio 23 lug 2026 alle ore 13:24 Karthick < [email protected]> ha scritto: > Hi all, > > I'm studying the Storm 2.0 heartbeat/liveness paths and want to confirm my > understanding of a design decision. > > As I read the 2.0 code, there are two distinct worker-originated > heartbeats: > > 1. *Liveness* — the worker writes an LSWorkerHeartbeat to local disk ( > Worker.doHeartBeat); the supervisor reads those files and relays a > batch to the leader Nimbus over *Thrift* (ReportWorkerHeartbeats → > Nimbus.sendSupervisorWorkerHeartbeats → HeartbeatCache), governed by > nimbus.task.timeout.secs. > 2. *Stats* — Worker.doExecutorHeartbeats writes a heartbeat object ( > time-secs + uptime + executor stats) to *ZooKeeper*, which the > UI/metrics consume (and which HeartbeatCache.updateFromZkHeartbeat can > still use for liveness on the ZK strategy). > > My understanding is that liveness was moved off ZooKeeper (the 1.x model, > and later Pacemaker) because high-volume per-worker heartbeat writes made > ZK a scaling bottleneck, and since heartbeats are ephemeral they don't need > ZK's persistence/consistency — so the supervisor-relay-over-Thrift model > removes those writes from ZK entirely. > > A few questions: > > - Is that the correct/primary motivation for the Thrift > supervisor-relay path, or were there other drivers (connection count, watch > load, Nimbus HA, recovery on leader change)? > - Why do executor *stats* still go through ZooKeeper rather than > riding the same Thrift path — is it purely that stats are lower-frequency > and UI-oriented, or is there a stronger reason? > - Is there a JIRA / design doc that captures this transition (beyond > docs/Pacemaker.md) that I could read? > > Thanks for any pointers — trying to make sure I document this accurately. >
