mwkang opened a new pull request, #8789: URL: https://github.com/apache/storm/pull/8789
## What is the purpose of the change Fixes #8786. The supervisor builds the heartbeat batch it sends to Nimbus from every worker directory present on disk (`SupervisorUtils.readWorkerHeartbeats` → `ReportWorkerHeartbeats.getSupervisorWorkerHeartbeatsFromLocal`), with no check on whether the worker is still alive. When a worker directory is left behind — e.g. a worker that died before the supervisor finished `Container.cleanUpForRestart()` — its frozen heartbeat keeps being reported on every reporting round. Orphaned directories are only reclaimed when `ReadClusterState` is constructed at supervisor startup, not from the periodic sync loop, so at runtime the stale heartbeat is reported indefinitely until the next supervisor restart. Because the reported topology is usually no longer assigned (and its conf may already be deleted), Nimbus repeatedly tries to read the topology conf while processing that heartbeat and floods its log with `Exception when getting heartbeat timeout` / `NotAliveException`. STORM-4022 only suppressed the `NotAliveException` logging on the Nimbus side; the underlying behavior — the supervisor reporting heartbeats for dead/orphaned workers — was never addressed, and the generic-exception branch still logs. This change filters stale heartbeats out in `ReportWorkerHeartbeats`: any heartbeat whose age exceeds `supervisor.worker.timeout.secs` is skipped and not forwarded to Nimbus. That is the same timeout `Slot` uses to decide a worker is dead, so a worker past it is already considered dead and should not be reported. A live worker always refreshes its heartbeat well within the timeout, so no valid heartbeat is ever dropped; a heartbeat exactly at the boundary (age == timeout) is still reported. ## How was the change tested Added `ReportWorkerHeartbeatsTest` (driven by `Time.SimulatedTime`) covering `getSupervisorWorkerHeartbeatsFromLocal`: - a fresh heartbeat is reported; - a heartbeat exactly at the timeout boundary (age == timeout) is still reported; - a heartbeat just past the timeout (age == timeout + 1) is filtered out; - a long-stale orphaned heartbeat (a day old) is filtered out; - null local heartbeats continue to be skipped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
