jkrauss82 opened a new issue, #9098:
URL: https://github.com/apache/storm/issues/9098
## Intro and AI Disclaimer
I am a long time storm user of a small storm cluster. In general, the
cluster stays robust and stable over the course of weeks but every now and then
we run into a situation where seemingly an error in a topology and, following
up, a zookeeper connection loss brings a supervisor down and then the whole
cluster becomes unstable as topologies get re-assigned, supervisors
blacklisted, and we have to wipe the whole thing and start fresh.
Since I never really could pin that down to a specific reason I used local
AI to dig through our logs and the storm code. It came up with the report seen
below, it also suggested a fix which I am happy to open a PR for, if that is
okay in this repo. We currently have the fix running in production, seems to
hold up so far.
Depending on the assessment of the storm dev team I can also overwork the
fix and implement a more suitable solution, should the proposed new class
`ConnectionAwareRetryPolicy` in storm-client not be ideal.
Everything below is written by AI but carefully reviewed by me.
## Bug Description
A transient ZooKeeper connection loss (e.g. a single ensemble member closing
its socket) causes the entire supervisor process to exit within ~600ms. All
topology workers on that node are killed as a result.
**Storm version:** 3.1.0
**Component:** `storm-server` (supervisor), `storm-client` (CuratorUtils /
ZK retry)
## Expected Behavior
The supervisor should survive transient ZK connection blips. The ZK client's
`SendThread` is already failovering to another ensemble member - the supervisor
should wait for that reconnection and retry the operation, not die.
## Actual Behavior
The supervisor process exits with code 20 within 629ms of the connection
loss.
## Root Cause
Three compounding issues:
1. **Curator `RetryLoop` races the ZK `SendThread`.** Both operate on the
same TCP connection. The `RetryLoop` (driven by
`StormBoundedExponentialBackoffRetry`) blindly sleeps between retries without
checking whether the `SendThread` has already reconnected to another ensemble
member.
2. **The retry window is too short.** With defaults
(`storm.zookeeper.retry.times=5`, `storm.zookeeper.retry.interval=1000`), the
total retry budget is ~30s. In practice the exception escapes before even one
retry sleep completes, because the `RetryLoop` and `SendThread` are racing on
the same connection.
3. **`DefaultUncaughtExceptionHandler` kills the process.** Any uncaught
`Throwable` on a `StormTimer` thread calls `Runtime.getRuntime().exit(20)`. A
transient `KeeperException$ConnectionLoss` is treated the same as a fatal bug.
## Evidence
From a supervisor daemon log:
```
08:39:53.907 ZK server closes socket (EndOfStreamException)
08:39:54.489 Curator ConnectionState → SUSPENDED (+582ms)
08:39:54.510 SupervisorHeartbeat → existsNode() →
KeeperException$ConnectionLoss (+603ms)
08:39:54.536 DefaultUncaughtExceptionHandler → Utils.exitProcess(20)
(+629ms) ← JVM exits
```
The ZK ensemble was healthy throughout - the restarted supervisor
reconnected to the same ensemble within 100ms.
## Proposed Fix
Two layers:
### 1. `ConnectionAwareRetryPolicy` (new class in `storm-client`)
A `RetryPolicy` wrapper that tracks the `ConnectionState` via a
`ConnectionStateListener`. When the connection is `SUSPENDED` or `LOST`, it
calls `CuratorFramework.blockUntilConnected(sessionTimeout)` to **yield to the
`SendThread`'s failover** instead of blind sleep+retry. On reconnection, the
operation retries immediately on the new connection. Uses
`storm.zookeeper.session.timeout` as the wait bound.
Wired up in `CuratorUtils.newCurator()` via `AtomicReference` late binding.
### 2. Safety net in `SupervisorHeartbeat`
Wrap the heartbeat body in a `try/catch` so that a total ZK outage (beyond
the session timeout) skips the cycle instead of killing the process. A missed
heartbeat is harmless: `nimbus.supervisor.timeout.secs` (default 30s) allows 6
missed beats.
## Files Changed
-
`storm-client/src/jvm/org/apache/storm/utils/ConnectionAwareRetryPolicy.java`
(new)
- `storm-client/src/jvm/org/apache/storm/utils/CuratorUtils.java`
-
`storm-server/src/main/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
