This is an automated email from the ASF dual-hosted git repository.

rzo1 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/storm.git


The following commit(s) were added to refs/heads/master by this push:
     new 44ac52f07 Fix supervisor crash on transient ZK connection loss (#9153)
44ac52f07 is described below

commit 44ac52f07ae9d78e412f911351d133487c9b5665
Author: Jonas <[email protected]>
AuthorDate: Wed Nov 4 08:13:47 2026 +0100

    Fix supervisor crash on transient ZK connection loss (#9153)
    
    * Fix supervisor crash on transient ZK connection loss
    
    Root cause: A transient ZooKeeper connection loss during the supervisor
    heartbeat cycle caused the process to die in 629ms via
    DefaultUncaughtExceptionHandler. The Curator RetryLoop blindly slept
    between retries, racing against the ZK client's SendThread reconnection
    instead of yielding to it.
    
    Fix (two layers):
    
    1. ConnectionAwareRetryPolicy (new): A RetryPolicy wrapper that tracks
       the ZK ConnectionState via a ConnectionStateListener. When the
       connection is SUSPENDED or LOST, it calls
       CuratorFramework.blockUntilConnected(sessionTimeout) to yield to the
       SendThread's failover, instead of blind sleep+retry. On reconnection,
       the operation retries immediately on the new connection. Uses
       storm.zookeeper.session.timeout as the wait bound (no magic numbers).
    
    2. SupervisorHeartbeat safety net: Wrap the heartbeat body in
       try/catch so that a total ZK outage (beyond session timeout) skips
       the cycle instead of killing the process. A missed heartbeat is
       harmless: nimbus.supervisor.timeout.secs (30s) allows 6 missed beats.
    
    Wired up in CuratorUtils.newCurator() via AtomicReference late binding
    (the CuratorFramework doesn't exist until builder.build() returns).
    
    * Address PR review: test fix, unit tests, retry bound, jitter, scoped catch
    
    - Fix CuratorUtilsTest ClassCastException (unwrap 
ConnectionAwareRetryPolicy)
    - Add @VisibleForTesting getDelegate() accessor
    - Add ConnectionAwareRetryPolicyTest (8 cases) and SupervisorHeartbeatTest 
(2 cases)
    - Add delegate budget check before blocking on SUSPENDED/LOST
    - Add RECONNECT_JITTER_MS constant (100ms) to avoid thundering herd
    - Scope SupervisorHeartbeat catch to ZK write only (not 
getNumaMap/buildSupervisorInfo)
    - Fix heartbeat comment (liveness via ephemeral ZK nodes, not heartbeat 
timestamps)
    
    * Scope ReadClusterState ZK catch to profiler lookup
    
    Per upstream review, don't drop the throw for the whole run() - that would
    swallow persistent local failures (mkSlot, port conflicts). The only ZK 
read in
    run() is the profiler request lookup; assignments come from the local 
backend.
    Scope the catch to getProfileActions so a transient ZK error skips the cycle
    (requests persist in ZK, retried in 10s) while persistent failures still 
fail
    fast. Adds ReadClusterStateTest.
    
    * Add a failing test for ConnectionAwareRetryPolicy on the ZooKeeper 
EventThread
    
    Curator also consults the retry policy from the ZooKeeper EventThread when
    it retries a background operation 
(CuratorFrameworkImpl.checkBackgroundRetry).
    Blocking that thread in blockUntilConnected keeps Curator from processing
    SyncConnected, so after the session timeout it injects a session expiration
    on a connection that had already recovered.
    
    The test stops a real ZooKeeper server, restarts it after 500 ms and calls
    the policy once it has seen SUSPENDED. Off the EventThread the retry is
    allowed after the reconnection. On the EventThread the call blocks for the
    whole session timeout, returns false and the client goes to LOST, so that
    case fails with the current implementation.
    
    * Scope PR to supervisor: remove ConnectionAwareRetryPolicy, narrow 
heartbeat catch
    
    Per maintainer feedback (GGraziadei, rzo1):
    - Remove ConnectionAwareRetryPolicy and CuratorUtils changes (move to 
follow-up PR)
    - The policy blocked the ZK EventThread via blockUntilConnected, causing 
session
      expiry on background operations (Nimbus LeaderLatch, etc.)
    - Narrow SupervisorHeartbeat catch to KeeperException causes only; rethrow
      non-ZK failures so persistent misconfigurations still surface
    - Update tests to verify ZK failures are swallowed and non-ZK failures 
propagate
    
    The two scoped catches (SupervisorHeartbeat + ReadClusterState) are 
sufficient
    to fix the supervisor crash on transient ZK connection loss.
    
    * Restore CuratorUtils and its test to master's formatting
    
    The retry policy was moved out of this PR, so these two files carry no
    change. They had picked up the formatting from before the checkstyle
    reformat in #9184.
    
    * Only swallow transient ZooKeeper failures in the supervisor heartbeat
    
    The heartbeat now only continues on connection loss, session expiry and
    operation timeout. Persistent ZooKeeper errors such as NoAuth or
    InvalidACL are rethrown, so a supervisor with wrong ZooKeeper credentials
    or ACLs still fails at launch instead of running without registering.
    Applies the review suggestions from GGraziadei and reiabreu.
    
    ---------
    
    Co-authored-by: Jonas Krauss <[email protected]>
    Co-authored-by: Gianluca Graziadei <[email protected]>
    Co-authored-by: Richard Zowalla <[email protected]>
---
 .../storm/daemon/supervisor/ReadClusterState.java  |  11 +-
 .../supervisor/timer/SupervisorHeartbeat.java      |  20 +++-
 .../daemon/supervisor/ReadClusterStateTest.java    |  86 ++++++++++++++
 .../supervisor/timer/SupervisorHeartbeatTest.java  | 130 +++++++++++++++++++++
 4 files changed, 245 insertions(+), 2 deletions(-)

diff --git 
a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/ReadClusterState.java
 
b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/ReadClusterState.java
index 0186230d0..cf308e893 100644
--- 
a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/ReadClusterState.java
+++ 
b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/ReadClusterState.java
@@ -14,6 +14,7 @@ package org.apache.storm.daemon.supervisor;
 
 import java.util.ArrayList;
 import java.util.Collection;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
@@ -140,7 +141,15 @@ public class ReadClusterState implements Runnable, 
AutoCloseable {
                 //Something odd happened try again later
                 return;
             }
-            Map<String, List<ProfileRequest>> topoIdToProfilerActions = 
getProfileActions(stormClusterState, stormIds);
+            Map<String, List<ProfileRequest>> topoIdToProfilerActions;
+            try {
+                topoIdToProfilerActions = getProfileActions(stormClusterState, 
stormIds);
+            } catch (Exception e) {
+                // Assignments come from the local backend; profiler requests 
are the only ZooKeeper read here.
+                // They stay in ZooKeeper, so skip them for this cycle instead 
of halting the supervisor.
+                LOG.warn("Failed to read profiler requests, will retry next 
cycle", e);
+                topoIdToProfilerActions = Collections.emptyMap();
+            }
 
             HashSet<Integer> assignedPorts = new HashSet<>();
             LOG.debug("Synchronizing supervisor");
diff --git 
a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.java
 
b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.java
index b613f36ee..488397d06 100644
--- 
a/storm-server/src/main/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.java
+++ 
b/storm-server/src/main/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeat.java
@@ -29,8 +29,10 @@ import org.apache.storm.generated.SupervisorInfo;
 import 
org.apache.storm.scheduler.resource.normalization.NormalizedResourceOffer;
 import org.apache.storm.scheduler.resource.normalization.NormalizedResources;
 import org.apache.storm.shade.org.apache.commons.collections4.ListUtils;
+import org.apache.storm.shade.org.apache.zookeeper.KeeperException;
 import org.apache.storm.utils.ObjectReader;
 import org.apache.storm.utils.Time;
+import org.apache.storm.utils.Utils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -162,7 +164,23 @@ public class SupervisorHeartbeat implements Runnable {
         Map<String, Object> validatedNumaMap = 
SupervisorUtils.getNumaMap(conf);
         Map<String, SupervisorInfo> supervisorInfoList = 
buildSupervisorInfo(conf, supervisor, validatedNumaMap);
         for (Map.Entry<String, SupervisorInfo> supervisorInfoEntry : 
supervisorInfoList.entrySet()) {
-            
stormClusterState.supervisorHeartbeat(supervisorInfoEntry.getKey(), 
supervisorInfoEntry.getValue());
+            try {
+                
stormClusterState.supervisorHeartbeat(supervisorInfoEntry.getKey(), 
supervisorInfoEntry.getValue());
+            } catch (Exception e) {
+                if (!isTransientZkFailure(e)) {
+                    throw e;
+                }
+                // A transient ZK failure: the ephemeral node is recreated on 
the next heartbeat.
+                // Local and persistent ZK errors (e.g. bad ACLs) are rethrown 
so they surface.
+                LOG.warn("Supervisor ZK heartbeat failed for {}, will retry 
next cycle",
+                    supervisorInfoEntry.getKey(), e);
+            }
         }
     }
+
+    private static boolean isTransientZkFailure(Throwable t) {
+        return 
Utils.exceptionCauseIsInstanceOf(KeeperException.ConnectionLossException.class, 
t)
+            || 
Utils.exceptionCauseIsInstanceOf(KeeperException.SessionExpiredException.class, 
t)
+            || 
Utils.exceptionCauseIsInstanceOf(KeeperException.OperationTimeoutException.class,
 t);
+    }
 }
diff --git 
a/storm-server/src/test/java/org/apache/storm/daemon/supervisor/ReadClusterStateTest.java
 
b/storm-server/src/test/java/org/apache/storm/daemon/supervisor/ReadClusterStateTest.java
new file mode 100644
index 000000000..950818258
--- /dev/null
+++ 
b/storm-server/src/test/java/org/apache/storm/daemon/supervisor/ReadClusterStateTest.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.  The 
ASF licenses this file to you under the Apache License, Version
+ * 2.0 (the "License"); you may not use this file except in compliance with 
the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 
See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor;
+
+import java.nio.file.Path;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.storm.Config;
+import org.apache.storm.DaemonConfig;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.scheduler.ISupervisor;
+import org.apache.storm.shade.org.apache.zookeeper.KeeperException;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class ReadClusterStateTest {
+
+    private static final String TOPOLOGY_ID = "topo-1";
+
+    @TempDir
+    Path localDir;
+
+    private final IStormClusterState clusterState = 
mock(IStormClusterState.class);
+    private final ISupervisor iSupervisor = mock(ISupervisor.class);
+
+    private ReadClusterState newReadClusterState() throws Exception {
+        Map<String, Object> conf = new HashMap<>();
+        conf.put(Config.STORM_CLUSTER_MODE, "local");
+        conf.put(Config.STORM_LOCAL_DIR, localDir.toString());
+        conf.put(DaemonConfig.SUPERVISOR_SLOTS_PORTS, Collections.emptyList());
+
+        Supervisor supervisor = mock(Supervisor.class);
+        when(supervisor.getConf()).thenReturn(conf);
+        when(supervisor.getStormClusterState()).thenReturn(clusterState);
+        when(supervisor.getAssignmentId()).thenReturn("supervisor-1");
+        when(supervisor.getiSupervisor()).thenReturn(iSupervisor);
+        when(supervisor.getHostName()).thenReturn("host-1");
+        when(supervisor.getCurrAssignment()).thenReturn(new 
AtomicReference<>(new HashMap<>()));
+        return new ReadClusterState(supervisor);
+    }
+
+    @Test
+    public void profilerRequestFailureDoesNotHaltAssignmentSync() throws 
Exception {
+        
when(clusterState.assignments(null)).thenReturn(Collections.singletonList(TOPOLOGY_ID));
+        
when(clusterState.assignmentsInfo()).thenReturn(Collections.emptyMap());
+        // What ClientZookeeper surfaces after Curator gives up, as in the 
reported supervisor crash.
+        
when(clusterState.getTopologyProfileRequests(TOPOLOGY_ID)).thenThrow(new 
RuntimeException(
+            new KeeperException.SessionExpiredException()));
+
+        ReadClusterState readClusterState = newReadClusterState();
+
+        assertDoesNotThrow(readClusterState::run);
+        verify(iSupervisor).assigned(any());
+    }
+
+    @Test
+    public void assignmentReadFailureStillPropagates() throws Exception {
+        when(clusterState.assignments(null)).thenThrow(new 
IllegalStateException("broken local backend"));
+
+        ReadClusterState readClusterState = newReadClusterState();
+
+        assertThrows(RuntimeException.class, readClusterState::run);
+        verify(clusterState, never()).getTopologyProfileRequests(anyString());
+    }
+}
diff --git 
a/storm-server/src/test/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeatTest.java
 
b/storm-server/src/test/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeatTest.java
new file mode 100644
index 000000000..5fbe35668
--- /dev/null
+++ 
b/storm-server/src/test/java/org/apache/storm/daemon/supervisor/timer/SupervisorHeartbeatTest.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.  The 
ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.daemon.supervisor.timer;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.storm.cluster.IStormClusterState;
+import org.apache.storm.daemon.supervisor.Supervisor;
+import org.apache.storm.generated.SupervisorInfo;
+import org.apache.storm.scheduler.ISupervisor;
+import org.apache.storm.shade.org.apache.zookeeper.KeeperException;
+import org.apache.storm.utils.Utils;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class SupervisorHeartbeatTest {
+
+    private static final String SUPERVISOR_ID = "supervisor-1";
+
+    private Supervisor mockSupervisor(IStormClusterState clusterState) {
+        Supervisor supervisor = mock(Supervisor.class);
+        ISupervisor iSupervisor = mock(ISupervisor.class);
+        when(iSupervisor.getMetadata()).thenReturn(Arrays.asList(6700, 6701));
+
+        when(supervisor.getStormClusterState()).thenReturn(clusterState);
+        when(supervisor.getId()).thenReturn(SUPERVISOR_ID);
+        when(supervisor.getiSupervisor()).thenReturn(iSupervisor);
+        when(supervisor.getCurrAssignment()).thenReturn(new 
AtomicReference<>(new HashMap<>()));
+        when(supervisor.getHostName()).thenReturn("host-1");
+        when(supervisor.getAssignmentId()).thenReturn(SUPERVISOR_ID);
+        when(supervisor.getThriftServerPort()).thenReturn(6627);
+        when(supervisor.getUpTime()).thenReturn(Utils.makeUptimeComputer());
+        when(supervisor.getStormVersion()).thenReturn("test-version");
+        return supervisor;
+    }
+
+    private Map<String, Object> baseConf() {
+        return new HashMap<>(Utils.readStormConfig());
+    }
+
+    @Test
+    public void 
transientZkHeartbeatFailureIsSwallowedInsteadOfKillingTheProcess() {
+        IStormClusterState clusterState = mock(IStormClusterState.class);
+        doThrow(new RuntimeException(
+            new KeeperException.ConnectionLossException()))
+            .when(clusterState).supervisorHeartbeat(anyString(), 
any(SupervisorInfo.class));
+
+        Supervisor supervisor = mockSupervisor(clusterState);
+        SupervisorHeartbeat heartbeat = new SupervisorHeartbeat(baseConf(), 
supervisor);
+
+        assertDoesNotThrow(heartbeat::run);
+        verify(clusterState, atLeastOnce()).supervisorHeartbeat(anyString(), 
any(SupervisorInfo.class));
+    }
+
+    @Test
+    public void otherTransientZkFailuresAreSwallowed() {
+        for (KeeperException failure : Arrays.asList(new 
KeeperException.SessionExpiredException(),
+            new KeeperException.OperationTimeoutException())) {
+            IStormClusterState clusterState = mock(IStormClusterState.class);
+            doThrow(new RuntimeException(failure))
+                .when(clusterState).supervisorHeartbeat(anyString(), 
any(SupervisorInfo.class));
+
+            SupervisorHeartbeat heartbeat = new 
SupervisorHeartbeat(baseConf(), mockSupervisor(clusterState));
+
+            assertDoesNotThrow(heartbeat::run);
+        }
+    }
+
+    @Test
+    public void persistentZkFailureStillPropagates() {
+        for (KeeperException failure : Arrays.asList(new 
KeeperException.NoAuthException(),
+            new KeeperException.InvalidACLException())) {
+            IStormClusterState clusterState = mock(IStormClusterState.class);
+            doThrow(new RuntimeException(failure))
+                .when(clusterState).supervisorHeartbeat(anyString(), 
any(SupervisorInfo.class));
+
+            SupervisorHeartbeat heartbeat = new 
SupervisorHeartbeat(baseConf(), mockSupervisor(clusterState));
+
+            assertThrows(RuntimeException.class, heartbeat::run);
+        }
+    }
+
+    @Test
+    public void nonZkHeartbeatFailureStillPropagates() {
+        IStormClusterState clusterState = mock(IStormClusterState.class);
+        doThrow(new IllegalStateException("broken local config"))
+            .when(clusterState).supervisorHeartbeat(anyString(), 
any(SupervisorInfo.class));
+
+        Supervisor supervisor = mockSupervisor(clusterState);
+        SupervisorHeartbeat heartbeat = new SupervisorHeartbeat(baseConf(), 
supervisor);
+
+        assertThrows(IllegalStateException.class, heartbeat::run);
+    }
+
+    @Test
+    public void successfulCycleSendsTheHeartbeat() {
+        IStormClusterState clusterState = mock(IStormClusterState.class);
+        Supervisor supervisor = mockSupervisor(clusterState);
+        SupervisorHeartbeat heartbeat = new SupervisorHeartbeat(baseConf(), 
supervisor);
+
+        assertDoesNotThrow(heartbeat::run);
+        verify(clusterState).supervisorHeartbeat(eq(SUPERVISOR_ID), 
any(SupervisorInfo.class));
+    }
+}

Reply via email to