Hello, We recently upgraded our Apache Storm Nimbus server to 2.8.8 from 2.8.0 and the storm-nimbus service just continuously restart with the error below. We are currently using SASL authentication using the DIGEST-MD5 mechanism between our storm and zookeeper servers which is working fine in version 2.8.0 but now we get this error in 2.8.8.
2026-06-02 17:26:29.157 o.a.s.v.ConfigValidation main [INFO] Will use [class org.apache.storm.DaemonConfig, class org.apache.storm.Config] for validation 2026-06-02 17:26:29.317 o.a.s.z.ClientZookeeper main [INFO] Starting ZK Curator 2026-06-02 17:26:29.317 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] Starting 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:zookeeper.version=3.9.5-293c895a8d966a3ecb92872be4a1daf87d725da2, built on 2026-02-11 20:18 UTC 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:host.name=clrv0000294762.ic.ing.net 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.version=17.0.19 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.vendor=Red Hat, Inc. 2026-06-02 17:26:29.322 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.home=/usr/lib/jvm/java-17-openjdk-17.0.19.0.10-2.el9.x86_64 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.class.path=/var/opt/storm/apache-storm-2.8.8/*:/var/opt/storm/apache-storm-2.8.8/lib/storm-shaded-deps-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/clojure-1.12.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-util-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-handler-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-marshalling-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-exec-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-jvm-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jspecify-1.0.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/slf4j-api-2.0.17.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-core-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-annotations-2.21.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-dataformat-smile-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/hadoop-auth-3.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-lang3-3.20.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jline-2.14.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-collections4-4.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/amqp-client-5.28.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-core-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-server-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/j2objc-annotations-3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-asn1-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.activation-1.2.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/javax.annotation-api-1.3.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-io-2.22.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/httpcore-4.4.16.jar:/var/opt/storm/apache-storm-2.8.8/lib/guava-33.6.0-jre.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-pkix-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-buffer-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-logging-1.3.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-server-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/audience-annotations-0.12.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.activation-api-1.2.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/zookeeper-jute-3.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/nimbus-jose-jwt-10.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-codec-1.22.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.xml.bind-api-4.0.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-slf4j2-impl-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/zookeeper-3.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-databind-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-fileupload-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/curator-framework-5.9.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-core-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-io-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/chill-java-0.9.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/hadoop-shaded-guava-1.5.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-clojure-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-util-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-graphite-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/jackson-core-2.21.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-util-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/curator-client-5.9.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-over-slf4j-2.0.17.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-client-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/reflectasm-1.11.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-transport-native-unix-common-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/rocksdbjni-10.10.1.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/snakeyaml-2.6.jar:/var/opt/storm/apache-storm-2.8.8/lib/spec.alpha-0.5.238.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-compression-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/error_prone_annotations-2.49.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerby-config-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/minlog-1.3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/kryo-5.6.2.jar:/var/opt/storm/apache-storm-2.8.8/lib/objenesis-3.5.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcprov-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-security-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcpkix-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-protobuf-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-transport-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jakarta.servlet-api-6.1.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/bcutil-jdk18on-1.84.jar:/var/opt/storm/apache-storm-2.8.8/lib/json-smart-2.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-session-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/storm-core-2.8.8.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-common-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/asm-9.9.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/failureaccess-1.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/core.specs.alpha-0.4.74.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-http-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-codec-base-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/log4j-api-2.25.4.jar:/var/opt/storm/apache-storm-2.8.8/lib/netty-resolver-4.2.12.Final.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-ee10-servlet-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/jetty-ee10-servlets-12.1.9.jar:/var/opt/storm/apache-storm-2.8.8/lib/httpclient-4.5.14.jar:/var/opt/storm/apache-storm-2.8.8/lib/kerb-crypto-2.0.3.jar:/var/opt/storm/apache-storm-2.8.8/lib/tools.logging-1.3.1.jar:/var/opt/storm/apache-storm-2.8.8/lib/accessors-smart-2.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/carbonite-1.6.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/metrics-jmx-4.2.38.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-compress-1.28.0.jar:/var/opt/storm/apache-storm-2.8.8/lib/commons-cli-1.11.0.jar:/var/opt/storm/apache-storm-2.8.8/extlib/*:/var/opt/storm/apache-storm-2.8.8/extlib-daemon/*:/var/opt/storm/current/conf 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.io.tmpdir=/var/storm-tmp 2026-06-02 17:26:29.323 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.compiler=<NA> 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.name=Linux 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.arch=amd64 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.version=5.14.0-611.55.1.el9_7.x86_64 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.name=gmttusr 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.home=/home/gmttusr 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.dir=/ 2026-06-02 17:26:29.324 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.memory.free=36MB 2026-06-02 17:26:29.325 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.memory.max=1024MB 2026-06-02 17:26:29.325 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.memory.total=60MB 2026-06-02 17:26:29.327 o.a.s.s.o.a.z.ZooKeeper main [INFO] Initiating client connection, connectString=clrv0000301358.ic.ing.net:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@23a9ba52 2026-06-02 17:26:29.332 o.a.s.s.o.a.z.c.X509Util main [INFO] Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2026-06-02 17:26:29.471 o.a.s.s.o.a.z.c.X509Util main [INFO] Default TLS protocol is TLSv1.3, supported TLS protocols are [TLSv1.3, TLSv1.2, TLSv1.1, TLSv1, SSLv3, SSLv2Hello] 2026-06-02 17:26:29.478 o.a.s.s.o.a.z.ClientCnxnSocket main [INFO] jute.maxbuffer value is 1048575 Bytes 2026-06-02 17:26:29.486 o.a.s.s.o.a.z.ClientCnxn main [INFO] zookeeper.request.timeout value is 0. feature enabled=false 2026-06-02 17:26:29.494 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] Default schema 2026-06-02 17:26:29.503 o.a.s.s.o.a.z.Login main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Client successfully logged in. 2026-06-02 17:26:29.506 o.a.s.s.o.a.z.c.ZooKeeperSaslClient main-SendThread(clrv0000301358.ic.ing.net:2181) [WARN] Client will not use DIGEST-MD5 as SASL mechanism, because FIPS mode is enabled. 2026-06-02 17:26:29.506 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Opening socket connection to server clrv0000301358.ic.ing.net/10.159.128.170:2181. 2026-06-02 17:26:29.507 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client' 2026-06-02 17:26:29.510 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Socket connection established, initiating session, client: /10.158.64.7:48370, server: clrv0000301358.ic.ing.net/10.159.128.170:2181 2026-06-02 17:26:29.515 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [INFO] Session establishment complete on server clrv0000301358.ic.ing.net/10.159.128.170:2181, session id = 0x100000051861675, negotiated timeout = 20000 2026-06-02 17:26:29.516 o.a.s.s.o.a.z.ClientCnxn main-SendThread(clrv0000301358.ic.ing.net:2181) [ERROR] SASL authentication with Zookeeper Quorum member failed. javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null. at org.apache.storm.shade.org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:402) at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1212) 2026-06-02 17:26:29.518 o.a.s.s.o.a.c.f.s.ConnectionStateManager main-EventThread [INFO] State change: CONNECTED 2026-06-02 17:26:29.520 o.a.s.s.o.a.z.ClientCnxn main-EventThread [ERROR] Error while calling watcher. java.lang.NullPointerException: Cannot invoke "org.apache.storm.shade.org.apache.zookeeper.Watcher$Event$KeeperState.ordinal()" because "state" is null at org.apache.storm.shade.org.apache.curator.ConnectionState.checkState(ConnectionState.java:206) at org.apache.storm.shade.org.apache.curator.ConnectionState.process(ConnectionState.java:146) at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:566) at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:541) 2026-06-02 17:26:29.521 o.a.s.s.o.a.z.ClientCnxn main-EventThread [INFO] EventThread shut down for session: 0x100000051861675 2026-06-02 17:26:29.524 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl Curator-Framework-0 [INFO] backgroundOperationsLoop exiting 2026-06-02 17:26:29.528 o.a.s.u.Utils main [ERROR] Received error in thread main.. terminating server... java.lang.Error: org.apache.storm.shade.org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /storm at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:668) at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:672) at org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:1055) at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1082) at java.base/java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1077) at java.base/java.lang.Thread.dispatchUncaughtException(Thread.java:2017) Caused by: org.apache.storm.shade.org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /storm at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:129) at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:53) at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1867) at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:247) at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:240) at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:88) at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForegroundStandard(ExistsBuilderImpl.java:240) at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:235) at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:202) at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:35) at org.apache.storm.zookeeper.AclEnforcement.verifyAcls(AclEnforcement.java:84) at org.apache.storm.daemon.nimbus.Nimbus.launch(Nimbus.java:1608) at org.apache.storm.daemon.nimbus.Nimbus.main(Nimbus.java:1615) Best regards, Norman ----------------------------------------------------------------- ATTENTION: The information in this e-mail is confidential and only meant for the intended recipient. If you are not the intended recipient, don't use or disclose it in any way. Please let the sender know and delete the message immediately. -----------------------------------------------------------------
