[
https://issues.apache.org/jira/browse/STORM-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pedro Azevedo updated STORM-4077:
---------------------------------
Description:
Hey guys, I'm using Storm v2.6.1 and every time I restart the nimbus leader
(currently I have 3 for high availability) the workers get reassigned and this
is a bad behaviour as every topology will have no workers running for a certain
period(until new workers are assigned) due to a Nimbus leadership change.
On another note, when stopping the nimbus I'm getting this error which seems to
be impacting the gracefully shutdown.
{code:java}
2024-08-21T15:09:46.647Z Nimbus [INFO] Shutting down master
2024-08-21T15:09:46.648Z CuratorFrameworkImpl [INFO] backgroundOperationsLoop
exiting 2024-08-21T15:09:46.752Z ClientCnxn [INFO] EventThread shut down for
session: 0x4000010be7caa5d 2024-08-21T15:09:46.752Z ZooKeeper [INFO] Session:
0x4000010be7caa5d closed 2024-08-21T15:09:46.752Z CuratorFrameworkImpl [INFO]
backgroundOperationsLoop exiting 2024-08-21T15:09:46.812Z ProcessFunction
[ERROR] Internal error processing getLeader java.lang.IllegalStateException:
Expected state [STARTED] was [STOPPED] at
org.apache.storm.shade.org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:835)
~[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:462)
~[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:507)
~[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.zookeeper.ClientZookeeper.getChildren(ClientZookeeper.java:209)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.cluster.ZKStateStorage.get_children(ZKStateStorage.java:155)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.cluster.StormClusterStateImpl.nimbuses(StormClusterStateImpl.java:279)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.daemon.nimbus.Nimbus.getLeader(Nimbus.java:4907)
~[storm-server-2.6.1.jar:2.6.1] at
org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5168)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5144)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:40)
[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:40)
[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:171)
[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492)
[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.thrift.server.Invocation.run(Invocation.java:19)
[storm-shaded-deps-2.6.1.jar:2.6.1] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?] at java.base/java.lang.Thread.run(Thread.java:829) [?:?]{code}
was:
Hey guys, I'm using Storm v2.6.1 and every time I restart the nimbus leader
(currently I have 3 for high availability) the workers get reassigned and this
is a bad behaviour as every topology will have no workers running for a certain
period(until new workers are assigned) due to a Nimbus leadership change.
On another note, when stopping the nimbus I'm getting this error which seems to
be impacting the gracefully shutdown.
2024-08-21T15:09:46.647Z Nimbus [INFO] Shutting down master
2024-08-21T15:09:46.648Z CuratorFrameworkImpl [INFO] backgroundOperationsLoop
exiting 2024-08-21T15:09:46.752Z ClientCnxn [INFO] EventThread shut down for
session: 0x4000010be7caa5d 2024-08-21T15:09:46.752Z ZooKeeper [INFO] Session:
0x4000010be7caa5d closed 2024-08-21T15:09:46.752Z CuratorFrameworkImpl [INFO]
backgroundOperationsLoop exiting 2024-08-21T15:09:46.812Z ProcessFunction
[ERROR] Internal error processing getLeader java.lang.IllegalStateException:
Expected state [STARTED] was [STOPPED] at
org.apache.storm.shade.org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:835)
~[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:462)
~[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:507)
~[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.zookeeper.ClientZookeeper.getChildren(ClientZookeeper.java:209)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.cluster.ZKStateStorage.get_children(ZKStateStorage.java:155)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.cluster.StormClusterStateImpl.nimbuses(StormClusterStateImpl.java:279)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.daemon.nimbus.Nimbus.getLeader(Nimbus.java:4907)
~[storm-server-2.6.1.jar:2.6.1] at
org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5168)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5144)
~[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:40)
[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:40)
[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:171)
[storm-client-2.6.1.jar:2.6.1] at
org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492)
[storm-shaded-deps-2.6.1.jar:2.6.1] at
org.apache.storm.thrift.server.Invocation.run(Invocation.java:19)
[storm-shaded-deps-2.6.1.jar:2.6.1] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?] at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
> Worker being reassigned when Nimbus leadership changes
> ------------------------------------------------------
>
> Key: STORM-4077
> URL: https://issues.apache.org/jira/browse/STORM-4077
> Project: Apache Storm
> Issue Type: Bug
> Affects Versions: 2.6.1
> Reporter: Pedro Azevedo
> Priority: Major
>
> Hey guys, I'm using Storm v2.6.1 and every time I restart the nimbus leader
> (currently I have 3 for high availability) the workers get reassigned and
> this is a bad behaviour as every topology will have no workers running for a
> certain period(until new workers are assigned) due to a Nimbus leadership
> change.
> On another note, when stopping the nimbus I'm getting this error which seems
> to be impacting the gracefully shutdown.
> {code:java}
> 2024-08-21T15:09:46.647Z Nimbus [INFO] Shutting down master
> 2024-08-21T15:09:46.648Z CuratorFrameworkImpl [INFO] backgroundOperationsLoop
> exiting 2024-08-21T15:09:46.752Z ClientCnxn [INFO] EventThread shut down for
> session: 0x4000010be7caa5d 2024-08-21T15:09:46.752Z ZooKeeper [INFO] Session:
> 0x4000010be7caa5d closed 2024-08-21T15:09:46.752Z CuratorFrameworkImpl [INFO]
> backgroundOperationsLoop exiting 2024-08-21T15:09:46.812Z ProcessFunction
> [ERROR] Internal error processing getLeader java.lang.IllegalStateException:
> Expected state [STARTED] was [STOPPED] at
> org.apache.storm.shade.org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:835)
> ~[storm-shaded-deps-2.6.1.jar:2.6.1] at
> org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:462)
> ~[storm-shaded-deps-2.6.1.jar:2.6.1] at
> org.apache.storm.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:507)
> ~[storm-shaded-deps-2.6.1.jar:2.6.1] at
> org.apache.storm.zookeeper.ClientZookeeper.getChildren(ClientZookeeper.java:209)
> ~[storm-client-2.6.1.jar:2.6.1] at
> org.apache.storm.cluster.ZKStateStorage.get_children(ZKStateStorage.java:155)
> ~[storm-client-2.6.1.jar:2.6.1] at
> org.apache.storm.cluster.StormClusterStateImpl.nimbuses(StormClusterStateImpl.java:279)
> ~[storm-client-2.6.1.jar:2.6.1] at
> org.apache.storm.daemon.nimbus.Nimbus.getLeader(Nimbus.java:4907)
> ~[storm-server-2.6.1.jar:2.6.1] at
> org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5168)
> ~[storm-client-2.6.1.jar:2.6.1] at
> org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:5144)
> ~[storm-client-2.6.1.jar:2.6.1] at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:40)
> [storm-shaded-deps-2.6.1.jar:2.6.1] at
> org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:40)
> [storm-shaded-deps-2.6.1.jar:2.6.1] at
> org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:171)
> [storm-client-2.6.1.jar:2.6.1] at
> org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492)
> [storm-shaded-deps-2.6.1.jar:2.6.1] at
> org.apache.storm.thrift.server.Invocation.run(Invocation.java:19)
> [storm-shaded-deps-2.6.1.jar:2.6.1] at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?] at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?] at java.base/java.lang.Thread.run(Thread.java:829) [?:?]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
