Aaron Gresch created STORM-3139:
-----------------------------------
Summary: worker fails to start - KeeperErrorCode = NoAuth for
/credentials/topologyname
Key: STORM-3139
URL: https://issues.apache.org/jira/browse/STORM-3139
Project: Apache Storm
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Aaron Gresch
Seeing a sporadic test failure internally for us with a worker that won't come
up. The test schedules a bunch of topologies, kills the supervisors, restarts
nimbus, and then starts up the supervisors and validates the topologies are all
fully running.
I've seen this test failure twice in the last two weeks. The worker has
migrated and cannot come up:
{code:java}
2018-06-30 10:15:24.102 b.s.util main [WARN] Expecting exception of class:
class java.nio.channels.ClosedByInterruptException, but exception chain only
contains: (#<RuntimeException java.lang.RuntimeException:
org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException:
KeeperErrorCode = NoAuth for
/credentials/topology-testHardCoreFaultTolerance-7-21-1530352966>
#<NoAuthException
org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException:
KeeperErrorCode = NoAuth for
/credentials/topology-testHardCoreFaultTolerance-7-21-1530352966>) 2018-06-30
10:15:24.102 b.s.d.worker main [ERROR] Error on initialization of server
mk-worker java.lang.RuntimeException:
org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException:
KeeperErrorCode = NoAuth for
/credentials/topology-testHardCoreFaultTolerance-7-21-1530352966 at
backtype.storm.util$wrap_in_runtime.invoke(util.clj:53)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
backtype.storm.zookeeper$get_data.invoke(zookeeper.clj:135)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
backtype.storm.cluster_state.zookeeper_state_factory$_mkState$reify__4249.get_data(zookeeper_state_factory.clj:125)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_131] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_131] at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_131] at
clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
~[clojure-1.6.0.jar:?] at
clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
~[clojure-1.6.0.jar:?] at
org.apache.storm.pacemaker.pacemaker_state_factory$_mkState$reify__4296.get_data(pacemaker_state_factory.clj:175)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_131] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_131] at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_131] at
clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
~[clojure-1.6.0.jar:?] at
clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
~[clojure-1.6.0.jar:?] at
backtype.storm.cluster$mk_storm_cluster_state$reify__3910.credentials(cluster.clj:563)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_131] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_131] at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_131] at
clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
~[clojure-1.6.0.jar:?] at
clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
~[clojure-1.6.0.jar:?] at
backtype.storm.daemon.worker$fn__7710$exec_fn__1599__auto____7711.invoke(worker.clj:623)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.6.0.jar:?] at
clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.6.0.jar:?] at
clojure.core$apply.invoke(core.clj:624) ~[clojure-1.6.0.jar:?] at
backtype.storm.daemon.worker$fn__7710$mk_worker__7803.doInvoke(worker.clj:598)
[storm-core-0.10.2.y.jar:0.10.2.y] at
clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:?] at
backtype.storm.daemon.worker$_main.invoke(worker.clj:810)
[storm-core-0.10.2.y.jar:0.10.2.y] at
clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.6.0.jar:?] at
clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] at
backtype.storm.daemon.worker.main(Unknown Source)
[storm-core-0.10.2.y.jar:0.10.2.y] Caused by:
org.apache.storm.shade.org.apache.zookeeper.KeeperException$NoAuthException:
KeeperErrorCode = NoAuth for
/credentials/topology-testHardCoreFaultTolerance-7-21-1530352966 at
org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
org.apache.storm.shade.org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:327)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
org.apache.storm.shade.org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:316)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
org.apache.storm.shade.org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
org.apache.storm.shade.org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:313)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
org.apache.storm.shade.org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:304)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
org.apache.storm.shade.org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:35)
~[storm-core-0.10.2.y.jar:0.10.2.y] at
backtype.storm.zookeeper$get_data.invoke(zookeeper.clj:131)
~[storm-core-0.10.2.y.jar:0.10.2.y] ... 31 more 2018-06-30 10:15:24.199
b.s.util main [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
