Aaron Gresch created STORM-3117:
-----------------------------------
Summary: Deleting blobs for running topologies hoses Nimbus
Key: STORM-3117
URL: https://issues.apache.org/jira/browse/STORM-3117
Project: Apache Storm
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Aaron Gresch
Assignee: Aaron Gresch
The following test pseudo-code causes issues:
{code:java}
cluster.submitTopology(cluster.getTopologiesJarFile(), topoName, config,
topology);
cluster.waitTopologyUp(topoName);
cluster.deleteAllBlobs();
{code}
This causes nimbus to get stuck and restart:
{code:java}
2018-06-20 15:48:14.273 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Received
topology submission for wc-topology-test (storm-0.10.2.y.251 JDK-1.8.0_131)
2018-06-20 15:48:14.629 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Activating
wc-topology-test: wc-topology-test-1-1529509694
2018-06-20 15:48:14.724 o.a.s.d.n.Nimbus pool-27-thread-703 [INFO] TRANSITION:
wc-topology-test-1-1529509694 KILL null true
2018-06-20 15:48:14.812 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob
for key wc-topology-test-1-1529509694-stormconf.ser
2018-06-20 15:48:14.830 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob
for key wc-topology-test-1-1529509694-stormcode.ser
2018-06-20 15:48:14.863 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob
for key wc-topology-test-1-1529509694-stormjar.jar
2018-06-20 15:48:18.449 o.a.s.s.r.s.p.DefaultSchedulingPriorityStrategy timer
[INFO] SIM Scheduling wc-topology-test-1-1529509694 with score of 0.3125
2018-06-20 15:48:18.492 o.a.s.s.Cluster timer [INFO] STATUS -
wc-topology-test-1-1529509694 Running - Fully Scheduled by
DefaultResourceAwareStrategy
2018-06-20 15:48:18.527 o.a.s.d.n.Nimbus timer [INFO] Setting new assignment
for topology id wc-topology-test-1-1529509694:
2018-06-20 15:48:18.979 o.a.s.d.n.Nimbus pool-27-thread-722 [WARN] get blob
meta exception.
org.apache.storm.utils.WrappedKeyNotFoundException:
wc-topology-test-1-1529509694-stormjar.jar
at
org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483)
[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011)
[storm-client-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990)
[storm-client-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38)
[shaded-deps-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
[shaded-deps-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147)
[storm-client-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291)
[shaded-deps-2.0.0.y.jar:2.0.0.y]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2018-06-20 15:48:22.884 o.a.s.d.n.Nimbus timer [INFO] Renewing Creds For
wc-topology-test-1-1529509694 with
org.apache.storm.security.auth.kerberos.AutoTGT@4482469c owned by
[email protected]
2018-06-20 15:48:37.947 o.a.s.d.n.Nimbus timer [ERROR] Error while processing
event
java.lang.RuntimeException:
KeyNotFoundException(msg:wc-topology-test-1-1529509694-stormcode.ser)
at
org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2822)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.StormTimer$1.run(StormTimer.java:111)
~[storm-client-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:227)
[storm-client-2.0.0.y.jar:2.0.0.y]
Caused by: org.apache.storm.utils.WrappedKeyNotFoundException:
wc-topology-test-1-1529509694-stormcode.ser
at
org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.blobstore.LocalFsBlobStore.getBlobReplication(LocalFsBlobStore.java:420)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.daemon.nimbus.Nimbus.getBlobReplicationCount(Nimbus.java:1517)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.daemon.nimbus.Nimbus.getClusterInfoImpl(Nimbus.java:2675)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.daemon.nimbus.Nimbus.sendClusterMetricsToExecutors(Nimbus.java:2686)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2819)
~[storm-server-2.0.0.y.jar:2.0.0.y]
... 2 more
2018-06-20 15:48:37.948 o.a.s.u.Utils timer [ERROR] Halting process: Error
while processing event
java.lang.RuntimeException: Halting process: Error while processing event
at org.apache.storm.utils.Utils.exitProcess(Utils.java:468)
~[storm-client-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.daemon.nimbus.Nimbus.lambda$new$17(Nimbus.java:488)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:252)
[storm-client-2.0.0.y.jar:2.0.0.y]
2018-06-20 15:48:37.950 o.a.s.d.n.Nimbus Thread-11 [INFO] Shutting down master
2018-06-20 15:48:37.950 o.a.s.u.Utils Thread-12 [INFO] Halting after 10 seconds
2018-06-20 15:48:46.672 o.a.s.d.n.Nimbus pool-27-thread-798 [WARN] get blob
meta exception.
org.apache.storm.utils.WrappedKeyNotFoundException:
wc-topology-test-1-1529509694-stormconf.ser
at
org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483)
[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011)
[storm-client-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990)
[storm-client-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38)
[shaded-deps-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
[shaded-deps-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147)
[storm-client-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291)
[shaded-deps-2.0.0.y.jar:2.0.0.y]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2018-06-20 15:48:47.950 o.a.s.u.Utils Thread-12 [WARN] Forcing Halt...
{code}
Nimbus then continually restarts:
{code:java}
2018-06-20 15:48:54.635 o.a.s.u.Utils main [ERROR] Received error in main
thread.. terminating server...
java.lang.Error: java.lang.IllegalStateException: Could not find credentials
for topology wc-topology-test-1-1529509694 at path /storms. Don't know how to
fix this automatically. Please add needed ACLs, or delete the path.
at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:603)
~[storm-client-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:582)
~[storm-client-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.utils.Utils$5.uncaughtException(Utils.java:931)
[storm-client-2.0.0.y.jar:2.0.0.y]
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
[?:1.8.0_131]
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
[?:1.8.0_131]
at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
[?:1.8.0_131]
Caused by: java.lang.IllegalStateException: Could not find credentials for
topology wc-topology-test-1-1529509694 at path /storms. Don't know how to fix
this automatically. Please add needed ACLs, or delete the path.
at
org.apache.storm.zookeeper.AclEnforcement.getTopoAcl(AclEnforcement.java:194)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.zookeeper.AclEnforcement.verifyParentWithTopoChildren(AclEnforcement.java:250)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.zookeeper.AclEnforcement.verifyParentWithReadOnlyTopoChildren(AclEnforcement.java:258)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at
org.apache.storm.zookeeper.AclEnforcement.verifyAcls(AclEnforcement.java:136)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.daemon.nimbus.Nimbus.launch(Nimbus.java:1155)
~[storm-server-2.0.0.y.jar:2.0.0.y]
at org.apache.storm.daemon.nimbus.Nimbus.main(Nimbus.java:1162)
~[storm-server-2.0.0.y.jar:2.0.0.y]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)