Aaron Gresch created STORM-3117:
-----------------------------------

             Summary: Deleting blobs for running topologies hoses Nimbus
                 Key: STORM-3117
                 URL: https://issues.apache.org/jira/browse/STORM-3117
             Project: Apache Storm
          Issue Type: Bug
    Affects Versions: 2.0.0
            Reporter: Aaron Gresch
            Assignee: Aaron Gresch


The following test pseudo-code causes issues:
{code:java}
cluster.submitTopology(cluster.getTopologiesJarFile(), topoName, config, 
topology);
cluster.waitTopologyUp(topoName);
cluster.deleteAllBlobs();
{code}
This causes nimbus to get stuck and restart:

 
{code:java}
2018-06-20 15:48:14.273 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Received 
topology submission for wc-topology-test (storm-0.10.2.y.251 JDK-1.8.0_131) 
2018-06-20 15:48:14.629 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Activating 
wc-topology-test: wc-topology-test-1-1529509694
2018-06-20 15:48:14.724 o.a.s.d.n.Nimbus pool-27-thread-703 [INFO] TRANSITION: 
wc-topology-test-1-1529509694 KILL null true
2018-06-20 15:48:14.812 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob 
for key wc-topology-test-1-1529509694-stormconf.ser
2018-06-20 15:48:14.830 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob 
for key wc-topology-test-1-1529509694-stormcode.ser
2018-06-20 15:48:14.863 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted blob 
for key wc-topology-test-1-1529509694-stormjar.jar
2018-06-20 15:48:18.449 o.a.s.s.r.s.p.DefaultSchedulingPriorityStrategy timer 
[INFO] SIM Scheduling wc-topology-test-1-1529509694 with score of 0.3125
2018-06-20 15:48:18.492 o.a.s.s.Cluster timer [INFO] STATUS - 
wc-topology-test-1-1529509694 Running - Fully Scheduled by 
DefaultResourceAwareStrategy
2018-06-20 15:48:18.527 o.a.s.d.n.Nimbus timer [INFO] Setting new assignment 
for topology id wc-topology-test-1-1529509694:

2018-06-20 15:48:18.979 o.a.s.d.n.Nimbus pool-27-thread-722 [WARN] get blob 
meta exception.
org.apache.storm.utils.WrappedKeyNotFoundException: 
wc-topology-test-1-1529509694-stormjar.jar
        at 
org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256)
 ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286)
 ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483) 
[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011)
 [storm-client-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990)
 [storm-client-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) 
[shaded-deps-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
[shaded-deps-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147)
 [storm-client-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291)
 [shaded-deps-2.0.0.y.jar:2.0.0.y]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_131]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

2018-06-20 15:48:22.884 o.a.s.d.n.Nimbus timer [INFO] Renewing Creds For 
wc-topology-test-1-1529509694 with 
org.apache.storm.security.auth.kerberos.AutoTGT@4482469c owned by 
[email protected]


2018-06-20 15:48:37.947 o.a.s.d.n.Nimbus timer [ERROR] Error while processing 
event
java.lang.RuntimeException: 
KeyNotFoundException(msg:wc-topology-test-1-1529509694-stormcode.ser)
        at 
org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2822) 
~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.StormTimer$1.run(StormTimer.java:111) 
~[storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:227) 
[storm-client-2.0.0.y.jar:2.0.0.y]
Caused by: org.apache.storm.utils.WrappedKeyNotFoundException: 
wc-topology-test-1-1529509694-stormcode.ser
        at 
org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256)
 ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.blobstore.LocalFsBlobStore.getBlobReplication(LocalFsBlobStore.java:420)
 ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.daemon.nimbus.Nimbus.getBlobReplicationCount(Nimbus.java:1517) 
~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.daemon.nimbus.Nimbus.getClusterInfoImpl(Nimbus.java:2675) 
~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.daemon.nimbus.Nimbus.sendClusterMetricsToExecutors(Nimbus.java:2686)
 ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2819) 
~[storm-server-2.0.0.y.jar:2.0.0.y]
        ... 2 more
2018-06-20 15:48:37.948 o.a.s.u.Utils timer [ERROR] Halting process: Error 
while processing event
java.lang.RuntimeException: Halting process: Error while processing event
        at org.apache.storm.utils.Utils.exitProcess(Utils.java:468) 
~[storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.lambda$new$17(Nimbus.java:488) 
~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:252) 
[storm-client-2.0.0.y.jar:2.0.0.y]
2018-06-20 15:48:37.950 o.a.s.d.n.Nimbus Thread-11 [INFO] Shutting down master
2018-06-20 15:48:37.950 o.a.s.u.Utils Thread-12 [INFO] Halting after 10 seconds

2018-06-20 15:48:46.672 o.a.s.d.n.Nimbus pool-27-thread-798 [WARN] get blob 
meta exception.
org.apache.storm.utils.WrappedKeyNotFoundException: 
wc-topology-test-1-1529509694-stormconf.ser
        at 
org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256)
 ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286)
 ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483) 
[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011)
 [storm-client-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990)
 [storm-client-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) 
[shaded-deps-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
[shaded-deps-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147)
 [storm-client-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291)
 [shaded-deps-2.0.0.y.jar:2.0.0.y]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_131]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2018-06-20 15:48:47.950 o.a.s.u.Utils Thread-12 [WARN] Forcing Halt...

{code}
Nimbus then continually restarts:
{code:java}
2018-06-20 15:48:54.635 o.a.s.u.Utils main [ERROR] Received error in main 
thread.. terminating server...
java.lang.Error: java.lang.IllegalStateException: Could not find credentials 
for topology wc-topology-test-1-1529509694 at path /storms. Don't know how to 
fix this automatically. Please add needed ACLs, or delete the path.
        at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:603) 
~[storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:582) 
~[storm-client-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.utils.Utils$5.uncaughtException(Utils.java:931) 
[storm-client-2.0.0.y.jar:2.0.0.y]
        at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057) 
[?:1.8.0_131]
        at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052) 
[?:1.8.0_131]
        at java.lang.Thread.dispatchUncaughtException(Thread.java:1959) 
[?:1.8.0_131]
Caused by: java.lang.IllegalStateException: Could not find credentials for 
topology wc-topology-test-1-1529509694 at path /storms. Don't know how to fix 
this automatically. Please add needed ACLs, or delete the path.
        at 
org.apache.storm.zookeeper.AclEnforcement.getTopoAcl(AclEnforcement.java:194) 
~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.zookeeper.AclEnforcement.verifyParentWithTopoChildren(AclEnforcement.java:250)
 ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.zookeeper.AclEnforcement.verifyParentWithReadOnlyTopoChildren(AclEnforcement.java:258)
 ~[storm-server-2.0.0.y.jar:2.0.0.y]
        at 
org.apache.storm.zookeeper.AclEnforcement.verifyAcls(AclEnforcement.java:136) 
~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.launch(Nimbus.java:1155) 
~[storm-server-2.0.0.y.jar:2.0.0.y]
        at org.apache.storm.daemon.nimbus.Nimbus.main(Nimbus.java:1162) 
~[storm-server-2.0.0.y.jar:2.0.0.y]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to