[
https://issues.apache.org/jira/browse/STORM-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-3117.
----------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~agresch],
I merged this into master
> Deleting blobs for running topologies hoses Nimbus
> --------------------------------------------------
>
> Key: STORM-3117
> URL: https://issues.apache.org/jira/browse/STORM-3117
> Project: Apache Storm
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Aaron Gresch
> Assignee: Aaron Gresch
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The following test pseudo-code causes issues:
> {code:java}
> cluster.submitTopology(cluster.getTopologiesJarFile(), topoName, config,
> topology);
> cluster.waitTopologyUp(topoName);
> cluster.deleteAllBlobs();
> {code}
> This causes nimbus to get stuck and restart:
>
> {code:java}
> 2018-06-20 15:48:14.273 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Received
> topology submission for wc-topology-test (storm-0.10.2.y.251 JDK-1.8.0_131)
> 2018-06-20 15:48:14.629 o.a.s.d.n.Nimbus pool-27-thread-694 [INFO] Activating
> wc-topology-test: wc-topology-test-1-1529509694
> 2018-06-20 15:48:14.724 o.a.s.d.n.Nimbus pool-27-thread-703 [INFO]
> TRANSITION: wc-topology-test-1-1529509694 KILL null true
> 2018-06-20 15:48:14.812 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted
> blob for key wc-topology-test-1-1529509694-stormconf.ser
> 2018-06-20 15:48:14.830 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted
> blob for key wc-topology-test-1-1529509694-stormcode.ser
> 2018-06-20 15:48:14.863 o.a.s.d.n.Nimbus pool-27-thread-704 [INFO] Deleted
> blob for key wc-topology-test-1-1529509694-stormjar.jar
> 2018-06-20 15:48:18.449 o.a.s.s.r.s.p.DefaultSchedulingPriorityStrategy timer
> [INFO] SIM Scheduling wc-topology-test-1-1529509694 with score of 0.3125
> 2018-06-20 15:48:18.492 o.a.s.s.Cluster timer [INFO] STATUS -
> wc-topology-test-1-1529509694 Running - Fully Scheduled by
> DefaultResourceAwareStrategy
> 2018-06-20 15:48:18.527 o.a.s.d.n.Nimbus timer [INFO] Setting new assignment
> for topology id wc-topology-test-1-1529509694:
> 2018-06-20 15:48:18.979 o.a.s.d.n.Nimbus pool-27-thread-722 [WARN] get blob
> meta exception.
> org.apache.storm.utils.WrappedKeyNotFoundException:
> wc-topology-test-1-1529509694-stormjar.jar
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483)
> [storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38)
> [shaded-deps-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [shaded-deps-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291)
> [shaded-deps-2.0.0.y.jar:2.0.0.y]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
> 2018-06-20 15:48:22.884 o.a.s.d.n.Nimbus timer [INFO] Renewing Creds For
> wc-topology-test-1-1529509694 with
> org.apache.storm.security.auth.kerberos.AutoTGT@4482469c owned by
> [email protected]
> 2018-06-20 15:48:37.947 o.a.s.d.n.Nimbus timer [ERROR] Error while processing
> event
> java.lang.RuntimeException:
> KeyNotFoundException(msg:wc-topology-test-1-1529509694-stormcode.ser)
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2822)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at org.apache.storm.StormTimer$1.run(StormTimer.java:111)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:227)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> Caused by: org.apache.storm.utils.WrappedKeyNotFoundException:
> wc-topology-test-1-1529509694-stormcode.ser
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getBlobReplication(LocalFsBlobStore.java:420)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.getBlobReplicationCount(Nimbus.java:1517)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.getClusterInfoImpl(Nimbus.java:2675)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.sendClusterMetricsToExecutors(Nimbus.java:2686)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$launchServer$48(Nimbus.java:2819)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> ... 2 more
> 2018-06-20 15:48:37.948 o.a.s.u.Utils timer [ERROR] Halting process: Error
> while processing event
> java.lang.RuntimeException: Halting process: Error while processing event
> at org.apache.storm.utils.Utils.exitProcess(Utils.java:468)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.lambda$new$17(Nimbus.java:488)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:252)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> 2018-06-20 15:48:37.950 o.a.s.d.n.Nimbus Thread-11 [INFO] Shutting down master
> 2018-06-20 15:48:37.950 o.a.s.u.Utils Thread-12 [INFO] Halting after 10
> seconds
> 2018-06-20 15:48:46.672 o.a.s.d.n.Nimbus pool-27-thread-798 [WARN] get blob
> meta exception.
> org.apache.storm.utils.WrappedKeyNotFoundException:
> wc-topology-test-1-1529509694-stormconf.ser
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:256)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:286)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3483)
> [storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4011)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3990)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38)
> [shaded-deps-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> [shaded-deps-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.security.auth.sasl.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:147)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291)
> [shaded-deps-2.0.0.y.jar:2.0.0.y]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
> 2018-06-20 15:48:47.950 o.a.s.u.Utils Thread-12 [WARN] Forcing Halt...
> {code}
> Nimbus then continually restarts:
> {code:java}
> 2018-06-20 15:48:54.635 o.a.s.u.Utils main [ERROR] Received error in main
> thread.. terminating server...
> java.lang.Error: java.lang.IllegalStateException: Could not find credentials
> for topology wc-topology-test-1-1529509694 at path /storms. Don't know how to
> fix this automatically. Please add needed ACLs, or delete the path.
> at
> org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:603)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:582)
> ~[storm-client-2.0.0.y.jar:2.0.0.y]
> at org.apache.storm.utils.Utils$5.uncaughtException(Utils.java:931)
> [storm-client-2.0.0.y.jar:2.0.0.y]
> at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
> [?:1.8.0_131]
> at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
> [?:1.8.0_131]
> at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
> [?:1.8.0_131]
> Caused by: java.lang.IllegalStateException: Could not find credentials for
> topology wc-topology-test-1-1529509694 at path /storms. Don't know how to fix
> this automatically. Please add needed ACLs, or delete the path.
> at
> org.apache.storm.zookeeper.AclEnforcement.getTopoAcl(AclEnforcement.java:194)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.zookeeper.AclEnforcement.verifyParentWithTopoChildren(AclEnforcement.java:250)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.zookeeper.AclEnforcement.verifyParentWithReadOnlyTopoChildren(AclEnforcement.java:258)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at
> org.apache.storm.zookeeper.AclEnforcement.verifyAcls(AclEnforcement.java:136)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at org.apache.storm.daemon.nimbus.Nimbus.launch(Nimbus.java:1155)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> at org.apache.storm.daemon.nimbus.Nimbus.main(Nimbus.java:1162)
> ~[storm-server-2.0.0.y.jar:2.0.0.y]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
