Had anyone faced this issue?
Submitting a Topology from any supervisor machine is returning a exception.
It is a new environment setup.
Running Storm 1.0.3 Redhat linux AWS with java.version=1.8.0_121
All ports are open between all machines, until the issue is solved .
Machines are communicating using internal IPs
Filesystem permissions were temporary set to 777 for the entire /opt/storm/* ,
until the issue is solved
#####exception when submitting the topology at the supervisor command line (
this exception does not show up on supervisor.log)
012 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for
MD5-digest: -9206962661854850720:-6919200339480855527
1111 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
1158 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus :
stormnimbusprod1:6627
1167 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus :
stormnimbusprod1:6627
1172 [main] INFO o.a.s.StormSubmitter - Uploading topology jar <JARNAME>.jar
to assigned location:
/opt/storm/storm-local/nimbus/inbox/stormjar-99668ef8-4136-468f-bc67-cbb5a8da9ae3.jar
1679 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology jar to
assigned location:
/opt/storm/storm-local/nimbus/inbox/stormjar-99668ef8-4136-468f-bc67-cbb5a8da9ae3.jar
1684 [main] INFO o.a.s.u.NimbusClient - Found leader nimbus :
stormnimbusprod1:6627
1685 [main] INFO o.a.s.StormSubmitter - Submitting topology <TOPOLOGYNAME> in
distributed mode with conf
{"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-9206962661854850720:-6919200339480855527","topology.workers":1,"topology.debug":false}
Exception in thread "main" java.lang.RuntimeException:
org.apache.storm.thrift.transport.TTransportException
at
org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:255)
at
org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:310)
at
org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at <my package>.<TOPOLOGYNAME>.main(<TOPOLOGYNAME>.java:46)
Caused by: org.apache.storm.thrift.transport.TTransportException
at
org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at
org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at
org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at
org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at
org.apache.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:294)
at
org.apache.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:278)
at
org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:243)
... 3 more
######exception at the Ninbus.log#################
2017-03-29 22:13:16.349 pool-15-thread-32 o.a.s.cluster [INFO]
setup-path/blobstore/<TOPOLOGYNAME>-1-1490839996-stormjar.jar/stormnimbusprod1:6627-1
2017-03-29 22:13:16.349 pool-15-thread-32-EventThread o.a.s.s.o.a.z.ClientCnxn
[INFO] EventThread shut down
2017-03-29 22:13:16.389 pool-15-thread-32 o.a.s.d.nimbus [WARN] Topology
submission exception. (topology name='<TOPOLOGYNAME>') #error {
:cause No such file or directory
:via
[{:type java.lang.RuntimeException
:message java.io.IOException: No such file or directory
:at [org.apache.storm.blobstore.LocalFsBlobStore createBlob
LocalFsBlobStore.java 117]}
{:type java.io.IOException
:message No such file or directory
:at [java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]}]
:trace
[[java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]
[java.io.File createNewFile File.java 1012]
[org.apache.storm.blobstore.LocalFsBlobStoreFile getOutputStream
LocalFsBlobStoreFile.java 109]
[org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream <init>
BlobStore.java 316]
[org.apache.storm.blobstore.LocalFsBlobStore createBlob LocalFsBlobStore.java
111]
[org.apache.storm.blobstore.BlobStore createBlob BlobStore.java 217]
[sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java
-2]
[sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java 62]
[sun.reflect.DelegatingMethodAccessorImpl invoke
DelegatingMethodAccessorImpl.java 43]
[java.lang.reflect.Method invoke Method.java 498]
[clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
[clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
[org.apache.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 474]
[org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138
submitTopologyWithOpts nimbus.clj 1525]
[org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138
submitTopology nimbus.clj 1541]
[org.apache.storm.generated.Nimbus$Processor$submitTopology getResult
Nimbus.java 2912]
[org.apache.storm.generated.Nimbus$Processor$submitTopology getResult
Nimbus.java 2896]
[org.apache.storm.thrift.ProcessFunction process ProcessFunction.java 39]
[org.apache.storm.thrift.TBaseProcessor process TBaseProcessor.java 39]
[org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor
process SimpleTransportPlugin.java 162]
[org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer invoke
AbstractNonblockingServer.java 518]
[org.apache.storm.thrift.server.Invocation run Invocation.java 18]
[java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java
1142]
[java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java
617]
[java.lang.Thread run Thread.java 745]]}
2017-03-29 22:13:16.389 pool-15-thread-32
o.a.s.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Unexpected throwable
while invoking!
java.lang.RuntimeException: java.io.IOException: No such file or directory
at
org.apache.storm.blobstore.LocalFsBlobStore.createBlob(LocalFsBlobStore.java:117)
at org.apache.storm.blobstore.BlobStore.createBlob(BlobStore.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at
org.apache.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:474)
at
org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138.submitTopologyWithOpts(nimbus.clj:1525)
at
org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138.submitTopology(nimbus.clj:1541)
at
org.apache.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:2912)
at
org.apache.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:2896)
at
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:162)
at
org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No such file or directory
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at
org.apache.storm.blobstore.LocalFsBlobStoreFile.getOutputStream(LocalFsBlobStoreFile.java:109)
at
org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.<init>(BlobStore.java:316)
at
org.apache.storm.blobstore.LocalFsBlobStore.createBlob(LocalFsBlobStore.java:111)
... 20 more
any help will be appreciated.
best,
IPVP
