Looks like zookeeper data dir permission issue. Check zoo.cfg and grant permission.
On Tuesday, April 4, 2017, I PVP <[email protected]> wrote: > Is Storm supposed to run as root ? > Running storm as root was the only thing that solved this issue. > > On April 3, 2017 at 5:49:24 PM, I PVP ([email protected]) wrote: > > Which directories should the user running Apache Storm be granted access? > Storm path is /opt/storm > user/group: storm:storm > Storm 1.0.3 on Red Hat Enterprise Linux 7.3. > Already did: > sudo chown -R storm:storm /opt/storm/ > sudo chmod -R 775 /opt/storm > 3 machines .. 1 running nimbus and 2 running supervisors. all services start fine without errors on logs.. the error described on the previous messages happens specifically when trying to submit the topology jar. > Unfortunately this error continues for days) even after terminating the instances and rebuilding instances from scratch and it is now preventing from going live with the MVP in production. > Any help will be appreciated. > best, > IPVP > > On April 3, 2017 at 1:56:20 PM, I PVP ([email protected]) wrote: > > No . I did not modified settings related to ACL. > The only changes on storm yaml were: > ----- > storm.zookeeper.servers: > nimbus.seeds: > storm.local.dir: "/opt/storm" > and adding supervisor.slots.ports > ----- > > On April 3, 2017 at 1:31:16 PM, Antoine Tran ( [email protected]) wrote: > > Have you modified in Storm.yaml settings related to ACL? The defaults one disabled ACL for blob > > On 03/04/2017 18:27, I PVP wrote: > > Thanks for everyone that is answering and helping to look into this issue. > > > I tried from /tmp. Same error. > I tried submitting the jar from the ec2-user $HOME directory. Same error > I created a new directory to which storm:storm is the owner and tried submitting the jar from there. Same error. > > The error log is not really clear on what is going on. > Where/what is setup-path that is mentioned on the log? is it the /opt/storm where storm was “installed” ? > What is the exact "No such file or directory” that is missing or inaccessible ? > > First it says that : > — > 2017-04-03 13:13:00.148 pool-15-thread-30 o.a.s.d.nimbus [INFO] Finished uploading file from client: /opt/storm/storm-local/nimbus/inbox/stormjar-f90ff743-e9ec-4e59-b945-878a47fd9057.j > ar > 2017-04-03 13:13:00.173 pool-15-thread-32 o.a.s.d.nimbus [INFO] Received topology submission for <COMPANYNAME>Topology with conf {"topology.max.task.parallelism" nil, "topology > .submitter.principal" "", "topology.acker.executors" nil, "topology.eventlogger.executors" 0, "topology.workers" 1, "topology.debug" false, "storm.zookeeper.superACL" nil, "topology.us > ers" (), "topology.submitter.user" "storm", "topology.kryo.register" nil, "topology.kryo.decorators" (), "storm.id" "<COMPANYNAME>Topology-3-1491235980", "topology.name" "Emplo > yeePasswordResetTopology"} > 2017-04-03 13:13:00.189 pool-15-thread-32 o.a.s.d.nimbus [INFO] uploadedJar /opt/storm/storm-local/nimbus/inbox/stormjar-f90ff743-e9ec-4e59-b945-878a47fd9057.jar > 2017-04-03 13:13:00.189 pool-15-thread-32 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting > .. > ---- > > Before trowing the exception it says : > —— > 2017-04-03 13:13:00.207 pool-15-thread-32 o.a.s.cluster [INFO] setup-path/blobstore/<COMPANYNAME>Topology-3-1491235980-stormjar.jar/ip-172-31-5-153.sa-east-1.compute.internal:6 > 627-1 > 2017-04-03 13:13:00.207 pool-15-thread-32-EventThread o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down > 2017-04-03 13:13:00.217 pool-15-thread-32 o.a.s.d.nimbus [WARN] Topology submission exception. (topology name='<COMPANYNAME>Topology') #error { > :cause No such file or directory > :via > [{:type java.lang.RuntimeException > :message java.io.IOException: No such file or directory > :at [org.apache.storm.blobstore.LocalFsBlobStore createBlob LocalFsBlobStore.java 117]} > ... > ----- > > > best, > IPVP > > On April 3, 2017 at 12:12:02 PM, Santosh Pingale ([email protected]) wrote: > > Can you try to submit the jar from a directory which is writable by storm user. Try from /tmp maybe. > On Mon, Apr 3, 2017 at 3:56 PM, I PVP <[email protected]> wrote: >> >> that is the error: >> >> 2017-04-03 10:31:44.535 pool-15-thread-56 o.a.s.d.nimbus [INFO] Uploading file from client to /opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar >> 2017-04-03 10:31:44.820 pool-15-thread-17 o.a.s.d.nimbus [INFO] Finished uploading file from client: /opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar >> 2017-04-03 10:31:44.842 pool-15-thread-20 o.a.s.d.nimbus [INFO] Received topology submission for <COMPANYNAME>Topology with conf {"topology.max.task.parallelism" nil, "topology.submitter.principal" "", "topology.acker.executors" nil, "topology.eventlogger.executors" 0, "topology.workers" 1, "topology.debug" false, "storm.zookeeper.superACL" nil, "topology.users" (), "topology.submitter.user" "storm", "topology.kryo.register" nil, "topology.kryo.decorators" (), "storm.id" "<COMPANYNAME>Topology-2-1491226304", "topology.name" "<COMPANYNAME>Topology"} >> 2017-04-03 10:31:44.846 pool-15-thread-20 o.a.s.d.nimbus [INFO] uploadedJar /opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar >> 2017-04-03 10:31:44.847 pool-15-thread-20 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting >> 2017-04-03 10:31:44.848 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=172.31.5.153:2181, 172.31.3.29:2181,172.31.0.218:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@13bf52ea >> 2017-04-03 10:31:44.851 pool-15-thread-20-SendThread(172.31.0.218:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server 172.31.0.218/172.31.0.218:2181. Will not attempt to authenticate using SASL (unknown error) >> 2017-04-03 10:31:44.852 pool-15-thread-20-SendThread(172.31.0.218:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to 172.31.0.218/172.31.0.218:2181, initiating session >> 2017-04-03 10:31:44.855 pool-15-thread-20-SendThread(172.31.0.218:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server 172.31.0.218/172.31.0.218:2181, sessionid = 0x35b33b21dad0005, negotiated timeout = 20000 >> 2017-04-03 10:31:44.855 pool-15-thread-20-EventThread o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED >> 2017-04-03 10:31:44.860 Curator-Framework-0 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting >> 2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x35b33b21dad0005 closed >> 2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.cluster [INFO] setup-path/blobstore/<COMPANYNAME>Topology-2-1491226304-stormjar.jar/ip-172-31-5-153.sa-east-1.compute.internal:6627-1 >> 2017-04-03 10:31:44.862 pool-15-thread-20-EventThread o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down >> 2017-04-03 10:31:44.877 pool-15-thread-20 o.a.s.d.nimbus [WARN] Topology submission exception. (topology name='<COMPANYNAME>Topology') #error { >> :cause No such file or directory >> :via >> [{:type java.lang.RuntimeException >> :message java.io.IOException: No such file or directory >> :at [org.apache.storm.blobstore.LocalFsBlobStore createBlob LocalFsBlobStore.java 117]} >> {:type java.io.IOException >> :message No such file or directory >> :at [java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]}] >> :trace >> [[java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2] >> [java.io.File createNewFile File.java 1012] >> [org.apache.storm.blobstore.LocalFsBlobStoreFile getOutputStream LocalFsBlobStoreFile.java 109] >> [org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream <init> BlobStore.java 316] >> [org.apache.storm.blobstore.LocalFsBlobStore createBlob LocalFsBlobStore.java 111] >> [org.apache.storm.blobstore.BlobStore createBlob BlobStore.java 217] >> [sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java -2] >> [sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java 62] >> [sun.reflect.DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl.java 43] >> [java.lang.reflect.Method invoke Method.java 498] >> [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93] >> [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28] >> [org.apache.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 474] >> [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138 submitTopologyWithOpts nimbus.clj 1525] >> [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138 submitTopology nimbus.clj 1541] >> [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult Nimbus.java 2912] >> [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult Nimbus.java 2896] >> [org.apache.storm.thrift.ProcessFunction process ProcessFunction.java 39] >> [org.apache.storm.thrift.TBaseProcessor process TBaseProcessor.java 39] >> [org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor process SimpleTransportPlugin.java 162] >> [org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer invoke AbstractNonblockingServer.java 518] >> [org.apache.storm.thrift.server.Invocation run Invocation.java 18] >> [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1142] >> [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 617] >> [java.lang.Thread run Thread.java 745]]} >> 2017-04-03 10:31:44.877 pool-15-thread-20 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Unexpected throwable while invoking! >> java.lang.RuntimeException: java.io.IOException: No such file or directory >> at org.apache.storm.blobstore.LocalFsBlobStore.createBlob(LocalFsBlobStore.java:117) >> at org.apache.storm.blobstore.BlobStore.createBlob(BlobStore.java:217) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) >> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) >> at org.apache.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:474) >> at org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138.submitTopologyWithOpts(nimbus.clj:1525) >> at org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138.submitTopology(nimbus.clj:1541) >> at org.apache.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:2912) >> at org.apache.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:2896) >> at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) >> at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) >> at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:162) >> at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) >> at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.io.IOException: No such file or directory >> at java.io.UnixFileSystem.createFileExclusively(Native Method) >> at java.io.File.createNewFile(File.java:1012) >> at org.apache.storm.blobstore.LocalFsBlobStoreFile.getOutputStream(LocalFsBlobStoreFile.java:109) >> at org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.<init>(BlobStore.java:316) >> at org.apache.storm.blobstore.LocalFsBlobStore.createBlob(LocalFsBlobStore.java:111) >> ... 20 more >> — >> Storm 1.0.3 on AWS Red Hat Enterprise Linux 7.3. >> Storm path is /opt/storm >> user: storm:storm >> Thanks >> IPVP >> >> On March 31, 2017 at 3:10:05 AM, Harsh Choudhary ([email protected]) wrote: >> >> Can you show the exact error you getting? >> >> Cheers! >> >> Harsh Choudhary >> >> On Thu, Mar 30, 2017 at 11:39 PM, I PVP <[email protected]> wrote: >>> >>> that is how it is being done as of now: >>> sudo chown -R storm:storm /opt/storm >>> sudo chmod -R 700 /opt/storm >>> but still facing some issues while submitting topologies. looks like Nimbus is not being able to create blob store directories or files. >>> i just tried sudo chmod -R 777 /opt/storm. but the issue still happening. >>> >>> On March 30, 2017 at 3:02:07 PM, Harsh Choudhary ([email protected]) wrote: >>> >>> It depends on from which user, you are running storm. That user must own the folders of storm. So, you need not use chmod but chown. >>> >>> Cheers! >>> >>> Harsh Choudhary >>> >>> On Thu, Mar 30, 2017 at 11:24 PM, I PVP <[email protected]> wrote: >>>> >>>> What are the recommended files/folders permissions for running Storm on Linux ? >>>> Can I just set the entire folder tree to 700 or are there specific files/folders that required a particular permission ? >>>> >>>> I am installing Storm 1.0.3 on Red Hat Enterprise Linux 7.3. >>>> Storm path is /opt/storm >>>> user: storm:storm. >>>> >>>> Thanks >>>> IPVP >>> >> > > > -- > My THALES email is [email protected]. > +33 (0)5 62 88 84 40 > Thales Services, Toulouse, France
