Which directories should the user running Apache Storm be granted access? Storm path is /opt/storm user/group: storm:storm Storm 1.0.3 on Red Hat Enterprise Linux 7.3. Already did: sudo chown -R storm:storm /opt/storm/ sudo chmod -R 775 /opt/storm
3 machines .. 1 running nimbus and 2 running supervisors. all services start fine without errors on logs.. the error described on the previous messages happens specifically when trying to submit the topology jar. Unfortunately this error continues for days) even after terminating the instances and rebuilding instances from scratch and it is now preventing from going live with the MVP in production. Any help will be appreciated. best, IPVP On April 3, 2017 at 1:56:20 PM, I PVP ([email protected]<mailto:[email protected]>) wrote: No . I did not modified settings related to ACL. The only changes on storm yaml were: ----- storm.zookeeper.servers: nimbus.seeds: storm.local.dir: "/opt/storm" and adding supervisor.slots.ports ----- On April 3, 2017 at 1:31:16 PM, Antoine Tran ([email protected]<mailto:[email protected]>) wrote: Have you modified in Storm.yaml settings related to ACL? The defaults one disabled ACL for blob On 03/04/2017 18:27, I PVP wrote: Thanks for everyone that is answering and helping to look into this issue. I tried from /tmp. Same error. I tried submitting the jar from the ec2-user $HOME directory. Same error I created a new directory to which storm:storm is the owner and tried submitting the jar from there. Same error. The error log is not really clear on what is going on. Where/what is setup-path that is mentioned on the log? is it the /opt/storm where storm was “installed” ? What is the exact "No such file or directory” that is missing or inaccessible ? First it says that : — 2017-04-03 13:13:00.148 pool-15-thread-30 o.a.s.d.nimbus [INFO] Finished uploading file from client: /opt/storm/storm-local/nimbus/inbox/stormjar-f90ff743-e9ec-4e59-b945-878a47fd9057.j ar 2017-04-03 13:13:00.173 pool-15-thread-32 o.a.s.d.nimbus [INFO] Received topology submission for <COMPANYNAME>Topology with conf {"topology.max.task.parallelism" nil, "topology .submitter.principal" "", "topology.acker.executors" nil, "topology.eventlogger.executors" 0, "topology.workers" 1, "topology.debug" false, "storm.zookeeper.superACL" nil, "topology.us ers" (), "topology.submitter.user" "storm", "topology.kryo.register" nil, "topology.kryo.decorators" (), "storm.id" "<COMPANYNAME>Topology-3-1491235980", "topology.name" "Emplo yeePasswordResetTopology"} 2017-04-03 13:13:00.189 pool-15-thread-32 o.a.s.d.nimbus [INFO] uploadedJar /opt/storm/storm-local/nimbus/inbox/stormjar-f90ff743-e9ec-4e59-b945-878a47fd9057.jar 2017-04-03 13:13:00.189 pool-15-thread-32 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting .. ---- Before trowing the exception it says : —— 2017-04-03 13:13:00.207 pool-15-thread-32 o.a.s.cluster [INFO] setup-path/blobstore/<COMPANYNAME>Topology-3-1491235980-stormjar.jar/ip-172-31-5-153.sa-east-1.compute.internal:6 627-1 2017-04-03 13:13:00.207 pool-15-thread-32-EventThread o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down 2017-04-03 13:13:00.217 pool-15-thread-32 o.a.s.d.nimbus [WARN] Topology submission exception. (topology name='<COMPANYNAME>Topology') #error { :cause No such file or directory :via [{:type java.lang.RuntimeException :message java.io.IOException: No such file or directory :at [org.apache.storm.blobstore.LocalFsBlobStore createBlob LocalFsBlobStore.java 117]} ... ----- best, IPVP On April 3, 2017 at 12:12:02 PM, Santosh Pingale ([email protected]<mailto:[email protected]>) wrote: Can you try to submit the jar from a directory which is writable by storm user. Try from /tmp maybe. On Mon, Apr 3, 2017 at 3:56 PM, I PVP <[email protected]<mailto:[email protected]>> wrote: that is the error: 2017-04-03 10:31:44.535 pool-15-thread-56 o.a.s.d.nimbus [INFO] Uploading file from client to /opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar 2017-04-03 10:31:44.820 pool-15-thread-17 o.a.s.d.nimbus [INFO] Finished uploading file from client: /opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar 2017-04-03 10:31:44.842 pool-15-thread-20 o.a.s.d.nimbus [INFO] Received topology submission for <COMPANYNAME>Topology with conf {"topology.max.task.parallelism" nil, "topology.submitter.principal" "", "topology.acker.executors" nil, "topology.eventlogger.executors" 0, "topology.workers" 1, "topology.debug" false, "storm.zookeeper.superACL" nil, "topology.users" (), "topology.submitter.user" "storm", "topology.kryo.register" nil, "topology.kryo.decorators" (), "storm.id<http://storm.id>" "<COMPANYNAME>Topology-2-1491226304", "topology.name<http://topology.name>" "<COMPANYNAME>Topology"} 2017-04-03 10:31:44.846 pool-15-thread-20 o.a.s.d.nimbus [INFO] uploadedJar /opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar 2017-04-03 10:31:44.847 pool-15-thread-20 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2017-04-03 10:31:44.848 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=172.31.5.153:2181<http://172.31.5.153:2181>,172.31.3.29:2181<http://172.31.3.29:2181>,172.31.0.218:2181/storm<http://172.31.0.218:2181/storm> sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@13bf52ea 2017-04-03 10:31:44.851 pool-15-thread-20-SendThread(172.31.0.218:2181<http://172.31.0.218:2181>) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server 172.31.0.218/172.31.0.218:2181<http://172.31.0.218/172.31.0.218:2181>. Will not attempt to authenticate using SASL (unknown error) 2017-04-03 10:31:44.852 pool-15-thread-20-SendThread(172.31.0.218:2181<http://172.31.0.218:2181>) o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to 172.31.0.218/172.31.0.218:2181<http://172.31.0.218/172.31.0.218:2181>, initiating session 2017-04-03 10:31:44.855 pool-15-thread-20-SendThread(172.31.0.218:2181<http://172.31.0.218:2181>) o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server 172.31.0.218/172.31.0.218:2181<http://172.31.0.218/172.31.0.218:2181>, sessionid = 0x35b33b21dad0005, negotiated timeout = 20000 2017-04-03 10:31:44.855 pool-15-thread-20-EventThread o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2017-04-03 10:31:44.860 Curator-Framework-0 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting 2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x35b33b21dad0005 closed 2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.cluster [INFO] setup-path/blobstore/<COMPANYNAME>Topology-2-1491226304-stormjar.jar/ip-172-31-5-153.sa-east-1.compute.internal:6627-1 2017-04-03 10:31:44.862 pool-15-thread-20-EventThread o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down 2017-04-03 10:31:44.877 pool-15-thread-20 o.a.s.d.nimbus [WARN] Topology submission exception. (topology name='<COMPANYNAME>Topology') #error { :cause No such file or directory :via [{:type java.lang.RuntimeException :message java.io.IOException: No such file or directory :at [org.apache.storm.blobstore.LocalFsBlobStore createBlob LocalFsBlobStore.java 117]} {:type java.io.IOException :message No such file or directory :at [java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]}] :trace [[java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2] [java.io.File createNewFile File.java 1012] [org.apache.storm.blobstore.LocalFsBlobStoreFile getOutputStream LocalFsBlobStoreFile.java 109] [org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream <init> BlobStore.java 316] [org.apache.storm.blobstore.LocalFsBlobStore createBlob LocalFsBlobStore.java 111] [org.apache.storm.blobstore.BlobStore createBlob BlobStore.java 217] [sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java -2] [sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java 62] [sun.reflect.DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl.java 43] [java.lang.reflect.Method invoke Method.java 498] [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93] [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28] [org.apache.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 474] [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138 submitTopologyWithOpts nimbus.clj 1525] [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138 submitTopology nimbus.clj 1541] [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult Nimbus.java 2912] [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult Nimbus.java 2896] [org.apache.storm.thrift.ProcessFunction process ProcessFunction.java 39] [org.apache.storm.thrift.TBaseProcessor process TBaseProcessor.java 39] [org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor process SimpleTransportPlugin.java 162] [org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer invoke AbstractNonblockingServer.java 518] [org.apache.storm.thrift.server.Invocation run Invocation.java 18] [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1142] [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 617] [java.lang.Thread run Thread.java 745]]} 2017-04-03 10:31:44.877 pool-15-thread-20 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer [ERROR] Unexpected throwable while invoking! java.lang.RuntimeException: java.io.IOException: No such file or directory at org.apache.storm.blobstore.LocalFsBlobStore.createBlob(LocalFsBlobStore.java:117) at org.apache.storm.blobstore.BlobStore.createBlob(BlobStore.java:217) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) at org.apache.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:474) at org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138.submitTopologyWithOpts(nimbus.clj:1525) at org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138.submitTopology(nimbus.clj:1541) at org.apache.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:2912) at org.apache.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:2896) at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:162) at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:1012) at org.apache.storm.blobstore.LocalFsBlobStoreFile.getOutputStream(LocalFsBlobStoreFile.java:109) at org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.<init>(BlobStore.java:316) at org.apache.storm.blobstore.LocalFsBlobStore.createBlob(LocalFsBlobStore.java:111) ... 20 more — Storm 1.0.3 on AWS Red Hat Enterprise Linux 7.3. Storm path is /opt/storm user: storm:storm Thanks IPVP On March 31, 2017 at 3:10:05 AM, Harsh Choudhary ([email protected]<mailto:[email protected]>) wrote: Can you show the exact error you getting? Cheers! Harsh Choudhary On Thu, Mar 30, 2017 at 11:39 PM, I PVP <[email protected]<mailto:[email protected]>> wrote: that is how it is being done as of now: sudo chown -R storm:storm /opt/storm sudo chmod -R 700 /opt/storm but still facing some issues while submitting topologies. looks like Nimbus is not being able to create blob store directories or files. i just tried sudo chmod -R 777 /opt/storm. but the issue still happening. On March 30, 2017 at 3:02:07 PM, Harsh Choudhary ([email protected]<mailto:[email protected]>) wrote: It depends on from which user, you are running storm. That user must own the folders of storm. So, you need not use chmod but chown. Cheers! Harsh Choudhary On Thu, Mar 30, 2017 at 11:24 PM, I PVP <[email protected]<mailto:[email protected]>> wrote: What are the recommended files/folders permissions for running Storm on Linux ? Can I just set the entire folder tree to 700 or are there specific files/folders that required a particular permission ? I am installing Storm 1.0.3 on Red Hat Enterprise Linux 7.3. Storm path is /opt/storm user: storm:storm. Thanks IPVP -- My THALES email is [email protected]<mailto:[email protected]>. +33 (0)5 62 88 84 40 Thales Services, Toulouse, France
