subject:"Re\: Storm files and folders permissions on Linux"

Re: Storm files and folders permissions on Linux

2017-04-03 Thread Antoine Tran

Have you modified in Storm.yaml settings related to ACL? The defaults 
one disabled ACL for blob



On 03/04/2017 18:27, I PVP wrote:

Thanks for everyone that is answering and helping to look into this issue.



I tried from /tmp. Same error.
I tried submitting the jar from the ec2-user $HOME directory. Same error
I created a new directory to which storm:storm is the owner and tried 
submitting the jar  from there. Same error.



The error log is not really clear on what is going on.

Where/what is setup-path that is mentioned on the log? is it the 
/opt/storm where  storm was “installed”  ?


What is the exact "No such file or directory” that  is missing or 
inaccessible ?



First it says that :
—
2017-04-03 13:13:00.148 pool-15-thread-30 o.a.s.d.nimbus [INFO] 
Finished uploading file from client: 
/opt/storm/storm-local/nimbus/inbox/stormjar-f90ff743-e9ec-4e59-b945-878a47fd9057.j

ar
2017-04-03 13:13:00.173 pool-15-thread-32 o.a.s.d.nimbus [INFO] 
Received topology submission for Topology with conf 
{"topology.max.task.parallelism" nil, "topology
.submitter.principal" "", "topology.acker.executors" nil, 
"topology.eventlogger.executors" 0, "topology.workers" 1, 
"topology.debug" false, "storm.zookeeper.superACL" nil, "topology.us
ers" (), "topology.submitter.user" "storm", "topology.kryo.register" 
nil, "topology.kryo.decorators" (), "storm.id" 
"Topology-3-1491235980", "topology.name" "Emplo

yeePasswordResetTopology"}
2017-04-03 13:13:00.189 pool-15-thread-32 o.a.s.d.nimbus [INFO] 
uploadedJar 
/opt/storm/storm-local/nimbus/inbox/stormjar-f90ff743-e9ec-4e59-b945-878a47fd9057.jar
2017-04-03 13:13:00.189 pool-15-thread-32 
o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting

..



Before trowing the exception it says :
——
2017-04-03 13:13:00.207 pool-15-thread-32 o.a.s.cluster [INFO] 
setup-path/blobstore/Topology-3-1491235980-stormjar.jar/ip-172-31-5-153.sa-east-1.compute.internal:6

627-1
2017-04-03 13:13:00.207 pool-15-thread-32-EventThread 
o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2017-04-03 13:13:00.217 pool-15-thread-32 o.a.s.d.nimbus [WARN] 
Topology submission exception. (topology name='Topology') 
#error {

 :cause No such file or directory
 :via
 [{:type java.lang.RuntimeException
   :message java.io.IOException: No such file or directory
   :at [org.apache.storm.blobstore.LocalFsBlobStore createBlob 
LocalFsBlobStore.java 117]}

...
-



best,
IPVP

On April 3, 2017 at 12:12:02 PM, Santosh Pingale 
(pingalesant...@gmail.com ) wrote:


Can you try to submit the jar from a directory which is writable by 
storm user.  Try from /tmp maybe.


On Mon, Apr 3, 2017 at 3:56 PM, I PVP > wrote:


that is the error:


2017-04-03 10:31:44.535 pool-15-thread-56 o.a.s.d.nimbus [INFO]
Uploading file from client to

/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.820 pool-15-thread-17 o.a.s.d.nimbus [INFO]
Finished uploading file from client:

/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.842 pool-15-thread-20 o.a.s.d.nimbus [INFO]
Received topology submission for Topology with conf
{"topology.max.task.parallelism" nil,
"topology.submitter.principal" "", "topology.acker.executors"
nil, "topology.eventlogger.executors" 0, "topology.workers" 1,
"topology.debug" false, "storm.zookeeper.superACL" nil,
"topology.users" (), "topology.submitter.user" "storm",
"topology.kryo.register" nil, "topology.kryo.decorators" (),
"storm.id "
"Topology-2-1491226304", "topology.name
" "Topology"}
2017-04-03 10:31:44.846 pool-15-thread-20 o.a.s.d.nimbus [INFO]
uploadedJar

/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.847 pool-15-thread-20
o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-04-03 10:31:44.848 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper
[INFO] Initiating client connection,
connectString=172.31.5.153:2181
,172.31.3.29:2181
,172.31.0.218:2181/storm
 sessionTimeout=2
watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@13bf52ea
2017-04-03 10:31:44.851
pool-15-thread-20-SendThread(172.31.0.218:2181
) o.a.s.s.o.a.z.ClientCnxn [INFO]
Opening socket connection to server
172.31.0.218/172.31.0.218:2181
. Will not attempt to
authenticate using SASL (unknown error)
2017-04-03 10:31:44.852
pool-15-thread-20-SendThread(172.31.0.218:2181
) o.a.s.s.o.a.z.ClientCnxn [INFO]
Socket connection established to 172.31.0.218/172.31.0.218:2181

Re: Storm files and folders permissions on Linux

2017-04-03 Thread Santosh Pingale

Can you try to submit the jar from a directory which is writable by storm
user.  Try from /tmp maybe.

On Mon, Apr 3, 2017 at 3:56 PM, I PVP  wrote:

> that is the error:
>
>
> 2017-04-03 10:31:44.535 pool-15-thread-56 o.a.s.d.nimbus [INFO] Uploading
> file from client to /opt/storm/storm-local/nimbus/
> inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
> 2017-04-03 10:31:44.820 pool-15-thread-17 o.a.s.d.nimbus [INFO] Finished
> uploading file from client: /opt/storm/storm-local/nimbus/
> inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
> 2017-04-03 10:31:44.842 pool-15-thread-20 o.a.s.d.nimbus [INFO] Received
> topology submission for Topology with conf 
> {"topology.max.task.parallelism"
> nil, "topology.submitter.principal" "", "topology.acker.executors" nil,
> "topology.eventlogger.executors" 0, "topology.workers" 1,
> "topology.debug" false, "storm.zookeeper.superACL" nil, "topology.users"
> (), "topology.submitter.user" "storm", "topology.kryo.register" nil,
> "topology.kryo.decorators" (), "storm.id" 
> "Topology-2-1491226304",
> "topology.name" "Topology"}
> 2017-04-03 10:31:44.846 pool-15-thread-20 o.a.s.d.nimbus [INFO]
> uploadedJar /opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-
> 415d-803d-ab73c0d2fafe.jar
> 2017-04-03 10:31:44.847 pool-15-thread-20 
> o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl
> [INFO] Starting
> 2017-04-03 10:31:44.848 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO]
> Initiating client connection, connectString=172.31.5.153:2181,
> 172.31.3.29:2181,172.31.0.218:2181/storm sessionTimeout=2
> watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@13bf52ea
> 2017-04-03 10:31:44.851 pool-15-thread-20-SendThread(172.31.0.218:2181)
> o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server
> 172.31.0.218/172.31.0.218:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 2017-04-03 10:31:44.852 pool-15-thread-20-SendThread(172.31.0.218:2181)
> o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to
> 172.31.0.218/172.31.0.218:2181, initiating session
> 2017-04-03 10:31:44.855 pool-15-thread-20-SendThread(172.31.0.218:2181)
> o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server
> 172.31.0.218/172.31.0.218:2181, sessionid = 0x35b33b21dad0005, negotiated
> timeout = 2
> 2017-04-03 10:31:44.855 pool-15-thread-20-EventThread 
> o.a.s.s.o.a.c.f.s.ConnectionStateManager
> [INFO] State change: CONNECTED
> 2017-04-03 10:31:44.860 Curator-Framework-0 
> o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl
> [INFO] backgroundOperationsLoop exiting
> 2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO]
> Session: 0x35b33b21dad0005 closed
> 2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.cluster [INFO]
> setup-path/blobstore/Topology-2-1491226304-stormjar.jar/ip-
> 172-31-5-153.sa-east-1.compute.internal:6627-1
> 2017-04-03 10:31:44.862 pool-15-thread-20-EventThread
> o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
> 2017-04-03 10:31:44.877 pool-15-thread-20 o.a.s.d.nimbus [WARN] Topology
> submission exception. (topology name='Topology') #error {
>  :cause No such file or directory
>  :via
>  [{:type java.lang.RuntimeException
>:message java.io.IOException: No such file or directory
>:at [org.apache.storm.blobstore.LocalFsBlobStore createBlob
> LocalFsBlobStore.java 117]}
>   {:type java.io.IOException
>:message No such file or directory
>:at [java.io.UnixFileSystem createFileExclusively UnixFileSystem.java
> -2]}]
>  :trace
>  [[java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]
>   [java.io.File createNewFile File.java 1012]
>   [org.apache.storm.blobstore.LocalFsBlobStoreFile getOutputStream
> LocalFsBlobStoreFile.java 109]
>   [org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream 
> BlobStore.java 316]
>   [org.apache.storm.blobstore.LocalFsBlobStore createBlob
> LocalFsBlobStore.java 111]
>   [org.apache.storm.blobstore.BlobStore createBlob BlobStore.java 217]
>   [sun.reflect.NativeMethodAccessorImpl invoke0
> NativeMethodAccessorImpl.java -2]
>   [sun.reflect.NativeMethodAccessorImpl invoke
> NativeMethodAccessorImpl.java 62]
>   [sun.reflect.DelegatingMethodAccessorImpl invoke
> DelegatingMethodAccessorImpl.java 43]
>   [java.lang.reflect.Method invoke Method.java 498]
>   [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
>   [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
>   [org.apache.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 474]
>   [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138
> submitTopologyWithOpts nimbus.clj 1525]
>   [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138
> submitTopology nimbus.clj 1541]
>   [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult
> Nimbus.java 2912]
>   [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult
> Nimbus.java 2896]
>

Re: Storm files and folders permissions on Linux

2017-04-03 Thread I PVP

that is the error:


2017-04-03 10:31:44.535 pool-15-thread-56 o.a.s.d.nimbus [INFO] Uploading file 
from client to 
/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.820 pool-15-thread-17 o.a.s.d.nimbus [INFO] Finished 
uploading file from client: 
/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.842 pool-15-thread-20 o.a.s.d.nimbus [INFO] Received 
topology submission for Topology with conf 
{"topology.max.task.parallelism" nil, "topology.submitter.principal" "", 
"topology.acker.executors" nil, "topology.eventlogger.executors" 0, 
"topology.workers" 1, "topology.debug" false, "storm.zookeeper.superACL" nil, 
"topology.users" (), "topology.submitter.user" "storm", 
"topology.kryo.register" nil, "topology.kryo.decorators" (), "storm.id" 
"Topology-2-1491226304", "topology.name" "Topology"}
2017-04-03 10:31:44.846 pool-15-thread-20 o.a.s.d.nimbus [INFO] uploadedJar 
/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.847 pool-15-thread-20 
o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-04-03 10:31:44.848 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO] 
Initiating client connection, 
connectString=172.31.5.153:2181,172.31.3.29:2181,172.31.0.218:2181/storm 
sessionTimeout=2 
watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@13bf52ea
2017-04-03 10:31:44.851 pool-15-thread-20-SendThread(172.31.0.218:2181) 
o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server 
172.31.0.218/172.31.0.218:2181. Will not attempt to authenticate using SASL 
(unknown error)
2017-04-03 10:31:44.852 pool-15-thread-20-SendThread(172.31.0.218:2181) 
o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to 
172.31.0.218/172.31.0.218:2181, initiating session
2017-04-03 10:31:44.855 pool-15-thread-20-SendThread(172.31.0.218:2181) 
o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server 
172.31.0.218/172.31.0.218:2181, sessionid = 0x35b33b21dad0005, negotiated 
timeout = 2
2017-04-03 10:31:44.855 pool-15-thread-20-EventThread 
o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2017-04-03 10:31:44.860 Curator-Framework-0 
o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting
2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO] 
Session: 0x35b33b21dad0005 closed
2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.cluster [INFO] 
setup-path/blobstore/Topology-2-1491226304-stormjar.jar/ip-172-31-5-153.sa-east-1.compute.internal:6627-1
2017-04-03 10:31:44.862 pool-15-thread-20-EventThread o.a.s.s.o.a.z.ClientCnxn 
[INFO] EventThread shut down
2017-04-03 10:31:44.877 pool-15-thread-20 o.a.s.d.nimbus [WARN] Topology 
submission exception. (topology name='Topology') #error {
 :cause No such file or directory
 :via
 [{:type java.lang.RuntimeException
   :message java.io.IOException: No such file or directory
   :at [org.apache.storm.blobstore.LocalFsBlobStore createBlob 
LocalFsBlobStore.java 117]}
  {:type java.io.IOException
   :message No such file or directory
   :at [java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]}]
 :trace
 [[java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]
  [java.io.File createNewFile File.java 1012]
  [org.apache.storm.blobstore.LocalFsBlobStoreFile getOutputStream 
LocalFsBlobStoreFile.java 109]
  [org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream  
BlobStore.java 316]
  [org.apache.storm.blobstore.LocalFsBlobStore createBlob LocalFsBlobStore.java 
111]
  [org.apache.storm.blobstore.BlobStore createBlob BlobStore.java 217]
  [sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java 
-2]
  [sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java 62]
  [sun.reflect.DelegatingMethodAccessorImpl invoke 
DelegatingMethodAccessorImpl.java 43]
  [java.lang.reflect.Method invoke Method.java 498]
  [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
  [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
  [org.apache.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 474]
  [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138 
submitTopologyWithOpts nimbus.clj 1525]
  [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138 
submitTopology nimbus.clj 1541]
  [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult 
Nimbus.java 2912]
  [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult 
Nimbus.java 2896]
  [org.apache.storm.thrift.ProcessFunction process ProcessFunction.java 39]
  [org.apache.storm.thrift.TBaseProcessor process TBaseProcessor.java 39]
  [org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor 
process SimpleTransportPlugin.java 162]
  [org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer invoke

Re: Storm files and folders permissions on Linux

2017-03-31 Thread Harsh Choudhary

Can you show the exact error you getting?

*Cheers!*

Harsh Choudhary

On Thu, Mar 30, 2017 at 11:39 PM, I PVP  wrote:

> that is how it is being done as of now:
>
> sudo chown -R storm:storm /opt/storm
> sudo chmod -R 700 /opt/storm
>
> but still facing some issues while submitting topologies. looks like
> Nimbus is not being able to create blob store directories or files.
>
> i just tried sudo chmod -R 777 /opt/storm. but the issue still happening.
>
>
> On March 30, 2017 at 3:02:07 PM, Harsh Choudhary (shry.ha...@gmail.com)
> wrote:
>
> It depends on from which user, you are running storm. That user must own
> the folders of storm. So, you need not use chmod but chown.
>
> *Cheers!*
>
> Harsh Choudhary
>
> On Thu, Mar 30, 2017 at 11:24 PM, I PVP  wrote:
>
>> What are the recommended files/folders permissions for running Storm on
>> Linux ?
>>
>> Can I just set the entire folder tree to 700 or are there specific
>> files/folders that required a particular permission ?
>>
>>
>> I am installing Storm 1.0.3 on Red Hat Enterprise Linux 7.3.
>> Storm path is /opt/storm
>> user: storm:storm.
>>
>>
>> Thanks
>>
>> IPVP
>>
>>
>

Re: Storm files and folders permissions on Linux

2017-03-30 Thread Harsh Choudhary

It depends on from which user, you are running storm. That user must own
the folders of storm. So, you need not use chmod but chown.

*Cheers!*

Harsh Choudhary

On Thu, Mar 30, 2017 at 11:24 PM, I PVP  wrote:

> What are the recommended files/folders permissions for running Storm on
> Linux ?
>
> Can I just set the entire folder tree to 700 or are there specific
> files/folders that required a particular permission ?
>
>
> I am installing Storm 1.0.3 on Red Hat Enterprise Linux 7.3.
> Storm path is /opt/storm
> user: storm:storm.
>
>
> Thanks
>
> IPVP
>
>

Re: Storm files and folders permissions on Linux

Re: Storm files and folders permissions on Linux

Re: Storm files and folders permissions on Linux

Re: Storm files and folders permissions on Linux

Re: Storm files and folders permissions on Linux

5 matches

Site Navigation

Mail list logo

Footer information