James Xu created STORM-152:
------------------------------
Summary: Permissions of resources are not preserved
Key: STORM-152
URL: https://issues.apache.org/jira/browse/STORM-152
Project: Apache Storm (Incubating)
Issue Type: Improvement
Reporter: James Xu
https://github.com/nathanmarz/storm/issues/229
When the resources are copied from the resources directory to something like
/tmp/09c4e529-7e64-496e-8afa-4990b7aba206/supervisor/stormdist/word-count-1-1337616827/resources
the permissions of the files are not preserved, but rather all set to 644.
This causes a problem because then we cannot execute binary files as bolts and
spouts.
----------
tobigue: I can second this issue. The workaround I ended up with for now is
creating a shell script like this in the same folder:
chmod u+x ./relative/path/to/mybinary
./relative/path/to/mybinary
and calling
super("sh", "myscript.sh")
in the ShellSpout/ShellBolt instead of calling the binary directly.
----------
analogue: Tobigue,
I don't see how your workaround actually works because myscript.sh will not be
executable after after being unzipped and will fail to run.
My workaround is as follows:
For a given spout or bolt:
public static class MySpout extends ShellSpout implements IRichSpout {
public MySpout() {
super("bash", "-c", "chmod +x ./pystorm;./pystorm my_spout.py arg1
arg2 argN");
}
And a shell script multilang/resources/pystorm:
#!/bin/sh
PYTHONPATH=`pwd` python $*
This also handles not being able to set environment variables :-)
I'm thinking that ShellSpout should have an additional constructor which takes
a ProcessBuilder to provide maximum flexibility. There are downsides to this
though. The author of the spout or bolt will then have access to the
ProcessBuilder and could do something to screw things up like inadvertently
reading/writing to the InputStream or OutputStream in an effort to debug things.
----------
tobigue: Hi!
I think for me it worked, because I had the permission to execute sh.
So this works:
sh myscript.sh
while this does not
./myscript.sh
as there are no execute permissions on the file itself.
----------
analogue: Ok, I see my problem. bash -c ./myscript.sh fails if the script is
not executable but bash ./myscript.sh doesn't.
----------
alien2150: I am having the same problem with a C++ programm. The original
permissions are:
ls -l target/classes/resources/xy
-rwxr-xr-x 1 tech tech 1080184 Jul 19 17:26 target/classes/resources/xy
But later it is (ls -l
/tmp/448ce6ba-ccd2-4335-a119-903381aee09c/supervisor/stormdist/foo-1374248239/resources):
-rw-rw-rw- 1 tech tech 1080184 Jul 19 17:37 xy
Is there any solution for this?
----------
revans2: jar files are really just zip files and do not store any kind of
permissions. There are some extensions to zip that do store permissions but jar
does not use those. Because these resources are pushed to the nodes through a
jar the permissions are being lost. Hadoop gets around this by doing the really
ugly thing of changing all of the permissions to be user executable after
unzipping the file. We could try something like this in the supervisor when it
unzips the files but I would rather provide a different way of distributing
files beyond a single jar file once the bit-torrent code is in place.
----------
patricklucas: We can reproduce this consistently by launching a topology to a
"fresh" cluster. It appears that if the storm-local directory has to be
created, unpacked files will not have the right permissions. If it is a
subsequent run, and the storm-local directory already exists, permissions of
unpacked files are correct.
Will this be fixed in 0.9.0?
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
