So I did some testing today, I was able to recreate the exact ID string on a server with access to the share. (Remember the ID string that marathon runs is different than the standard user... for some reason the only groups that show up in marathon are the user's group and root(0). I recreated that exact same setup, and was still able to create the files running directly (not through mesos/marathon). The containerization I am using is docker,mesos would that play a role here?
Any other thoughts on what could be blocking the write? On Tue, May 12, 2015 at 3:09 PM, John Omernik <[email protected]> wrote: > Root IS able to write to the share outside of Mesos. I am working with > MapR to understand the NFS component better. > > > > On Tue, May 12, 2015 at 11:28 AM, Bjoern Metzdorf <[email protected]> > wrote: > >> Is there anything in the nfs server log files? Maybe it squashes root by >> default and the root group membership of darkness falls into that? >> >> Regards, >> Bjoern >> >> On May 12, 2015, at 5:53 AM, John Omernik <[email protected]> wrote: >> >> So I tried su darkness and su - darkness and both allowed a file write >> with no issues. On the group thing, while it is "weird" would that >> actually hurt ti to contain that group? Even if I set the directory to 777 >> I still get a failure. on a create within it. I am guessing this is >> something more to do with MapRs NFS than Mesos at this point, but if anyone >> would have any other tips on troubleshooting to confirm that, I'd >> appreciate it. >> >> John >> >> On Mon, May 11, 2015 at 5:18 PM, Marco Massenzio <[email protected]> >> wrote: >> >>> Looks to me that while 'uid' is 1000 >>> uid=1000(darkness) gid=1000(darkness) groups=1000(darkness),0(root) >>> >>> this is still root's env when run from Mesos (also, weird that groups >>> contains 0(root)): >>> USER=root >>> >>> again - not sure how we su to a different user, but this usually happens >>> if one does `su darkness` (instead of `su - darkness`) from the shell, at >>> any rate. >>> >>> *Marco Massenzio* >>> *Distributed Systems Engineer* >>> >>> On Mon, May 11, 2015 at 6:54 AM, John Omernik <[email protected]> wrote: >>> >>>> Paul: I checked in multiple places and I don't see rootsquash being >>>> used. I am using the MapR NFS server, and I do not believe that is a common >>>> option in the default setup ( I will follow up closer on that). >>>> >>>> Adam and Maxime: So I included the output of both id (instead of >>>> whoami) and env (as seen below) and I believe that your ideas may be >>>> getting somewhere. There are a number of things that strike me as odd in >>>> the outputs, and I'd like your thoughts on them. First of all, remember >>>> that the permissions on the folders are 775 right now, so with the primary >>>> group set (which it appears to be based on id) and the user set, it still >>>> should have write access. That said, the SUed process doesn't have any of >>>> the other groups (which I want to test if any of those controls access, >>>> especially with MapR). At risk of exposing to much information about my >>>> test network in a public forum, I left all the details in the ENV to see if >>>> there are things other may see that could be causing me issues. >>>> >>>> Thanks for the replies so far! >>>> >>>> >>>> >>>> >>>> >>>> *New Script:* >>>> >>>> #!/bin/bash >>>> >>>> echo "Writing id information to stderr for one stop logging" 1>&2 >>>> >>>> id 1>&2 >>>> >>>> >>>> echo "" 1>&2 >>>> >>>> >>>> echo "Printing out the env command to std err for one stop loggins" 1>& >>>> 2 >>>> >>>> env 1>&2 >>>> >>>> >>>> mkdir /mapr/brewpot/mesos/storm/test/test1 >>>> >>>> touch /mapr/brewpot/mesos/storm/test/test1/testing.go >>>> >>>> >>>> >>>> >>>> >>>> *Run within Mesos:* >>>> >>>> I0511 08:41:02.804448 8048 exec.cpp:132] Version: 0.21.0 >>>> I0511 08:41:02.814324 8059 exec.cpp:206] Executor registered on slave >>>> 20150505-145508-1644210368-5050-8608-S2 >>>> Writing id information to stderr for one stop logging >>>> uid=1000(darkness) gid=1000(darkness) groups=1000(darkness),0(root) >>>> >>>> Printing out the env command to std err for one stop loggins >>>> LIBPROCESS_IP=192.168.0.98 >>>> HOST=hadoopmapr3.brewingintel.com >>>> SHELL=/bin/bash >>>> TERM=unknown >>>> PORT_10005=31783 >>>> >>>> MESOS_DIRECTORY=/tmp/mesos/slaves/20150505-145508-1644210368-5050-8608-S2/frameworks/20150302-094409-1644210368-5050-2134-0003/executors/permtest.5f822976-f7e3-11e4-a22d-56847afe9799/runs/e53dc010-dd3c-4993-8f39-f8b532e5cf8b >>>> PORT0=31783 >>>> MESOS_TASK_ID=permtest.5f822976-f7e3-11e4-a22d-56847afe9799 >>>> USER=root >>>> LD_LIBRARY_PATH=:/usr/local/lib >>>> SUDO_USER=darkness >>>> MESOS_EXECUTOR_ID=permtest.5f822976-f7e3-11e4-a22d-56847afe9799 >>>> SUDO_UID=1000 >>>> USERNAME=root >>>> >>>> PATH=/home/darkness:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>> MAIL=/var/mail/root >>>> >>>> PWD=/opt/mapr/mesos/tmp/slave/slaves/20150505-145508-1644210368-5050-8608-S2/frameworks/20150302-094409-1644210368-5050-2134-0003/executors/permtest.5f822976-f7e3-11e4-a22d-56847afe9799/runs/e53dc010-dd3c-4993-8f39-f8b532e5cf8b >>>> MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos-0.21.0.so >>>> MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos-0.21.0.so >>>> LANG=en_US.UTF-8 >>>> PORTS=31783 >>>> MESOS_SLAVE_PID=slave(1)@192.168.0.98:5051 >>>> MESOS_FRAMEWORK_ID=20150302-094409-1644210368-5050-2134-0003 >>>> MESOS_CHECKPOINT=1 >>>> SUDO_COMMAND=/usr/local/bin/mesos daemon.sh mesos-slave --master= >>>> 192.168.0.98:5050 --ip=192.168.0.98 >>>> --log_dir=/opt/mapr/mesos/tmp/slave_log/ --containerizers=docker,mesos >>>> --gc_delay=600mins --disk_watch_interval=60secs >>>> HOME=/home/darkness >>>> SHLVL=2 >>>> LIBPROCESS_PORT=0 >>>> MARATHON_APP_ID=/permtest >>>> PYTHONPATH=:/usr/local/libexec/mesos/python >>>> MARATHON_APP_VERSION=2015-05-11T13:41:04.218Z >>>> LOGNAME=root >>>> MESOS_SLAVE_ID=20150505-145508-1644210368-5050-8608-S2 >>>> PORT=31783 >>>> SUDO_GID=1000 >>>> MESOS_RECOVERY_TIMEOUT=15mins >>>> _=/usr/bin/env >>>> mkdir: cannot create directory `/mapr/brewpot/mesos/storm/test/test1': >>>> Permission denied >>>> touch: cannot touch `/mapr/brewpot/mesos/storm/test/test1/testing.go': >>>> No such file or directory >>>> >>>> >>>> *Run from command line:* >>>> >>>> Writing id information to stderr for one stop logging >>>> uid=1000(darkness) gid=1000(darkness) >>>> groups=1000(darkness),4(adm),24(cdrom),27(sudo),30(dip),42(shadow),46(plugdev),111(lpadmin),112(sambashare),700(mapr),2000(brewclub),2001(lcusers) >>>> >>>> Printing out the env command to std err for one stop loggins >>>> SHELL=/bin/bash >>>> TERM=xterm-256color >>>> >>>> XDG_SESSION_COOKIE=fd12ce903630f14654f11d12000006ce-1431349941.139006-807917506 >>>> SSH_CLIENT=192.168.0.186 57204 22 >>>> SSH_TTY=/dev/pts/0 >>>> USER=darkness >>>> >>>> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36: >>>> >>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/lib/scala/bin >>>> MAIL=/var/mail/darkness >>>> PWD=/mnt >>>> LANG=en_US.UTF-8 >>>> NODE_PATH=/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript >>>> HOME=/home/darkness >>>> SHLVL=2 >>>> LOGNAME=darkness >>>> SSH_CONNECTION=192.168.0.186 57204 192.168.0.100 22 >>>> LESSOPEN=| /usr/bin/lesspipe %s >>>> LESSCLOSE=/usr/bin/lesspipe %s %s >>>> _=/usr/bin/env >>>> >>>> >>>> On Mon, May 11, 2015 at 1:05 AM, Maxime Brugidou < >>>> [email protected]> wrote: >>>> >>>>> Mesos does not set the groups of the process correctly. There is a >>>>> JIRA ticket for that. It only set the gid. I believe that this could >>>>> explain the issue if your user is in a specific NFS group to be able go >>>>> write. >>>>> >>>>> See >>>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/MESOS-719 >>>>> On May 11, 2015 3:51 AM, "Paul Brett" <[email protected]> wrote: >>>>> >>>>>> Can you check on the NFS server to see if the filesystem has been >>>>>> exported with the rootsquash option? That's a commonly used option which >>>>>> converts root uid on NFS clients to nobody on the server. >>>>>> >>>>>> -- Paul Brett >>>>>> On May 10, 2015 5:15 PM, "Adam Bordelon" <[email protected]> wrote: >>>>>> >>>>>>> Go ahead and run `env` in your script too, and see if there are any >>>>>>> interesting differences when run via Marathon vs. directly. >>>>>>> Maybe you're running in a different shell? >>>>>>> >>>>>>> On Sun, May 10, 2015 at 2:21 PM, John Omernik <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I believe the slave IS running as root. FWIW when I ran the script >>>>>>>> from above as root, it did work as intended (created the files on the >>>>>>>> NFS >>>>>>>> share). >>>>>>>> >>>>>>>> On Sun, May 10, 2015 at 9:08 AM, Dick Davies < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Any idea what user mesos is running as? This could just be a >>>>>>>>> filesystem permission >>>>>>>>> thing (ISTR last time I used NFS mounts, they had a 'root squash' >>>>>>>>> option that prevented >>>>>>>>> local root from writing to the NFS mount). >>>>>>>>> >>>>>>>>> On 9 May 2015 at 22:13, John Omernik <[email protected]> wrote: >>>>>>>>> > I am not specifying isolators. The Default? :) Is that a per >>>>>>>>> slave setting? >>>>>>>>> > >>>>>>>>> > On Sat, May 9, 2015 at 3:33 PM, James DeFelice < >>>>>>>>> [email protected]> >>>>>>>>> > wrote: >>>>>>>>> >> >>>>>>>>> >> What isolators are you using? >>>>>>>>> >> >>>>>>>>> >> On Sat, May 9, 2015 at 3:48 PM, John Omernik <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>> >>>>>>>>> >>> Marco... great idea... thank you. I just tried it and it >>>>>>>>> worked when I >>>>>>>>> >>> had a /mnt/permtesting with the same permissions. So it >>>>>>>>> appears something >>>>>>>>> >>> to do with NFS and Mesos (Remember I tested just NFS that >>>>>>>>> worked fine, it's >>>>>>>>> >>> the combination that is causing this). >>>>>>>>> >>> >>>>>>>>> >>> On Sat, May 9, 2015 at 1:09 PM, Marco Massenzio < >>>>>>>>> [email protected]> >>>>>>>>> >>> wrote: >>>>>>>>> >>>> >>>>>>>>> >>>> Out of my own curiousity (sorry, I have no fresh insights >>>>>>>>> into the issue >>>>>>>>> >>>> here) did you try to run the script and write to a non-NFS >>>>>>>>> mounted >>>>>>>>> >>>> directory? (same ownership/permissions) >>>>>>>>> >>>> >>>>>>>>> >>>> This way we could at least find out whether it's something >>>>>>>>> related to >>>>>>>>> >>>> NFS, or a more general permission-related issue. >>>>>>>>> >>>> >>>>>>>>> >>>> Marco Massenzio >>>>>>>>> >>>> Distributed Systems Engineer >>>>>>>>> >>>> >>>>>>>>> >>>> On Sat, May 9, 2015 at 5:10 AM, John Omernik < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>> >>>>>>>>> >>>>> Here is the testing I am doing. I used a simple script >>>>>>>>> (run.sh) It >>>>>>>>> >>>>> writes the user it is running as to stderr (so it's the same >>>>>>>>> log as the >>>>>>>>> >>>>> errors from file writing) and then tries to make a directory >>>>>>>>> in nfs, and >>>>>>>>> >>>>> then touch a file in nfs. Note: This script directly run >>>>>>>>> works on every >>>>>>>>> >>>>> node. You can see the JSON I used in marathon, and in the >>>>>>>>> sandbox results, >>>>>>>>> >>>>> you can see the user is indeed darkness and the directory >>>>>>>>> cannot be created. >>>>>>>>> >>>>> However when directly run, it the script, with the same >>>>>>>>> user, creates the >>>>>>>>> >>>>> directory with no issue. Now, I realize this COULD still >>>>>>>>> be a NFS quirk >>>>>>>>> >>>>> here, however, this testing points at some restriction in >>>>>>>>> how marathon kicks >>>>>>>>> >>>>> off the cmd. Any thoughts on where to look would be very >>>>>>>>> helpful! >>>>>>>>> >>>>> >>>>>>>>> >>>>> John >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> Script: >>>>>>>>> >>>>> >>>>>>>>> >>>>> #!/bin/bash >>>>>>>>> >>>>> echo "Writing whoami to stderr for one stop logging" 1>&2 >>>>>>>>> >>>>> whoami 1>&2 >>>>>>>>> >>>>> mkdir /mapr/brewpot/mesos/storm/test/test1 >>>>>>>>> >>>>> touch /mapr/brewpot/mesos/storm/test/test1/testing.go >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> Run Via Marathon >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> { >>>>>>>>> >>>>> "cmd": "/mapr/brewpot/mesos/storm/run.sh", >>>>>>>>> >>>>> "cpus": 1.0, >>>>>>>>> >>>>> "mem": 1024, >>>>>>>>> >>>>> "id": "permtest", >>>>>>>>> >>>>> "user": "darkness", >>>>>>>>> >>>>> "instances": 1 >>>>>>>>> >>>>> } >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> I0509 07:02:52.457242 9562 exec.cpp:132] Version: 0.21.0 >>>>>>>>> >>>>> I0509 07:02:52.462700 9570 exec.cpp:206] Executor >>>>>>>>> registered on slave >>>>>>>>> >>>>> 20150505-145508-1644210368-5050-8608-S0 >>>>>>>>> >>>>> Writing whoami to stderr for one stop logging >>>>>>>>> >>>>> darkness >>>>>>>>> >>>>> mkdir: cannot create directory >>>>>>>>> `/mapr/brewpot/mesos/storm/test/test1': >>>>>>>>> >>>>> Permission denied >>>>>>>>> >>>>> touch: cannot touch >>>>>>>>> `/mapr/brewpot/mesos/storm/test/test1/testing.go': >>>>>>>>> >>>>> No such file or directory >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> Run Via Shell: >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> $ /mapr/brewpot/mesos/storm/run.sh >>>>>>>>> >>>>> Writing whoami to stderr for one stop logging >>>>>>>>> >>>>> darkness >>>>>>>>> >>>>> darkness@hadoopmapr1:/mapr/brewpot/mesos/storm$ ls ./test/ >>>>>>>>> >>>>> test1 >>>>>>>>> >>>>> darkness@hadoopmapr1:/mapr/brewpot/mesos/storm$ ls >>>>>>>>> ./test/test1/ >>>>>>>>> >>>>> testing.go >>>>>>>>> >>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>>> On Sat, May 9, 2015 at 3:14 AM, Adam Bordelon < >>>>>>>>> [email protected]> >>>>>>>>> >>>>> wrote: >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> I don't know of anything inside of Mesos that would prevent >>>>>>>>> you from >>>>>>>>> >>>>>> writing to NFS. Maybe examine the environment variables set >>>>>>>>> when running as >>>>>>>>> >>>>>> that user. Or are you running in a Docker container? Those >>>>>>>>> can have >>>>>>>>> >>>>>> additional restrictions. >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> On Fri, May 8, 2015 at 4:44 PM, John Omernik < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> I am doing something where people may recommend against my >>>>>>>>> course of >>>>>>>>> >>>>>>> action. However, I am curious if there is "a way" >>>>>>>>> basically I have a process >>>>>>>>> >>>>>>> being kicked off in marathon that is trying to write to a >>>>>>>>> nfs location. The >>>>>>>>> >>>>>>> permissions of the user running the task and the nfs >>>>>>>>> location are good. So >>>>>>>>> >>>>>>> what component of mesos or marathon is keeping me from >>>>>>>>> writing here ? ( I >>>>>>>>> >>>>>>> am getting permission denied). Is this one of those things >>>>>>>>> that is just not >>>>>>>>> >>>>>>> allowed, or is there an option to pass to marathon to >>>>>>>>> allow this? Thanks ! >>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>> -- >>>>>>>>> >>>>>>> Sent from my iThing >>>>>>>>> >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> >>>>> >>>>>>>>> >>>> >>>>>>>>> >>> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> -- >>>>>>>>> >> James DeFelice >>>>>>>>> >> 585.241.9488 (voice) >>>>>>>>> >> 650.649.6071 (fax) >>>>>>>>> > >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>> >>> >> >
