Mesos does not set the groups of the process correctly. There is a JIRA ticket for that. It only set the gid. I believe that this could explain the issue if your user is in a specific NFS group to be able go write.
See https://issues.apache.org/jira/plugins/servlet/mobile#issue/MESOS-719 On May 11, 2015 3:51 AM, "Paul Brett" <[email protected]> wrote: > Can you check on the NFS server to see if the filesystem has been exported > with the rootsquash option? That's a commonly used option which converts > root uid on NFS clients to nobody on the server. > > -- Paul Brett > On May 10, 2015 5:15 PM, "Adam Bordelon" <[email protected]> wrote: > >> Go ahead and run `env` in your script too, and see if there are any >> interesting differences when run via Marathon vs. directly. >> Maybe you're running in a different shell? >> >> On Sun, May 10, 2015 at 2:21 PM, John Omernik <[email protected]> wrote: >> >>> I believe the slave IS running as root. FWIW when I ran the script from >>> above as root, it did work as intended (created the files on the NFS >>> share). >>> >>> On Sun, May 10, 2015 at 9:08 AM, Dick Davies <[email protected]> >>> wrote: >>> >>>> Any idea what user mesos is running as? This could just be a >>>> filesystem permission >>>> thing (ISTR last time I used NFS mounts, they had a 'root squash' >>>> option that prevented >>>> local root from writing to the NFS mount). >>>> >>>> On 9 May 2015 at 22:13, John Omernik <[email protected]> wrote: >>>> > I am not specifying isolators. The Default? :) Is that a per slave >>>> setting? >>>> > >>>> > On Sat, May 9, 2015 at 3:33 PM, James DeFelice < >>>> [email protected]> >>>> > wrote: >>>> >> >>>> >> What isolators are you using? >>>> >> >>>> >> On Sat, May 9, 2015 at 3:48 PM, John Omernik <[email protected]> >>>> wrote: >>>> >>> >>>> >>> Marco... great idea... thank you. I just tried it and it worked >>>> when I >>>> >>> had a /mnt/permtesting with the same permissions. So it appears >>>> something >>>> >>> to do with NFS and Mesos (Remember I tested just NFS that worked >>>> fine, it's >>>> >>> the combination that is causing this). >>>> >>> >>>> >>> On Sat, May 9, 2015 at 1:09 PM, Marco Massenzio < >>>> [email protected]> >>>> >>> wrote: >>>> >>>> >>>> >>>> Out of my own curiousity (sorry, I have no fresh insights into the >>>> issue >>>> >>>> here) did you try to run the script and write to a non-NFS mounted >>>> >>>> directory? (same ownership/permissions) >>>> >>>> >>>> >>>> This way we could at least find out whether it's something related >>>> to >>>> >>>> NFS, or a more general permission-related issue. >>>> >>>> >>>> >>>> Marco Massenzio >>>> >>>> Distributed Systems Engineer >>>> >>>> >>>> >>>> On Sat, May 9, 2015 at 5:10 AM, John Omernik <[email protected]> >>>> wrote: >>>> >>>>> >>>> >>>>> Here is the testing I am doing. I used a simple script (run.sh) >>>> It >>>> >>>>> writes the user it is running as to stderr (so it's the same log >>>> as the >>>> >>>>> errors from file writing) and then tries to make a directory in >>>> nfs, and >>>> >>>>> then touch a file in nfs. Note: This script directly run works >>>> on every >>>> >>>>> node. You can see the JSON I used in marathon, and in the >>>> sandbox results, >>>> >>>>> you can see the user is indeed darkness and the directory cannot >>>> be created. >>>> >>>>> However when directly run, it the script, with the same user, >>>> creates the >>>> >>>>> directory with no issue. Now, I realize this COULD still be a >>>> NFS quirk >>>> >>>>> here, however, this testing points at some restriction in how >>>> marathon kicks >>>> >>>>> off the cmd. Any thoughts on where to look would be very >>>> helpful! >>>> >>>>> >>>> >>>>> John >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> Script: >>>> >>>>> >>>> >>>>> #!/bin/bash >>>> >>>>> echo "Writing whoami to stderr for one stop logging" 1>&2 >>>> >>>>> whoami 1>&2 >>>> >>>>> mkdir /mapr/brewpot/mesos/storm/test/test1 >>>> >>>>> touch /mapr/brewpot/mesos/storm/test/test1/testing.go >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> Run Via Marathon >>>> >>>>> >>>> >>>>> >>>> >>>>> { >>>> >>>>> "cmd": "/mapr/brewpot/mesos/storm/run.sh", >>>> >>>>> "cpus": 1.0, >>>> >>>>> "mem": 1024, >>>> >>>>> "id": "permtest", >>>> >>>>> "user": "darkness", >>>> >>>>> "instances": 1 >>>> >>>>> } >>>> >>>>> >>>> >>>>> >>>> >>>>> I0509 07:02:52.457242 9562 exec.cpp:132] Version: 0.21.0 >>>> >>>>> I0509 07:02:52.462700 9570 exec.cpp:206] Executor registered on >>>> slave >>>> >>>>> 20150505-145508-1644210368-5050-8608-S0 >>>> >>>>> Writing whoami to stderr for one stop logging >>>> >>>>> darkness >>>> >>>>> mkdir: cannot create directory >>>> `/mapr/brewpot/mesos/storm/test/test1': >>>> >>>>> Permission denied >>>> >>>>> touch: cannot touch >>>> `/mapr/brewpot/mesos/storm/test/test1/testing.go': >>>> >>>>> No such file or directory >>>> >>>>> >>>> >>>>> >>>> >>>>> Run Via Shell: >>>> >>>>> >>>> >>>>> >>>> >>>>> $ /mapr/brewpot/mesos/storm/run.sh >>>> >>>>> Writing whoami to stderr for one stop logging >>>> >>>>> darkness >>>> >>>>> darkness@hadoopmapr1:/mapr/brewpot/mesos/storm$ ls ./test/ >>>> >>>>> test1 >>>> >>>>> darkness@hadoopmapr1:/mapr/brewpot/mesos/storm$ ls ./test/test1/ >>>> >>>>> testing.go >>>> >>>>> >>>> >>>>> >>>> >>>>> On Sat, May 9, 2015 at 3:14 AM, Adam Bordelon <[email protected] >>>> > >>>> >>>>> wrote: >>>> >>>>>> >>>> >>>>>> I don't know of anything inside of Mesos that would prevent you >>>> from >>>> >>>>>> writing to NFS. Maybe examine the environment variables set when >>>> running as >>>> >>>>>> that user. Or are you running in a Docker container? Those can >>>> have >>>> >>>>>> additional restrictions. >>>> >>>>>> >>>> >>>>>> On Fri, May 8, 2015 at 4:44 PM, John Omernik <[email protected]> >>>> wrote: >>>> >>>>>>> >>>> >>>>>>> I am doing something where people may recommend against my >>>> course of >>>> >>>>>>> action. However, I am curious if there is "a way" basically I >>>> have a process >>>> >>>>>>> being kicked off in marathon that is trying to write to a nfs >>>> location. The >>>> >>>>>>> permissions of the user running the task and the nfs location >>>> are good. So >>>> >>>>>>> what component of mesos or marathon is keeping me from writing >>>> here ? ( I >>>> >>>>>>> am getting permission denied). Is this one of those things that >>>> is just not >>>> >>>>>>> allowed, or is there an option to pass to marathon to allow >>>> this? Thanks ! >>>> >>>>>>> >>>> >>>>>>> -- >>>> >>>>>>> Sent from my iThing >>>> >>>>>> >>>> >>>>>> >>>> >>>>> >>>> >>>> >>>> >>> >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> James DeFelice >>>> >> 585.241.9488 (voice) >>>> >> 650.649.6071 (fax) >>>> > >>>> > >>>> >>> >>> >>
