Chris, Thanks for the info - we do suspect it's abount mesos-execute, yes.
Thanks, June Taylor System Administrator, Minnesota Population Center University of Minnesota On Thu, Jun 30, 2016 at 1:10 PM, Chris Baker <[email protected]> wrote: > I tested with "/usr/bin/env && /usr/bin/sleep 30" to give myself a chance > to look at stdout before the scheduler removed itself. I noticed that the > problem is still in place; all environment variables have quotes and > escaped slashes. > > This isn't a problem with launching a similar job via marathon: > { > "id": "/chris/test", > "cmd": "/usr/bin/env && /usr/bin/sleep 30", > "cpus": 0.1, > "mem": 128, > "disk": 0, > "instances": 1, > "env": { > "MY_FILEPATH": "/home/john/temp_output/test.csv" > } > } > > stdout: > Registered executor on 10.0.2.56 > Starting task chris_test.4fa0ac52-3eed-11e6-99b5-06094a33b9e2 > sh -c '/usr/bin/env && /usr/bin/sleep 30' > Forked command at 7174 > ... > MY_FILEPATH=/home/john/temp_output/test.csv > ... > > So, I would assume it's something in mesos-exec. I haven't found the code > that does it, so I can't figure out how to subvert it. > > On Thu, Jun 30, 2016 at 1:52 PM haosdent <[email protected]> wrote: > >> May you show the content of /home/john/mesos/error_msg.py I would like >> to try it in my machine instead of executing `env` command. >> >> On Fri, Jul 1, 2016 at 1:40 AM, June Taylor <[email protected]> wrote: >> >>> Both of these suggestions are still resulting in this content being seen >>> by the running application: \\/home\\/john\\/temp_output\\/test.csv" >>> >>> Thank you for your continued suggestions. Please try them out yourself >>> and let me know if you can get a non-escaped path to print out successfully >>> in your executing program, and let me know what you use to do that. >>> >>> >>> Thanks, >>> June Taylor >>> System Administrator, Minnesota Population Center >>> University of Minnesota >>> >>> On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker <[email protected]> >>> wrote: >>> >>>> +1 >>>> >>>> On Thu, Jun 30, 2016 at 1:15 PM haosdent <[email protected]> wrote: >>>> >>>>> Actually you could write your json into a file and pass the file path >>>>> into the flag. For example >>>>> >>>>> ``` >>>>> $ cat /tmp/a.json >>>>> {"MY_FILEPATH": "/home/john/temp_output/test.csv"} >>>>> ``` >>>>> >>>>> And use `file:///tmp/a.json` as the value of env flag. >>>>> >>>>> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers <[email protected]> >>>>> wrote: >>>>> >>>>>> Heh, fair enough. I know for the master/agent daemons you can put >>>>>> the config values into files, which makes this pretty clean (though >>>>>> that probably depends on the OS/init-launcher-daemon you are using). But >>>>>> yeah, when using the CLI you have to be pretty careful. >>>>>> >>>>>> - Erik >>>>>> >>>>>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> On a side note.... requiring people to put JSON on the command line >>>>>>> is a sadistic thing to do. >>>>>>> >>>>>>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> +1 I would wrap every string in quotes... otherwise your shell >>>>>>>> doesn't know what you mean. i.e., >>>>>>>> >>>>>>>> How is the shell supposed to know that you want this all to be 1 >>>>>>>> string value for the --command parameter? >>>>>>>> >>>>>>>> --command=/home/john/anaconda3/bin/python >>>>>>>> /home/john/mesos/error_msg.py read >>>>>>>> >>>>>>>> Similarly how is the shell to know that you *don't* want to end >>>>>>>> this command and start a new one when you have a bare semicolon? >>>>>>>> >>>>>>>> --resources=cpus:1;mem:128 >>>>>>>> >>>>>>>> - Erik >>>>>>>> >>>>>>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I use bash as well. How about >>>>>>>>> ``` >>>>>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program >>>>>>>>> --command="/home/john/anaconda3/bin/python >>>>>>>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128" >>>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}' >>>>>>>>> ``` >>>>>>>>> >>>>>>>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> hasodent, >>>>>>>>>> >>>>>>>>>> We've tried your suggestion but it seems like that's not working >>>>>>>>>> out: >>>>>>>>>> >>>>>>>>>> mesos-execute --master=cluster.example.com:5050 >>>>>>>>>> --name=test-program --command=/home/john/anaconda3/bin/python >>>>>>>>>> /home/john/mesos/error_msg.py read --resources=cpus:1;mem:128 >>>>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}' >>>>>>>>>> >>>>>>>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load >>>>>>>>>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax >>>>>>>>>> error >>>>>>>>>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}' >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We're using the Bash shell. Is anyone else passing file-paths >>>>>>>>>> within their environment variable argument? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> June Taylor >>>>>>>>>> System Administrator, Minnesota Population Center >>>>>>>>>> University of Minnesota >>>>>>>>>> >>>>>>>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Yes, need add quotes for --env='{"key": "value"}' flag. >>>>>>>>>>> >>>>>>>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> It's your shell: >>>>>>>>>>>> >>>>>>>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"} >>>>>>>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv} >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> hasodent, >>>>>>>>>>>>> >>>>>>>>>>>>> I admit I don't see much of a difference in your version. Are >>>>>>>>>>>>> you saying that Mesos doesn't escape the values, and that our >>>>>>>>>>>>> program may >>>>>>>>>>>>> be doing that on the executor end? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> June Taylor >>>>>>>>>>>>> System Administrator, Minnesota Population Center >>>>>>>>>>>>> University of Minnesota >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, @Taylor I use >>>>>>>>>>>>>> >>>>>>>>>>>>>> ``` >>>>>>>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH": >>>>>>>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" >>>>>>>>>>>>>> --command=env >>>>>>>>>>>>>> --name="test" >>>>>>>>>>>>>> ``` >>>>>>>>>>>>>> >>>>>>>>>>>>>> to test in my env. The output looks correct in my side >>>>>>>>>>>>>> >>>>>>>>>>>>>> ``` >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv >>>>>>>>>>>>>> >>>>>>>>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5 >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> ``` >>>>>>>>>>>>>> >>>>>>>>>>>>>> Not sure if your problem caused by your program >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Greetings, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We are using mesos-execute and passing an argument that >>>>>>>>>>>>>>> contains a Unix filepath, as follows: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *mesos-execute --master=cluster.example.com:5050 >>>>>>>>>>>>>>> <http://cluster.example.com:5050> --name=test-program >>>>>>>>>>>>>>> --command=/home/john/anaconda3/bin/python >>>>>>>>>>>>>>> /home/john/mesos/error_msg.py >>>>>>>>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH": >>>>>>>>>>>>>>> "/home/john/temp_output/test.csv"}* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The slashes in the file path are then apparently escaped by >>>>>>>>>>>>>>> mesos-execute and therefore fail to be useful once picked up by >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> executor: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory: >>>>>>>>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to >>>>>>>>>>>>>>> shutdown* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Are other people experiencing this? What is the method to >>>>>>>>>>>>>>> avoid it? >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> June Taylor >>>>>>>>>>>>>>> System Administrator, Minnesota Population Center >>>>>>>>>>>>>>> University of Minnesota >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Haosdent Huang >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Best Regards, >>>>>>>>>>> Haosdent Huang >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Best Regards, >>>>>>>>> Haosdent Huang >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards, >>>>> Haosdent Huang >>>>> >>>> >>> >> >> >> -- >> Best Regards, >> Haosdent Huang >> >
