I tested with "/usr/bin/env && /usr/bin/sleep 30" to give myself a chance
to look at stdout before the scheduler removed itself. I noticed that the
problem is still in place; all environment variables have quotes and
escaped slashes.
This isn't a problem with launching a similar job via marathon:
{
"id": "/chris/test",
"cmd": "/usr/bin/env && /usr/bin/sleep 30",
"cpus": 0.1,
"mem": 128,
"disk": 0,
"instances": 1,
"env": {
"MY_FILEPATH": "/home/john/temp_output/test.csv"
}
}
stdout:
Registered executor on 10.0.2.56
Starting task chris_test.4fa0ac52-3eed-11e6-99b5-06094a33b9e2
sh -c '/usr/bin/env && /usr/bin/sleep 30'
Forked command at 7174
...
MY_FILEPATH=/home/john/temp_output/test.csv
...
So, I would assume it's something in mesos-exec. I haven't found the code
that does it, so I can't figure out how to subvert it.
On Thu, Jun 30, 2016 at 1:52 PM haosdent <haosd...@gmail.com> wrote:
> May you show the content of /home/john/mesos/error_msg.py I would like
> to try it in my machine instead of executing `env` command.
>
> On Fri, Jul 1, 2016 at 1:40 AM, June Taylor <j...@umn.edu> wrote:
>
>> Both of these suggestions are still resulting in this content being seen
>> by the running application: \\/home\\/john\\/temp_output\\/test.csv"
>>
>> Thank you for your continued suggestions. Please try them out yourself
>> and let me know if you can get a non-escaped path to print out successfully
>> in your executing program, and let me know what you use to do that.
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker <ch...@galacticfog.com>
>> wrote:
>>
>>> +1
>>>
>>> On Thu, Jun 30, 2016 at 1:15 PM haosdent <haosd...@gmail.com> wrote:
>>>
>>>> Actually you could write your json into a file and pass the file path
>>>> into the flag. For example
>>>>
>>>> ```
>>>> $ cat /tmp/a.json
>>>> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>> ```
>>>>
>>>> And use `file:///tmp/a.json` as the value of env flag.
>>>>
>>>> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers <eweath...@groupon.com>
>>>> wrote:
>>>>
>>>>> Heh, fair enough. I know for the master/agent daemons you can put the
>>>>> config values into files, which makes this pretty clean (though
>>>>> that probably depends on the OS/init-launcher-daemon you are using). But
>>>>> yeah, when using the CLI you have to be pretty careful.
>>>>>
>>>>> - Erik
>>>>>
>>>>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker <ch...@galacticfog.com>
>>>>> wrote:
>>>>>
>>>>>> On a side note.... requiring people to put JSON on the command line
>>>>>> is a sadistic thing to do.
>>>>>>
>>>>>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <eweath...@groupon.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 I would wrap every string in quotes... otherwise your shell
>>>>>>> doesn't know what you mean. i.e.,
>>>>>>>
>>>>>>> How is the shell supposed to know that you want this all to be 1
>>>>>>> string value for the --command parameter?
>>>>>>>
>>>>>>> --command=/home/john/anaconda3/bin/python
>>>>>>> /home/john/mesos/error_msg.py read
>>>>>>>
>>>>>>> Similarly how is the shell to know that you *don't* want to end this
>>>>>>> command and start a new one when you have a bare semicolon?
>>>>>>>
>>>>>>> --resources=cpus:1;mem:128
>>>>>>>
>>>>>>> - Erik
>>>>>>>
>>>>>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent <haosd...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I use bash as well. How about
>>>>>>>> ```
>>>>>>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>>>>>>> --command="/home/john/anaconda3/bin/python
>>>>>>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>> ```
>>>>>>>>
>>>>>>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor <j...@umn.edu> wrote:
>>>>>>>>
>>>>>>>>> hasodent,
>>>>>>>>>
>>>>>>>>> We've tried your suggestion but it seems like that's not working
>>>>>>>>> out:
>>>>>>>>>
>>>>>>>>> mesos-execute --master=cluster.example.com:5050
>>>>>>>>> --name=test-program --command=/home/john/anaconda3/bin/python
>>>>>>>>> /home/john/mesos/error_msg.py read --resources=cpus:1;mem:128
>>>>>>>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>>>
>>>>>>>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
>>>>>>>>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax
>>>>>>>>> error
>>>>>>>>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We're using the Bash shell. Is anyone else passing file-paths
>>>>>>>>> within their environment variable argument?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> June Taylor
>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>> University of Minnesota
>>>>>>>>>
>>>>>>>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent <haosd...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <
>>>>>>>>>> ch...@galacticfog.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> It's your shell:
>>>>>>>>>>>
>>>>>>>>>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>>>>>>>>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor <j...@umn.edu>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> hasodent,
>>>>>>>>>>>>
>>>>>>>>>>>> I admit I don't see much of a difference in your version. Are
>>>>>>>>>>>> you saying that Mesos doesn't escape the values, and that our
>>>>>>>>>>>> program may
>>>>>>>>>>>> be doing that on the executor end?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> June Taylor
>>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent <haosd...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, @Taylor I use
>>>>>>>>>>>>>
>>>>>>>>>>>>> ```
>>>>>>>>>>>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>>>>>>>>>>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050"
>>>>>>>>>>>>> --command=env
>>>>>>>>>>>>> --name="test"
>>>>>>>>>>>>> ```
>>>>>>>>>>>>>
>>>>>>>>>>>>> to test in my env. The output looks correct in my side
>>>>>>>>>>>>>
>>>>>>>>>>>>> ```
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>>>>>>>>>>>
>>>>>>>>>>>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-0000/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> ```
>>>>>>>>>>>>>
>>>>>>>>>>>>> Not sure if your problem caused by your program
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor <j...@umn.edu>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Greetings,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We are using mesos-execute and passing an argument that
>>>>>>>>>>>>>> contains a Unix filepath, as follows:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *mesos-execute --master=cluster.example.com:5050
>>>>>>>>>>>>>> <http://cluster.example.com:5050> --name=test-program
>>>>>>>>>>>>>> --command=/home/john/anaconda3/bin/python
>>>>>>>>>>>>>> /home/john/mesos/error_msg.py
>>>>>>>>>>>>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>>>>>>>>>>>>> "/home/john/temp_output/test.csv"}*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The slashes in the file path are then apparently escaped by
>>>>>>>>>>>>>> mesos-execute and therefore fail to be useful once picked up by
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> executor:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>>>>>>>>>>>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>>>>>>>>>>>>> shutdown*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Are other people experiencing this? What is the method to
>>>>>>>>>>>>>> avoid it?
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> June Taylor
>>>>>>>>>>>>>> System Administrator, Minnesota Population Center
>>>>>>>>>>>>>> University of Minnesota
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Haosdent Huang
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>> Haosdent Huang
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards,
>>>>>>>> Haosdent Huang
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Haosdent Huang
>>>>
>>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
