Re: Retrieving Allocated GPU Resource ID for Task

[Kevin Klues]

What is the GPU ID you are referring to?

The UUID of the GPU? The short ID listed by `nvidia-smi` when listing GPUs?
The minor number associated with the underlying /dev device (which may be
different than the number appearing at the end of /dev/nvidia*). Or do you
just care about the number on /dev/nvidia* so that you can detect which
device you actually have access to on the file system?

Either way, there is no builtin support in mesos for getting any of these
values. However, you could easily run some script as a "pre-command" to get
at any of thees numbers.

For the UUID and short ID from `nvidia-smi`:
$ nvidia-smi -L

For the minor numbers:
$ nvidia-smi -q | grep Minor
Minor Number : 0
Minor Number : 1
Minor Number : 2
Minor Number : 3

For the actual /dev devices you have access to:
Loop through and call `touch` on each /dev/nvidia* device and see which
one's don't give you an error.

On Thu, Jun 30, 2016 at 2:47 PM Royce Cheng-Yue  wrote:

> Hi everyone,
>
> So far, I'm able to run a Mesos cluster with GPU resource allocation and
> can issue commands using mesos-execute; however, the commands I am planning
> to run require the allocated GPU resource IDs. Specifically, before our
> command executes, we need to set an environment variable which specifies
> the GPU ID to run the command on. Is there a way to retrieve the GPU ID
> after allocation and use the ID in our command before task execution?
>
> Thanks,
> Royce
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
>

Re: Using Mesos?

[Jie Yu]

John, merged.


On Thu, Jun 30, 2016 at 4:23 PM, John Crawford  wrote:

> I made an addition to this list: https://github.com/apache/mesos/pull/121
>
> On Thu, Jun 30, 2016 at 5:17 PM, Jay Taylor  wrote:
>
>> Ah, gotcha.  Thanks Jie and Kevin for the info!
>>
>> On Thu, Jun 30, 2016 at 2:14 PM, Kevin Klues  wrote:
>>
>>> It was committed.  We just do rebases instead of merges, so sometimes
>>> github gets confused as to whether it was actually merged or not. If you
>>> click on the SHA in the description next to the notification it was closed
>>> you can see that it was pushed back to master on apache/mesos.
>>>
>>> On Thu, Jun 30, 2016 at 2:11 PM Jay Taylor  wrote:
>>>
 I just tried this but it appears my PR was closed without comment.

 https://github.com/apache/mesos/pull/119

 What am I missing here? :)

 On Thu, Jun 30, 2016 at 1:45 PM, Benjamin Mahler 
 wrote:

> Just a reminder. If you're using Mesos and want to be featured in our
> list of users, send a PR to get your organization added:
>
> https://github.com/apache/mesos/blob/master/docs/powered-by-mesos.md
>
> If you've built a framework, and would like it featured in our list of
> frameworks, send a PR to get your framework listed:
>
> https://github.com/apache/mesos/blob/master/docs/frameworks.md
>
> Thanks!
>
> Ben
>


>>
>
>
> --
> John Crawford
> Co-founder & CTO
>
> 733 Foster Street  l  Suite 500
> Durham, NC 27701
>
> j...@ndustrial.io 
> mobile l 704.437.9000
>

Re: Using Mesos?

[John Crawford]

I made an addition to this list: https://github.com/apache/mesos/pull/121

On Thu, Jun 30, 2016 at 5:17 PM, Jay Taylor  wrote:

> Ah, gotcha.  Thanks Jie and Kevin for the info!
>
> On Thu, Jun 30, 2016 at 2:14 PM, Kevin Klues  wrote:
>
>> It was committed.  We just do rebases instead of merges, so sometimes
>> github gets confused as to whether it was actually merged or not. If you
>> click on the SHA in the description next to the notification it was closed
>> you can see that it was pushed back to master on apache/mesos.
>>
>> On Thu, Jun 30, 2016 at 2:11 PM Jay Taylor  wrote:
>>
>>> I just tried this but it appears my PR was closed without comment.
>>>
>>> https://github.com/apache/mesos/pull/119
>>>
>>> What am I missing here? :)
>>>
>>> On Thu, Jun 30, 2016 at 1:45 PM, Benjamin Mahler 
>>> wrote:
>>>
 Just a reminder. If you're using Mesos and want to be featured in our
 list of users, send a PR to get your organization added:

 https://github.com/apache/mesos/blob/master/docs/powered-by-mesos.md

 If you've built a framework, and would like it featured in our list of
 frameworks, send a PR to get your framework listed:

 https://github.com/apache/mesos/blob/master/docs/frameworks.md

 Thanks!

 Ben

>>>
>>>
>


-- 
John Crawford
Co-founder & CTO

733 Foster Street  l  Suite 500
Durham, NC 27701

j...@ndustrial.io 
mobile l 704.437.9000

Retrieving Allocated GPU Resource ID for Task

[Royce Cheng-Yue]

Hi everyone,

So far, I'm able to run a Mesos cluster with GPU resource allocation and
can issue commands using mesos-execute; however, the commands I am planning
to run require the allocated GPU resource IDs. Specifically, before our
command executes, we need to set an environment variable which specifies
the GPU ID to run the command on. Is there a way to retrieve the GPU ID
after allocation and use the ID in our command before task execution?

Thanks,
Royce

-- 
 

The information in this email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this email 
by anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be 
taken in reliance on it, is prohibited and may be unlawful.

Re: Using Mesos?

[Jay Taylor]

Ah, gotcha.  Thanks Jie and Kevin for the info!

On Thu, Jun 30, 2016 at 2:14 PM, Kevin Klues  wrote:

> It was committed.  We just do rebases instead of merges, so sometimes
> github gets confused as to whether it was actually merged or not. If you
> click on the SHA in the description next to the notification it was closed
> you can see that it was pushed back to master on apache/mesos.
>
> On Thu, Jun 30, 2016 at 2:11 PM Jay Taylor  wrote:
>
>> I just tried this but it appears my PR was closed without comment.
>>
>> https://github.com/apache/mesos/pull/119
>>
>> What am I missing here? :)
>>
>> On Thu, Jun 30, 2016 at 1:45 PM, Benjamin Mahler 
>> wrote:
>>
>>> Just a reminder. If you're using Mesos and want to be featured in our
>>> list of users, send a PR to get your organization added:
>>>
>>> https://github.com/apache/mesos/blob/master/docs/powered-by-mesos.md
>>>
>>> If you've built a framework, and would like it featured in our list of
>>> frameworks, send a PR to get your framework listed:
>>>
>>> https://github.com/apache/mesos/blob/master/docs/frameworks.md
>>>
>>> Thanks!
>>>
>>> Ben
>>>
>>
>>

Re: Using Mesos?

[Kevin Klues]

It was committed.  We just do rebases instead of merges, so sometimes
github gets confused as to whether it was actually merged or not. If you
click on the SHA in the description next to the notification it was closed
you can see that it was pushed back to master on apache/mesos.

On Thu, Jun 30, 2016 at 2:11 PM Jay Taylor  wrote:

> I just tried this but it appears my PR was closed without comment.
>
> https://github.com/apache/mesos/pull/119
>
> What am I missing here? :)
>
> On Thu, Jun 30, 2016 at 1:45 PM, Benjamin Mahler 
> wrote:
>
>> Just a reminder. If you're using Mesos and want to be featured in our
>> list of users, send a PR to get your organization added:
>>
>> https://github.com/apache/mesos/blob/master/docs/powered-by-mesos.md
>>
>> If you've built a framework, and would like it featured in our list of
>> frameworks, send a PR to get your framework listed:
>>
>> https://github.com/apache/mesos/blob/master/docs/frameworks.md
>>
>> Thanks!
>>
>> Ben
>>
>
>

Re: Using Mesos?

[John Sirois]

Granted its not effusive, but follow the asfgit comment sha link to here:
https://github.com/apache/mesos/commit/377ece0c206d958c7384f7d59706aa0320812da0
You'll see the commit on master.

On Thu, Jun 30, 2016 at 3:11 PM, Jay Taylor  wrote:

> I just tried this but it appears my PR was closed without comment.
>
> https://github.com/apache/mesos/pull/119
>
> What am I missing here? :)
>
> On Thu, Jun 30, 2016 at 1:45 PM, Benjamin Mahler 
> wrote:
>
>> Just a reminder. If you're using Mesos and want to be featured in our
>> list of users, send a PR to get your organization added:
>>
>> https://github.com/apache/mesos/blob/master/docs/powered-by-mesos.md
>>
>> If you've built a framework, and would like it featured in our list of
>> frameworks, send a PR to get your framework listed:
>>
>> https://github.com/apache/mesos/blob/master/docs/frameworks.md
>>
>> Thanks!
>>
>> Ben
>>
>
>

Re: Using Mesos?

[Jie Yu]

Jay, it was merged.

- Jie

On Thu, Jun 30, 2016 at 2:11 PM, Jay Taylor  wrote:

> I just tried this but it appears my PR was closed without comment.
>
> https://github.com/apache/mesos/pull/119
>
> What am I missing here? :)
>
> On Thu, Jun 30, 2016 at 1:45 PM, Benjamin Mahler 
> wrote:
>
>> Just a reminder. If you're using Mesos and want to be featured in our
>> list of users, send a PR to get your organization added:
>>
>> https://github.com/apache/mesos/blob/master/docs/powered-by-mesos.md
>>
>> If you've built a framework, and would like it featured in our list of
>> frameworks, send a PR to get your framework listed:
>>
>> https://github.com/apache/mesos/blob/master/docs/frameworks.md
>>
>> Thanks!
>>
>> Ben
>>
>
>

Re: Using Mesos?

[Jay Taylor]

I just tried this but it appears my PR was closed without comment.

https://github.com/apache/mesos/pull/119

What am I missing here? :)

On Thu, Jun 30, 2016 at 1:45 PM, Benjamin Mahler  wrote:

> Just a reminder. If you're using Mesos and want to be featured in our list
> of users, send a PR to get your organization added:
>
> https://github.com/apache/mesos/blob/master/docs/powered-by-mesos.md
>
> If you've built a framework, and would like it featured in our list of
> frameworks, send a PR to get your framework listed:
>
> https://github.com/apache/mesos/blob/master/docs/frameworks.md
>
> Thanks!
>
> Ben
>

Using Mesos?

[Benjamin Mahler]

Just a reminder. If you're using Mesos and want to be featured in our list
of users, send a PR to get your organization added:

https://github.com/apache/mesos/blob/master/docs/powered-by-mesos.md

If you've built a framework, and would like it featured in our list of
frameworks, send a PR to get your framework listed:

https://github.com/apache/mesos/blob/master/docs/frameworks.md

Thanks!

Ben

Re: Escaped characters in the 'env' argument passed to mesos-execute

[June Taylor]

Chris,

Thanks for the info - we do suspect it's abount mesos-execute, yes.


Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota

On Thu, Jun 30, 2016 at 1:10 PM, Chris Baker  wrote:

> I tested with "/usr/bin/env && /usr/bin/sleep 30" to give myself a chance
> to look at stdout before the scheduler removed itself. I noticed that the
> problem is still in place; all environment variables have quotes and
> escaped slashes.
>
> This isn't a problem with launching a similar job via marathon:
> {
>   "id": "/chris/test",
>   "cmd": "/usr/bin/env && /usr/bin/sleep 30",
>   "cpus": 0.1,
>   "mem": 128,
>   "disk": 0,
>   "instances": 1,
>   "env": {
> "MY_FILEPATH": "/home/john/temp_output/test.csv"
>   }
> }
>
> stdout:
> Registered executor on 10.0.2.56
> Starting task chris_test.4fa0ac52-3eed-11e6-99b5-06094a33b9e2
> sh -c '/usr/bin/env && /usr/bin/sleep 30'
> Forked command at 7174
> ...
> MY_FILEPATH=/home/john/temp_output/test.csv
> ...
>
> So, I would assume it's something in mesos-exec. I haven't found the code
> that does it, so I can't figure out how to subvert it.
>
> On Thu, Jun 30, 2016 at 1:52 PM haosdent  wrote:
>
>> May you show the content of  /home/john/mesos/error_msg.py  I would like
>> to try it in my machine instead of executing `env` command.
>>
>> On Fri, Jul 1, 2016 at 1:40 AM, June Taylor  wrote:
>>
>>> Both of these suggestions are still resulting in this content being seen
>>> by the running application: \\/home\\/john\\/temp_output\\/test.csv"
>>>
>>> Thank you for your continued suggestions. Please try them out yourself
>>> and let me know if you can get a non-escaped path to print out successfully
>>> in your executing program, and let me know what you use to do that.
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker 
>>> wrote:
>>>
 +1

 On Thu, Jun 30, 2016 at 1:15 PM haosdent  wrote:

> Actually you could write your json into a file and pass the file path
> into the flag. For example
>
> ```
> $ cat /tmp/a.json
> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
> ```
>
> And use `file:///tmp/a.json` as the value of env flag.
>
> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers 
> wrote:
>
>> Heh, fair enough.  I know for the master/agent daemons you can put
>> the config values into files, which makes this pretty clean (though
>> that probably depends on the OS/init-launcher-daemon you are using).  But
>> yeah, when using the CLI you have to be pretty careful.
>>
>> - Erik
>>
>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker 
>> wrote:
>>
>>> On a side note requiring people to put JSON on the command line
>>> is a sadistic thing to do.
>>>
>>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers <
>>> eweath...@groupon.com> wrote:
>>>
 +1 I would wrap every string in quotes...  otherwise your shell
 doesn't know what you mean.  i.e.,

 How is the shell supposed to know that you want this all to be 1
 string value for the --command parameter?

 --command=/home/john/anaconda3/bin/python
 /home/john/mesos/error_msg.py read

 Similarly how is the shell to know that you *don't* want to end
 this command and start a new one when you have a bare semicolon?

 --resources=cpus:1;mem:128

 - Erik

 On Thu, Jun 30, 2016 at 9:08 AM, haosdent 
 wrote:

> I use bash as well. How about
> ```
> mesos-execute --master=cluster.example.com:5050 --name=test-program
> --command="/home/john/anaconda3/bin/python
> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
> ```
>
> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:
>
>> hasodent,
>>
>> We've tried your suggestion but it seems like that's not working
>> out:
>>
>> mesos-execute --master=cluster.example.com:5050
>> --name=test-program --command=/home/john/anaconda3/bin/python
>> /home/john/mesos/error_msg.py read --resources=cpus:1;mem:128
>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>
>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax 
>> error
>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>
>>
>> We're 

Re: 1.0.0 RC2

[Vinod Kone]

Update: We still have about 6 blockers for the RC2 cut :( Good news is that
all of them are either reviewable or in progress :). I'll cut RC2 whenever
they land, whether it's tomorrow or coming tuesday.

Dashboard to track progress:
https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12328715

On Tue, Jun 21, 2016 at 11:48 AM, Vinod Kone  wrote:

> There are still 8 outstanding issues, including 1 blocker. We are waiting
> for these to land for RC2.
>
>
> On Fri, Jun 17, 2016 at 5:11 PM, Vinod Kone  wrote:
>
>> We still have 12 issues, including 1 blocker, targeted for 1.0.
>>
>> Dashboard: https://issues.apache.org/jira/secure/Dashboard.jspa
>>
>> So I'll wait until *monday morning PST *to cut RC2, for the blocker to
>> get resolved and any other targeted issues to land.
>>
>> Also note that with RC2 we will create a 1.0.x branch and update the
>> version on trunk to 1.1.0. Any further fixes for RC2 will be cherry picked
>> on to that branch.
>>
>>
>> On Wed, Jun 15, 2016 at 4:09 PM, Vinod Kone  wrote:
>>
>>> There are still 17 un-resolved issues targeted for 1.0. We have only
>>> couple more days left for the RC cut. Whoever is driving & shepherding
>>> these please make sure to land them.
>>>
>>>
>>>
>>> On Mon, Jun 13, 2016 at 1:58 PM, Vinod Kone 
>>> wrote:
>>>
 Hi folks,

 I'm planning to cut 1.0 RC2 later this week (likely friday). So please
 make sure to get any patches targeted for 1.0 (esp. blockers) upstreamed.

 The dashboard for the release is here:
 https://issues.apache.org/jira/issues/?filter=12335793

 Thanks,
 Vinod

>>>
>>>
>>
>

Re: Escaped characters in the 'env' argument passed to mesos-execute

[Chris Baker]

I tested with "/usr/bin/env && /usr/bin/sleep 30" to give myself a chance
to look at stdout before the scheduler removed itself. I noticed that the
problem is still in place; all environment variables have quotes and
escaped slashes.

This isn't a problem with launching a similar job via marathon:
{
  "id": "/chris/test",
  "cmd": "/usr/bin/env && /usr/bin/sleep 30",
  "cpus": 0.1,
  "mem": 128,
  "disk": 0,
  "instances": 1,
  "env": {
"MY_FILEPATH": "/home/john/temp_output/test.csv"
  }
}

stdout:
Registered executor on 10.0.2.56
Starting task chris_test.4fa0ac52-3eed-11e6-99b5-06094a33b9e2
sh -c '/usr/bin/env && /usr/bin/sleep 30'
Forked command at 7174
...
MY_FILEPATH=/home/john/temp_output/test.csv
...

So, I would assume it's something in mesos-exec. I haven't found the code
that does it, so I can't figure out how to subvert it.

On Thu, Jun 30, 2016 at 1:52 PM haosdent  wrote:

> May you show the content of  /home/john/mesos/error_msg.py  I would like
> to try it in my machine instead of executing `env` command.
>
> On Fri, Jul 1, 2016 at 1:40 AM, June Taylor  wrote:
>
>> Both of these suggestions are still resulting in this content being seen
>> by the running application: \\/home\\/john\\/temp_output\\/test.csv"
>>
>> Thank you for your continued suggestions. Please try them out yourself
>> and let me know if you can get a non-escaped path to print out successfully
>> in your executing program, and let me know what you use to do that.
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker 
>> wrote:
>>
>>> +1
>>>
>>> On Thu, Jun 30, 2016 at 1:15 PM haosdent  wrote:
>>>
 Actually you could write your json into a file and pass the file path
 into the flag. For example

 ```
 $ cat /tmp/a.json
 {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
 ```

 And use `file:///tmp/a.json` as the value of env flag.

 On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers 
 wrote:

> Heh, fair enough.  I know for the master/agent daemons you can put the
> config values into files, which makes this pretty clean (though
> that probably depends on the OS/init-launcher-daemon you are using).  But
> yeah, when using the CLI you have to be pretty careful.
>
> - Erik
>
> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker 
> wrote:
>
>> On a side note requiring people to put JSON on the command line
>> is a sadistic thing to do.
>>
>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers 
>> wrote:
>>
>>> +1 I would wrap every string in quotes...  otherwise your shell
>>> doesn't know what you mean.  i.e.,
>>>
>>> How is the shell supposed to know that you want this all to be 1
>>> string value for the --command parameter?
>>>
>>> --command=/home/john/anaconda3/bin/python
>>> /home/john/mesos/error_msg.py read
>>>
>>> Similarly how is the shell to know that you *don't* want to end this
>>> command and start a new one when you have a bare semicolon?
>>>
>>> --resources=cpus:1;mem:128
>>>
>>> - Erik
>>>
>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent 
>>> wrote:
>>>
 I use bash as well. How about
 ```
 mesos-execute --master=cluster.example.com:5050 --name=test-program
 --command="/home/john/anaconda3/bin/python
 /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
 --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
 ```

 On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:

> hasodent,
>
> We've tried your suggestion but it seems like that's not working
> out:
>
> mesos-execute --master=cluster.example.com:5050
> --name=test-program --command=/home/john/anaconda3/bin/python
> /home/john/mesos/error_msg.py read --resources=cpus:1;mem:128
> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>
> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax 
> error
> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>
>
> We're using the Bash shell. Is anyone else passing file-paths
> within their environment variable argument?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Jun 29, 2016 at 8:55 PM, haosdent 
> wrote:
>
>> Yes, need add quotes for 

Re: Escaped characters in the 'env' argument passed to mesos-execute

[haosdent]

May you show the content of  /home/john/mesos/error_msg.py  I would like to
try it in my machine instead of executing `env` command.

On Fri, Jul 1, 2016 at 1:40 AM, June Taylor  wrote:

> Both of these suggestions are still resulting in this content being seen
> by the running application: \\/home\\/john\\/temp_output\\/test.csv"
>
> Thank you for your continued suggestions. Please try them out yourself and
> let me know if you can get a non-escaped path to print out successfully in
> your executing program, and let me know what you use to do that.
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker 
> wrote:
>
>> +1
>>
>> On Thu, Jun 30, 2016 at 1:15 PM haosdent  wrote:
>>
>>> Actually you could write your json into a file and pass the file path
>>> into the flag. For example
>>>
>>> ```
>>> $ cat /tmp/a.json
>>> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>> ```
>>>
>>> And use `file:///tmp/a.json` as the value of env flag.
>>>
>>> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers 
>>> wrote:
>>>
 Heh, fair enough.  I know for the master/agent daemons you can put the
 config values into files, which makes this pretty clean (though
 that probably depends on the OS/init-launcher-daemon you are using).  But
 yeah, when using the CLI you have to be pretty careful.

 - Erik

 On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker 
 wrote:

> On a side note requiring people to put JSON on the command line is
> a sadistic thing to do.
>
> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers 
> wrote:
>
>> +1 I would wrap every string in quotes...  otherwise your shell
>> doesn't know what you mean.  i.e.,
>>
>> How is the shell supposed to know that you want this all to be 1
>> string value for the --command parameter?
>>
>> --command=/home/john/anaconda3/bin/python
>> /home/john/mesos/error_msg.py read
>>
>> Similarly how is the shell to know that you *don't* want to end this
>> command and start a new one when you have a bare semicolon?
>>
>> --resources=cpus:1;mem:128
>>
>> - Erik
>>
>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent  wrote:
>>
>>> I use bash as well. How about
>>> ```
>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>> --command="/home/john/anaconda3/bin/python
>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>> ```
>>>
>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:
>>>
 hasodent,

 We've tried your suggestion but it seems like that's not working
 out:

 mesos-execute --master=cluster.example.com:5050
 --name=test-program --command=/home/john/anaconda3/bin/python
 /home/john/mesos/error_msg.py read --resources=cpus:1;mem:128
 --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'

 john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
 value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax 
 error
 at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'


 We're using the Bash shell. Is anyone else passing file-paths
 within their environment variable argument?


 Thanks,
 June Taylor
 System Administrator, Minnesota Population Center
 University of Minnesota

 On Wed, Jun 29, 2016 at 8:55 PM, haosdent 
 wrote:

> Yes, need add quotes for --env='{"key": "value"}' flag.
>
> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker <
> ch...@galacticfog.com> wrote:
>
>> It's your shell:
>>
>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>
>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:
>>
>>> hasodent,
>>>
>>> I admit I don't see much of a difference in your version. Are
>>> you saying that Mesos doesn't escape the values, and that our 
>>> program may
>>> be doing that on the executor end?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent 
>>> wrote:
>>>
 Hi, @Taylor I use

 ```
 ./src/mesos-execute 

Re: Escaped characters in the 'env' argument passed to mesos-execute

[June Taylor]

Both of these suggestions are still resulting in this content being seen by
the running application: \\/home\\/john\\/temp_output\\/test.csv"

Thank you for your continued suggestions. Please try them out yourself and
let me know if you can get a non-escaped path to print out successfully in
your executing program, and let me know what you use to do that.


Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota

On Thu, Jun 30, 2016 at 12:28 PM, Chris Baker  wrote:

> +1
>
> On Thu, Jun 30, 2016 at 1:15 PM haosdent  wrote:
>
>> Actually you could write your json into a file and pass the file path
>> into the flag. For example
>>
>> ```
>> $ cat /tmp/a.json
>> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>> ```
>>
>> And use `file:///tmp/a.json` as the value of env flag.
>>
>> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers 
>> wrote:
>>
>>> Heh, fair enough.  I know for the master/agent daemons you can put the
>>> config values into files, which makes this pretty clean (though
>>> that probably depends on the OS/init-launcher-daemon you are using).  But
>>> yeah, when using the CLI you have to be pretty careful.
>>>
>>> - Erik
>>>
>>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker 
>>> wrote:
>>>
 On a side note requiring people to put JSON on the command line is
 a sadistic thing to do.

 On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers 
 wrote:

> +1 I would wrap every string in quotes...  otherwise your shell
> doesn't know what you mean.  i.e.,
>
> How is the shell supposed to know that you want this all to be 1
> string value for the --command parameter?
>
> --command=/home/john/anaconda3/bin/python
> /home/john/mesos/error_msg.py read
>
> Similarly how is the shell to know that you *don't* want to end this
> command and start a new one when you have a bare semicolon?
>
> --resources=cpus:1;mem:128
>
> - Erik
>
> On Thu, Jun 30, 2016 at 9:08 AM, haosdent  wrote:
>
>> I use bash as well. How about
>> ```
>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>> --command="/home/john/anaconda3/bin/python
>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>> ```
>>
>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:
>>
>>> hasodent,
>>>
>>> We've tried your suggestion but it seems like that's not working out:
>>>
>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>>> "/home/john/temp_output/test.csv"}'
>>>
>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
>>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax 
>>> error
>>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>
>>>
>>> We're using the Bash shell. Is anyone else passing file-paths within
>>> their environment variable argument?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent 
>>> wrote:
>>>
 Yes, need add quotes for --env='{"key": "value"}' flag.

 On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker  wrote:

> It's your shell:
>
> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
> {MY_FILEPATH: /home/john/temp_output/test.csv}
>
> On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:
>
>> hasodent,
>>
>> I admit I don't see much of a difference in your version. Are you
>> saying that Mesos doesn't escape the values, and that our program 
>> may be
>> doing that on the executor end?
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent 
>> wrote:
>>
>>> Hi, @Taylor I use
>>>
>>> ```
>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" 
>>> --command=env
>>> --name="test"
>>> ```
>>>
>>> to test in my env. The output looks correct in my side
>>>
>>> ```
>>> ...
>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>

Re: Escaped characters in the 'env' argument passed to mesos-execute

[Chris Baker]

+1

On Thu, Jun 30, 2016 at 1:15 PM haosdent  wrote:

> Actually you could write your json into a file and pass the file path into
> the flag. For example
>
> ```
> $ cat /tmp/a.json
> {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
> ```
>
> And use `file:///tmp/a.json` as the value of env flag.
>
> On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers 
> wrote:
>
>> Heh, fair enough.  I know for the master/agent daemons you can put the
>> config values into files, which makes this pretty clean (though
>> that probably depends on the OS/init-launcher-daemon you are using).  But
>> yeah, when using the CLI you have to be pretty careful.
>>
>> - Erik
>>
>> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker 
>> wrote:
>>
>>> On a side note requiring people to put JSON on the command line is a
>>> sadistic thing to do.
>>>
>>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers 
>>> wrote:
>>>
 +1 I would wrap every string in quotes...  otherwise your shell doesn't
 know what you mean.  i.e.,

 How is the shell supposed to know that you want this all to be 1 string
 value for the --command parameter?

 --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
 read

 Similarly how is the shell to know that you *don't* want to end this
 command and start a new one when you have a bare semicolon?

 --resources=cpus:1;mem:128

 - Erik

 On Thu, Jun 30, 2016 at 9:08 AM, haosdent  wrote:

> I use bash as well. How about
> ```
> mesos-execute --master=cluster.example.com:5050 --name=test-program
> --command="/home/john/anaconda3/bin/python
> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
> ```
>
> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:
>
>> hasodent,
>>
>> We've tried your suggestion but it seems like that's not working out:
>>
>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}'
>>
>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load
>> value ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax 
>> error
>> at line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>
>>
>> We're using the Bash shell. Is anyone else passing file-paths within
>> their environment variable argument?
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent  wrote:
>>
>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>
>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker 
>>> wrote:
>>>
 It's your shell:

 $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
 {MY_FILEPATH: /home/john/temp_output/test.csv}

 On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:

> hasodent,
>
> I admit I don't see much of a difference in your version. Are you
> saying that Mesos doesn't escape the values, and that our program may 
> be
> doing that on the executor end?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Jun 29, 2016 at 2:02 PM, haosdent 
> wrote:
>
>> Hi, @Taylor I use
>>
>> ```
>> ./src/mesos-execute --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" 
>> --command=env
>> --name="test"
>> ```
>>
>> to test in my env. The output looks correct in my side
>>
>> ```
>> ...
>> MY_FILEPATH=/home/john/temp_output/test.csv
>>
>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>> ...
>> ```
>>
>> Not sure if your problem caused by your program
>>
>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor 
>> wrote:
>>
>>> Greetings,
>>>
>>> We are using mesos-execute and passing an argument that contains
>>> a Unix filepath, as follows:
>>>
>>> *mesos-execute --master=cluster.example.com:5050
>>>  

Re: Escaped characters in the 'env' argument passed to mesos-execute

[haosdent]

Actually you could write your json into a file and pass the file path into
the flag. For example

```
$ cat /tmp/a.json
{"MY_FILEPATH": "/home/john/temp_output/test.csv"}
```

And use `file:///tmp/a.json` as the value of env flag.

On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers 
wrote:

> Heh, fair enough.  I know for the master/agent daemons you can put the
> config values into files, which makes this pretty clean (though
> that probably depends on the OS/init-launcher-daemon you are using).  But
> yeah, when using the CLI you have to be pretty careful.
>
> - Erik
>
> On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker 
> wrote:
>
>> On a side note requiring people to put JSON on the command line is a
>> sadistic thing to do.
>>
>> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers 
>> wrote:
>>
>>> +1 I would wrap every string in quotes...  otherwise your shell doesn't
>>> know what you mean.  i.e.,
>>>
>>> How is the shell supposed to know that you want this all to be 1 string
>>> value for the --command parameter?
>>>
>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>> read
>>>
>>> Similarly how is the shell to know that you *don't* want to end this
>>> command and start a new one when you have a bare semicolon?
>>>
>>> --resources=cpus:1;mem:128
>>>
>>> - Erik
>>>
>>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent  wrote:
>>>
 I use bash as well. How about
 ```
 mesos-execute --master=cluster.example.com:5050 --name=test-program
 --command="/home/john/anaconda3/bin/python
 /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
 --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
 ```

 On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:

> hasodent,
>
> We've tried your suggestion but it seems like that's not working out:
>
> mesos-execute --master=cluster.example.com:5050 --name=test-program
> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}'
>
> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>
>
> We're using the Bash shell. Is anyone else passing file-paths within
> their environment variable argument?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Jun 29, 2016 at 8:55 PM, haosdent  wrote:
>
>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>
>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker 
>> wrote:
>>
>>> It's your shell:
>>>
>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>
>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:
>>>
 hasodent,

 I admit I don't see much of a difference in your version. Are you
 saying that Mesos doesn't escape the values, and that our program may 
 be
 doing that on the executor end?


 Thanks,
 June Taylor
 System Administrator, Minnesota Population Center
 University of Minnesota

 On Wed, Jun 29, 2016 at 2:02 PM, haosdent 
 wrote:

> Hi, @Taylor I use
>
> ```
> ./src/mesos-execute --env='{"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}' --master="localhost:5050" 
> --command=env
> --name="test"
> ```
>
> to test in my env. The output looks correct in my side
>
> ```
> ...
> MY_FILEPATH=/home/john/temp_output/test.csv
>
> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
> ...
> ```
>
> Not sure if your problem caused by your program
>
> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor  wrote:
>
>> Greetings,
>>
>> We are using mesos-execute and passing an argument that contains
>> a Unix filepath, as follows:
>>
>> *mesos-execute --master=cluster.example.com:5050
>>  --name=test-program
>> --command=/home/john/anaconda3/bin/python 
>> /home/john/mesos/error_msg.py
>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}*
>>

Re: Mesos on hybrid AWS - Best practices?

[Sharma Podila]

I would second the suggestion of separate Mesos clusters for DC and AWS,
with a layer on top for picking one or either based on the job SLAs and
resource requirements.
The local storage on cloud instances are more ephemeral than I'd expect the
DC instances to be. So, persistent storage of job metadata needs
consideration. Using something like DynamoDB may work, however, depending
on the scale of your operations, you may have to plan for EC2 rate limiting
its API calls and/or paying for higher IOPS for data storage/access.
Treating the cloud instances as immutable infrastructure has additional
benefits. For example, we deploy new Mesos master ASG for version upgrades,
let them join the quorum, and then "tear down" the old master ASG. Same for
agents. Although, for agent migration our framework does coordinate
migration of jobs from old agent ASG to new one with some SLAs on not too
many instances of a service being down at a time. Sort of what the
maintenance primitives from Mesos aim to address.


On Thu, Jun 30, 2016 at 9:41 AM, Ken Sipe  wrote:

> I would suggest a cluster on AWS and a cluster on-prem.Then tooling on
> top to manage between the 2.
> It is unlikely that a failure of a task on-prem should have a scheduled
> replacement on AWS or vise versa.It is likely that you will end up
> creating constraints to statically partition the clusters anyway IMO.
> 2 Clusters eliminates most of your proposed questions.
>
> ken
>
> > On Jun 30, 2016, at 10:57 AM, Florian Pfeiffer  wrote:
> >
> > Hi,
> >
> > the last 2 years I managed a mesos cluster with bare-metal on-premise.
> Now at my new company, the situation is a little bit different, and I'm
> wondering if there are some kind of best practices:
> > The company is in the middle of a transition from on-premise to AWS. The
> old stuff is still running in the DC, the newer micro services are running
> within autoscales groups on AWS and other AWS services like DynamoDB,
> Kinesis and Lambda are also on the rise.
> >
> > So in my naive view of the world (where no problems occur. never!)
> I'm thinking that it would be great to span a hybrid mesos cluster over
> AWS to leverage the still available resources in the DC which gets more
> and more underutilized over the time.
> >
> > Now my naive world view slowly crumbles, and I realize that I'm missing
> the experience with AWS. Questions that are already popping up (beside all
> those Questions, where I currently don't know that I will have them...) are:
> > * Is Virtual Private Gateway to my VPC enough, or do I need to aim for a
> Direct Connect?
> > * Put everything into one Account, or use a Multi-Account strategy?
> (Mainly to prevent things running amok and drag stuff down while running
> into an account wide shared limit?)
> > * Will e.g. DynamoDb be "fast" enough if it's accessed from the
> Datacenter.
> >
> > I'll appreciate any feedback or lessons learned about that topic :)
> >
> > Thanks,
> > Florian
> >
>
>

Re: GPU channel on slack

[Kevin Klues]

https://reviews.apache.org/r/49456/

On Thu, Jun 30, 2016 at 8:55 AM Vinod Kone  wrote:

> Mind updating
> https://github.com/apache/mesos/blob/master/docs/working-groups.md with
> this info?
>
> On Thu, Jun 30, 2016 at 8:44 AM, Kevin Klues  wrote:
>
> > If you are interested in the ongoing GPU work on Mesos, please join the
> > #gpus channel at mesos.slack.com. The big announcements for the GPU work
> > will still happen on this mailing list, but the day to day discussions
> will
> > likely happen on the slack channel going forward.
> >
>

Re: Mesos on hybrid AWS - Best practices?

[Ken Sipe]

I would suggest a cluster on AWS and a cluster on-prem.Then tooling on top 
to manage between the 2.
It is unlikely that a failure of a task on-prem should have a scheduled 
replacement on AWS or vise versa.It is likely that you will end up creating 
constraints to statically partition the clusters anyway IMO. 
2 Clusters eliminates most of your proposed questions.

ken

> On Jun 30, 2016, at 10:57 AM, Florian Pfeiffer  wrote:
> 
> Hi,
> 
> the last 2 years I managed a mesos cluster with bare-metal on-premise. Now at 
> my new company, the situation is a little bit different, and I'm wondering if 
> there are some kind of best practices:
> The company is in the middle of a transition from on-premise to AWS. The old 
> stuff is still running in the DC, the newer micro services are running within 
> autoscales groups on AWS and other AWS services like DynamoDB, Kinesis and 
> Lambda are also on the rise. 
> 
> So in my naive view of the world (where no problems occur. never!) I'm 
> thinking that it would be great to span a hybrid mesos cluster over AWS to 
> leverage the still available resources in the DC which gets more and more 
> underutilized over the time. 
> 
> Now my naive world view slowly crumbles, and I realize that I'm missing the 
> experience with AWS. Questions that are already popping up (beside all those 
> Questions, where I currently don't know that I will have them...) are:
> * Is Virtual Private Gateway to my VPC enough, or do I need to aim for a 
> Direct Connect?
> * Put everything into one Account, or use a Multi-Account strategy? (Mainly 
> to prevent things running amok and drag stuff down while running into an 
> account wide shared limit?)
> * Will e.g. DynamoDb be "fast" enough if it's accessed from the Datacenter.
> 
> I'll appreciate any feedback or lessons learned about that topic :)
> 
> Thanks,
> Florian
> 

Re: Escaped characters in the 'env' argument passed to mesos-execute

[Erik Weathers]

Heh, fair enough.  I know for the master/agent daemons you can put the
config values into files, which makes this pretty clean (though
that probably depends on the OS/init-launcher-daemon you are using).  But
yeah, when using the CLI you have to be pretty careful.

- Erik

On Thu, Jun 30, 2016 at 9:31 AM, Chris Baker  wrote:

> On a side note requiring people to put JSON on the command line is a
> sadistic thing to do.
>
> On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers 
> wrote:
>
>> +1 I would wrap every string in quotes...  otherwise your shell doesn't
>> know what you mean.  i.e.,
>>
>> How is the shell supposed to know that you want this all to be 1 string
>> value for the --command parameter?
>>
>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read
>>
>> Similarly how is the shell to know that you *don't* want to end this
>> command and start a new one when you have a bare semicolon?
>>
>> --resources=cpus:1;mem:128
>>
>> - Erik
>>
>> On Thu, Jun 30, 2016 at 9:08 AM, haosdent  wrote:
>>
>>> I use bash as well. How about
>>> ```
>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>> --command="/home/john/anaconda3/bin/python
>>> /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128"
>>> --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>> ```
>>>
>>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:
>>>
 hasodent,

 We've tried your suggestion but it seems like that's not working out:

 mesos-execute --master=cluster.example.com:5050 --name=test-program
 --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
 read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
 "/home/john/temp_output/test.csv"}'

 john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
 ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
 line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'


 We're using the Bash shell. Is anyone else passing file-paths within
 their environment variable argument?


 Thanks,
 June Taylor
 System Administrator, Minnesota Population Center
 University of Minnesota

 On Wed, Jun 29, 2016 at 8:55 PM, haosdent  wrote:

> Yes, need add quotes for --env='{"key": "value"}' flag.
>
> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker 
> wrote:
>
>> It's your shell:
>>
>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>
>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:
>>
>>> hasodent,
>>>
>>> I admit I don't see much of a difference in your version. Are you
>>> saying that Mesos doesn't escape the values, and that our program may be
>>> doing that on the executor end?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent 
>>> wrote:
>>>
 Hi, @Taylor I use

 ```
 ./src/mesos-execute --env='{"MY_FILEPATH":
 "/home/john/temp_output/test.csv"}' --master="localhost:5050" 
 --command=env
 --name="test"
 ```

 to test in my env. The output looks correct in my side

 ```
 ...
 MY_FILEPATH=/home/john/temp_output/test.csv

 MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
 ...
 ```

 Not sure if your problem caused by your program

 On Thu, Jun 30, 2016 at 1:58 AM, June Taylor  wrote:

> Greetings,
>
> We are using mesos-execute and passing an argument that contains a
> Unix filepath, as follows:
>
> *mesos-execute --master=cluster.example.com:5050
>  --name=test-program
> --command=/home/john/anaconda3/bin/python 
> /home/john/mesos/error_msg.py
> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}*
>
> The slashes in the file path are then apparently escaped by
> mesos-execute and therefore fail to be useful once picked up by the
> executor:
>
> *FileNotFoundError: [Errno 2] No such file or directory:
> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>
> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
> shutdown*
>
> Are other people experiencing this? What is 

Re: Mesos on hybrid AWS - Best practices?

[Chris Baker]

I would also be concerned regarding the latency involved in having a Mesos
cluster span across the DC and the cloud provider. There have been some
discussions previously about tolerable latency for master/master and
master/slave; you might search the archives for this.

On Thu, Jun 30, 2016 at 11:57 AM Florian Pfeiffer  wrote:

> Hi,
>
> the last 2 years I managed a mesos cluster with bare-metal on-premise. Now
> at my new company, the situation is a little bit different, and I'm
> wondering if there are some kind of best practices:
> The company is in the middle of a transition from on-premise to AWS. The
> old stuff is still running in the DC, the newer micro services are running
> within autoscales groups on AWS and other AWS services like DynamoDB,
> Kinesis and Lambda are also on the rise.
>
> So in my naive view of the world (where no problems occur. never!) I'm
> thinking that it would be great to span a hybrid mesos cluster over AWS
> to leverage the still available resources in the DC which gets more and
> more underutilized over the time.
>
> Now my naive world view slowly crumbles, and I realize that I'm missing
> the experience with AWS. Questions that are already popping up (beside all
> those Questions, where I currently don't know that I will have them...) are:
> * Is Virtual Private Gateway to my VPC enough, or do I need to aim for a
> Direct Connect?
> * Put everything into one Account, or use a Multi-Account strategy?
> (Mainly to prevent things running amok and drag stuff down while running
> into an account wide shared limit?)
> * Will e.g. DynamoDb be "fast" enough if it's accessed from the Datacenter.
>
> I'll appreciate any feedback or lessons learned about that topic :)
>
> Thanks,
> Florian
>
>

Re: Escaped characters in the 'env' argument passed to mesos-execute

[Chris Baker]

On a side note requiring people to put JSON on the command line is a
sadistic thing to do.

On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers 
wrote:

> +1 I would wrap every string in quotes...  otherwise your shell doesn't
> know what you mean.  i.e.,
>
> How is the shell supposed to know that you want this all to be 1 string
> value for the --command parameter?
>
> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read
>
> Similarly how is the shell to know that you *don't* want to end this
> command and start a new one when you have a bare semicolon?
>
> --resources=cpus:1;mem:128
>
> - Erik
>
> On Thu, Jun 30, 2016 at 9:08 AM, haosdent  wrote:
>
>> I use bash as well. How about
>> ```
>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>> --command="/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read" --resources="cpus:1;mem:128" --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}'
>> ```
>>
>> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:
>>
>>> hasodent,
>>>
>>> We've tried your suggestion but it seems like that's not working out:
>>>
>>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>>> "/home/john/temp_output/test.csv"}'
>>>
>>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
>>> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
>>> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>>
>>>
>>> We're using the Bash shell. Is anyone else passing file-paths within
>>> their environment variable argument?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent  wrote:
>>>
 Yes, need add quotes for --env='{"key": "value"}' flag.

 On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker 
 wrote:

> It's your shell:
>
> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
> {MY_FILEPATH: /home/john/temp_output/test.csv}
>
> On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:
>
>> hasodent,
>>
>> I admit I don't see much of a difference in your version. Are you
>> saying that Mesos doesn't escape the values, and that our program may be
>> doing that on the executor end?
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent  wrote:
>>
>>> Hi, @Taylor I use
>>>
>>> ```
>>> ./src/mesos-execute --env='{"MY_FILEPATH":
>>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" 
>>> --command=env
>>> --name="test"
>>> ```
>>>
>>> to test in my env. The output looks correct in my side
>>>
>>> ```
>>> ...
>>> MY_FILEPATH=/home/john/temp_output/test.csv
>>>
>>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>>> ...
>>> ```
>>>
>>> Not sure if your problem caused by your program
>>>
>>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor  wrote:
>>>
 Greetings,

 We are using mesos-execute and passing an argument that contains a
 Unix filepath, as follows:

 *mesos-execute --master=cluster.example.com:5050
  --name=test-program
 --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
 read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
 "/home/john/temp_output/test.csv"}*

 The slashes in the file path are then apparently escaped by
 mesos-execute and therefore fail to be useful once picked up by the
 executor:

 *FileNotFoundError: [Errno 2] No such file or directory:
 '"\\/home\\/john\\/temp_output\\/test.csv"'*

 *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
 shutdown*

 Are other people experiencing this? What is the method to avoid it?
 Thanks,
 June Taylor
 System Administrator, Minnesota Population Center
 University of Minnesota

>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>>


 --
 Best Regards,
 Haosdent Huang

>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>

Re: Escaped characters in the 'env' argument passed to mesos-execute

[Erik Weathers]

+1 I would wrap every string in quotes...  otherwise your shell doesn't
know what you mean.  i.e.,

How is the shell supposed to know that you want this all to be 1 string
value for the --command parameter?

--command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py read

Similarly how is the shell to know that you *don't* want to end this
command and start a new one when you have a bare semicolon?

--resources=cpus:1;mem:128

- Erik

On Thu, Jun 30, 2016 at 9:08 AM, haosdent  wrote:

> I use bash as well. How about
> ```
> mesos-execute --master=cluster.example.com:5050 --name=test-program
> --command="/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read" --resources="cpus:1;mem:128" --env='{"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}'
> ```
>
> On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:
>
>> hasodent,
>>
>> We've tried your suggestion but it seems like that's not working out:
>>
>> mesos-execute --master=cluster.example.com:5050 --name=test-program
>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}'
>>
>> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
>> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
>> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>>
>>
>> We're using the Bash shell. Is anyone else passing file-paths within
>> their environment variable argument?
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Jun 29, 2016 at 8:55 PM, haosdent  wrote:
>>
>>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>>
>>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker 
>>> wrote:
>>>
 It's your shell:

 $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
 {MY_FILEPATH: /home/john/temp_output/test.csv}

 On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:

> hasodent,
>
> I admit I don't see much of a difference in your version. Are you
> saying that Mesos doesn't escape the values, and that our program may be
> doing that on the executor end?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Jun 29, 2016 at 2:02 PM, haosdent  wrote:
>
>> Hi, @Taylor I use
>>
>> ```
>> ./src/mesos-execute --env='{"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}' --master="localhost:5050" 
>> --command=env
>> --name="test"
>> ```
>>
>> to test in my env. The output looks correct in my side
>>
>> ```
>> ...
>> MY_FILEPATH=/home/john/temp_output/test.csv
>>
>> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
>> ...
>> ```
>>
>> Not sure if your problem caused by your program
>>
>> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor  wrote:
>>
>>> Greetings,
>>>
>>> We are using mesos-execute and passing an argument that contains a
>>> Unix filepath, as follows:
>>>
>>> *mesos-execute --master=cluster.example.com:5050
>>>  --name=test-program
>>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>>> "/home/john/temp_output/test.csv"}*
>>>
>>> The slashes in the file path are then apparently escaped by
>>> mesos-execute and therefore fail to be useful once picked up by the
>>> executor:
>>>
>>> *FileNotFoundError: [Errno 2] No such file or directory:
>>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>>
>>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to
>>> shutdown*
>>>
>>> Are other people experiencing this? What is the method to avoid it?
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>

Re: Escaped characters in the 'env' argument passed to mesos-execute

[haosdent]

I use bash as well. How about
```
mesos-execute --master=cluster.example.com:5050 --name=test-program
--command="/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
read" --resources="cpus:1;mem:128" --env='{"MY_FILEPATH":
"/home/john/temp_output/test.csv"}'
```

On Fri, Jul 1, 2016 at 12:04 AM, June Taylor  wrote:

> hasodent,
>
> We've tried your suggestion but it seems like that's not working out:
>
> mesos-execute --master=cluster.example.com:5050 --name=test-program
> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}'
>
> john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
> ''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
> line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'
>
>
> We're using the Bash shell. Is anyone else passing file-paths within their
> environment variable argument?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Jun 29, 2016 at 8:55 PM, haosdent  wrote:
>
>> Yes, need add quotes for --env='{"key": "value"}' flag.
>>
>> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker 
>> wrote:
>>
>>> It's your shell:
>>>
>>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>>
>>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:
>>>
 hasodent,

 I admit I don't see much of a difference in your version. Are you
 saying that Mesos doesn't escape the values, and that our program may be
 doing that on the executor end?


 Thanks,
 June Taylor
 System Administrator, Minnesota Population Center
 University of Minnesota

 On Wed, Jun 29, 2016 at 2:02 PM, haosdent  wrote:

> Hi, @Taylor I use
>
> ```
> ./src/mesos-execute --env='{"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}' --master="localhost:5050" 
> --command=env
> --name="test"
> ```
>
> to test in my env. The output looks correct in my side
>
> ```
> ...
> MY_FILEPATH=/home/john/temp_output/test.csv
>
> MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
> ...
> ```
>
> Not sure if your problem caused by your program
>
> On Thu, Jun 30, 2016 at 1:58 AM, June Taylor  wrote:
>
>> Greetings,
>>
>> We are using mesos-execute and passing an argument that contains a
>> Unix filepath, as follows:
>>
>> *mesos-execute --master=cluster.example.com:5050
>>  --name=test-program
>> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
>> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
>> "/home/john/temp_output/test.csv"}*
>>
>> The slashes in the file path are then apparently escaped by
>> mesos-execute and therefore fail to be useful once picked up by the
>> executor:
>>
>> *FileNotFoundError: [Errno 2] No such file or directory:
>> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>>
>> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to shutdown*
>>
>> Are other people experiencing this? What is the method to avoid it?
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>


>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>


-- 
Best Regards,
Haosdent Huang

Re: Escaped characters in the 'env' argument passed to mesos-execute

[June Taylor]

hasodent,

We've tried your suggestion but it seems like that's not working out:

mesos-execute --master=cluster.example.com:5050 --name=test-program
--command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":
"/home/john/temp_output/test.csv"}'

john@cluster:~/mesos$ Failed to load flag 'env': Failed to load value
''{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'': syntax error at
line 1 near: '{"MY_FILEPATH": "/home/john/temp_output/test.csv"}'


We're using the Bash shell. Is anyone else passing file-paths within their
environment variable argument?


Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota

On Wed, Jun 29, 2016 at 8:55 PM, haosdent  wrote:

> Yes, need add quotes for --env='{"key": "value"}' flag.
>
> On Thu, Jun 30, 2016 at 4:39 AM, Chris Baker 
> wrote:
>
>> It's your shell:
>>
>> $ echo {"MY_FILEPATH": "/home/john/temp_output/test.csv"}
>> {MY_FILEPATH: /home/john/temp_output/test.csv}
>>
>> On Wed, Jun 29, 2016 at 4:05 PM June Taylor  wrote:
>>
>>> hasodent,
>>>
>>> I admit I don't see much of a difference in your version. Are you saying
>>> that Mesos doesn't escape the values, and that our program may be doing
>>> that on the executor end?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Jun 29, 2016 at 2:02 PM, haosdent  wrote:
>>>
 Hi, @Taylor I use

 ```
 ./src/mesos-execute --env='{"MY_FILEPATH":
 "/home/john/temp_output/test.csv"}' --master="localhost:5050" --command=env
 --name="test"
 ```

 to test in my env. The output looks correct in my side

 ```
 ...
 MY_FILEPATH=/home/john/temp_output/test.csv

 MESOS_SANDBOX=/tmp/mesos/slaves/fe6123b9-8757-4015-be4b-fda901247fce-S0/frameworks/fe6123b9-8757-4015-be4b-fda901247fce-/executors/test/runs/fe818890-2a22-4b2b-aab2-816ed0b631d5
 ...
 ```

 Not sure if your problem caused by your program

 On Thu, Jun 30, 2016 at 1:58 AM, June Taylor  wrote:

> Greetings,
>
> We are using mesos-execute and passing an argument that contains a
> Unix filepath, as follows:
>
> *mesos-execute --master=cluster.example.com:5050
>  --name=test-program
> --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py
> read --resources=cpus:1;mem:128 --env={"MY_FILEPATH":
> "/home/john/temp_output/test.csv"}*
>
> The slashes in the file path are then apparently escaped by
> mesos-execute and therefore fail to be useful once picked up by the
> executor:
>
> *FileNotFoundError: [Errno 2] No such file or directory:
> '"\\/home\\/john\\/temp_output\\/test.csv"'*
>
> *I0629 10:13:13.814870 14025 exec.cpp:390] Executor asked to shutdown*
>
> Are other people experiencing this? What is the method to avoid it?
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>



 --
 Best Regards,
 Haosdent Huang

>>>
>>>
>
>
> --
> Best Regards,
> Haosdent Huang
>

Mesos on hybrid AWS - Best practices?

[Florian Pfeiffer]

Hi,

the last 2 years I managed a mesos cluster with bare-metal on-premise. Now
at my new company, the situation is a little bit different, and I'm
wondering if there are some kind of best practices:
The company is in the middle of a transition from on-premise to AWS. The
old stuff is still running in the DC, the newer micro services are running
within autoscales groups on AWS and other AWS services like DynamoDB,
Kinesis and Lambda are also on the rise.

So in my naive view of the world (where no problems occur. never!) I'm
thinking that it would be great to span a hybrid mesos cluster over AWS
to leverage the still available resources in the DC which gets more and
more underutilized over the time.

Now my naive world view slowly crumbles, and I realize that I'm missing the
experience with AWS. Questions that are already popping up (beside all
those Questions, where I currently don't know that I will have them...) are:
* Is Virtual Private Gateway to my VPC enough, or do I need to aim for a
Direct Connect?
* Put everything into one Account, or use a Multi-Account strategy? (Mainly
to prevent things running amok and drag stuff down while running into an
account wide shared limit?)
* Will e.g. DynamoDb be "fast" enough if it's accessed from the Datacenter.

I'll appreciate any feedback or lessons learned about that topic :)

Thanks,
Florian

Re: GPU channel on slack

[Vinod Kone]

Mind updating
https://github.com/apache/mesos/blob/master/docs/working-groups.md with
this info?

On Thu, Jun 30, 2016 at 8:44 AM, Kevin Klues  wrote:

> If you are interested in the ongoing GPU work on Mesos, please join the
> #gpus channel at mesos.slack.com. The big announcements for the GPU work
> will still happen on this mailing list, but the day to day discussions will
> likely happen on the slack channel going forward.
>

GPU channel on slack

[Kevin Klues]

If you are interested in the ongoing GPU work on Mesos, please join the
#gpus channel at mesos.slack.com. The big announcements for the GPU work
will still happen on this mailing list, but the day to day discussions will
likely happen on the slack channel going forward.