GPU channel on slack

If you are interested in the ongoing GPU work on Mesos, please join the #gpus channel at mesos.slack.com. The big announcements for the GPU work will still happen on this mailing list, but the day to day discussions will likely happen on the slack channel going forward.

Re: Escaped characters in the 'env' argument passed to mesos-execute

hasodent, We've tried your suggestion but it seems like that's not working out: mesos-execute --master=cluster.example.com:5050 --name=test-program --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py read --resources=cpus:1;mem:128 --env='{"MY_FILEPATH":

Re: GPU channel on slack

Mind updating https://github.com/apache/mesos/blob/master/docs/working-groups.md with this info? On Thu, Jun 30, 2016 at 8:44 AM, Kevin Klues wrote: > If you are interested in the ongoing GPU work on Mesos, please join the > #gpus channel at mesos.slack.com. The big

Re: Escaped characters in the 'env' argument passed to mesos-execute

I use bash as well. How about ``` mesos-execute --master=cluster.example.com:5050 --name=test-program --command="/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py read" --resources="cpus:1;mem:128" --env='{"MY_FILEPATH": "/home/john/temp_output/test.csv"}' ``` On Fri, Jul 1, 2016 at

Mesos on hybrid AWS - Best practices?

Hi, the last 2 years I managed a mesos cluster with bare-metal on-premise. Now at my new company, the situation is a little bit different, and I'm wondering if there are some kind of best practices: The company is in the middle of a transition from on-premise to AWS. The old stuff is still

Re: Escaped characters in the 'env' argument passed to mesos-execute

Both of these suggestions are still resulting in this content being seen by the running application: \\/home\\/john\\/temp_output\\/test.csv" Thank you for your continued suggestions. Please try them out yourself and let me know if you can get a non-escaped path to print out successfully in your

Re: Escaped characters in the 'env' argument passed to mesos-execute

On a side note requiring people to put JSON on the command line is a sadistic thing to do. On Thu, Jun 30, 2016 at 12:28 PM Erik Weathers wrote: > +1 I would wrap every string in quotes... otherwise your shell doesn't > know what you mean. i.e., > > How is the shell

Re: Escaped characters in the 'env' argument passed to mesos-execute

Heh, fair enough. I know for the master/agent daemons you can put the config values into files, which makes this pretty clean (though that probably depends on the OS/init-launcher-daemon you are using). But yeah, when using the CLI you have to be pretty careful. - Erik On Thu, Jun 30, 2016 at

Re: GPU channel on slack

https://reviews.apache.org/r/49456/ On Thu, Jun 30, 2016 at 8:55 AM Vinod Kone wrote: > Mind updating > https://github.com/apache/mesos/blob/master/docs/working-groups.md with > this info? > > On Thu, Jun 30, 2016 at 8:44 AM, Kevin Klues wrote: > > > If

Re: Escaped characters in the 'env' argument passed to mesos-execute

+1 On Thu, Jun 30, 2016 at 1:15 PM haosdent wrote: > Actually you could write your json into a file and pass the file path into > the flag. For example > > ``` > $ cat /tmp/a.json > {"MY_FILEPATH": "/home/john/temp_output/test.csv"} > ``` > > And use `file:///tmp/a.json` as

Re: Escaped characters in the 'env' argument passed to mesos-execute

+1 I would wrap every string in quotes... otherwise your shell doesn't know what you mean. i.e., How is the shell supposed to know that you want this all to be 1 string value for the --command parameter? --command=/home/john/anaconda3/bin/python /home/john/mesos/error_msg.py read Similarly

Re: Mesos on hybrid AWS - Best practices?

I would also be concerned regarding the latency involved in having a Mesos cluster span across the DC and the cloud provider. There have been some discussions previously about tolerable latency for master/master and master/slave; you might search the archives for this. On Thu, Jun 30, 2016 at

Re: Mesos on hybrid AWS - Best practices?

I would suggest a cluster on AWS and a cluster on-prem.Then tooling on top to manage between the 2. It is unlikely that a failure of a task on-prem should have a scheduled replacement on AWS or vise versa.It is likely that you will end up creating constraints to statically partition the

Re: Mesos on hybrid AWS - Best practices?

I would second the suggestion of separate Mesos clusters for DC and AWS, with a layer on top for picking one or either based on the job SLAs and resource requirements. The local storage on cloud instances are more ephemeral than I'd expect the DC instances to be. So, persistent storage of job

Re: Escaped characters in the 'env' argument passed to mesos-execute

Actually you could write your json into a file and pass the file path into the flag. For example ``` $ cat /tmp/a.json {"MY_FILEPATH": "/home/john/temp_output/test.csv"} ``` And use `file:///tmp/a.json` as the value of env flag. On Fri, Jul 1, 2016 at 12:40 AM, Erik Weathers

Re: Escaped characters in the 'env' argument passed to mesos-execute

May you show the content of /home/john/mesos/error_msg.py I would like to try it in my machine instead of executing `env` command. On Fri, Jul 1, 2016 at 1:40 AM, June Taylor wrote: > Both of these suggestions are still resulting in this content being seen > by the running

Re: Escaped characters in the 'env' argument passed to mesos-execute

I tested with "/usr/bin/env && /usr/bin/sleep 30" to give myself a chance to look at stdout before the scheduler removed itself. I noticed that the problem is still in place; all environment variables have quotes and escaped slashes. This isn't a problem with launching a similar job via marathon:

Re: Escaped characters in the 'env' argument passed to mesos-execute

Chris, Thanks for the info - we do suspect it's abount mesos-execute, yes. Thanks, June Taylor System Administrator, Minnesota Population Center University of Minnesota On Thu, Jun 30, 2016 at 1:10 PM, Chris Baker wrote: > I tested with "/usr/bin/env && /usr/bin/sleep

Re: 1.0.0 RC2

Update: We still have about 6 blockers for the RC2 cut :( Good news is that all of them are either reviewable or in progress :). I'll cut RC2 whenever they land, whether it's tomorrow or coming tuesday. Dashboard to track progress:

Using Mesos?

Just a reminder. If you're using Mesos and want to be featured in our list of users, send a PR to get your organization added: https://github.com/apache/mesos/blob/master/docs/powered-by-mesos.md If you've built a framework, and would like it featured in our list of frameworks, send a PR to get

Re: Using Mesos?

I just tried this but it appears my PR was closed without comment. https://github.com/apache/mesos/pull/119 What am I missing here? :) On Thu, Jun 30, 2016 at 1:45 PM, Benjamin Mahler wrote: > Just a reminder. If you're using Mesos and want to be featured in our list > of

Re: Using Mesos?

Jay, it was merged. - Jie On Thu, Jun 30, 2016 at 2:11 PM, Jay Taylor wrote: > I just tried this but it appears my PR was closed without comment. > > https://github.com/apache/mesos/pull/119 > > What am I missing here? :) > > On Thu, Jun 30, 2016 at 1:45 PM, Benjamin Mahler

Re: Using Mesos?

It was committed. We just do rebases instead of merges, so sometimes github gets confused as to whether it was actually merged or not. If you click on the SHA in the description next to the notification it was closed you can see that it was pushed back to master on apache/mesos. On Thu, Jun 30,

Re: Using Mesos?

Granted its not effusive, but follow the asfgit comment sha link to here: https://github.com/apache/mesos/commit/377ece0c206d958c7384f7d59706aa0320812da0 You'll see the commit on master. On Thu, Jun 30, 2016 at 3:11 PM, Jay Taylor wrote: > I just tried this but it appears my

Re: Using Mesos?

Ah, gotcha. Thanks Jie and Kevin for the info! On Thu, Jun 30, 2016 at 2:14 PM, Kevin Klues wrote: > It was committed. We just do rebases instead of merges, so sometimes > github gets confused as to whether it was actually merged or not. If you > click on the SHA in the

Re: Using Mesos?

I made an addition to this list: https://github.com/apache/mesos/pull/121 On Thu, Jun 30, 2016 at 5:17 PM, Jay Taylor wrote: > Ah, gotcha. Thanks Jie and Kevin for the info! > > On Thu, Jun 30, 2016 at 2:14 PM, Kevin Klues wrote: > >> It was committed.

Retrieving Allocated GPU Resource ID for Task

Hi everyone, So far, I'm able to run a Mesos cluster with GPU resource allocation and can issue commands using mesos-execute; however, the commands I am planning to run require the allocated GPU resource IDs. Specifically, before our command executes, we need to set an environment variable which

Re: Using Mesos?

John, merged. On Thu, Jun 30, 2016 at 4:23 PM, John Crawford wrote: > I made an addition to this list: https://github.com/apache/mesos/pull/121 > > On Thu, Jun 30, 2016 at 5:17 PM, Jay Taylor wrote: > >> Ah, gotcha. Thanks Jie and Kevin for the info! >>

Re: Retrieving Allocated GPU Resource ID for Task

What is the GPU ID you are referring to? The UUID of the GPU? The short ID listed by `nvidia-smi` when listing GPUs? The minor number associated with the underlying /dev device (which may be different than the number appearing at the end of /dev/nvidia*). Or do you just care about the number on