RE: Support for tasks groups aka pods in Mesos

2016-09-21 Thread Hubert.Asamer 
Thanks for the clarification – I’ll try to dig into this and use execute.cpp  
as a starting point for my problem…
Kind Regards,
Hubert

From: Guangya Liu [mailto:gyliu...@gmail.com]
Sent: Mittwoch, 21. September 2016 09:19
To: user@mesos.apache.org
Cc: dev; Vinod Kone
Subject: Re: Support for tasks groups aka pods in Mesos

The answer is No, the taskGroup can be treated as Pod in Kubernetes, it will be 
a set of containers co-located and co-managed on an agent that share some 
resources (e.g., network namespace, volumes). For your case, you may want to 
split your 50 tasks to small task groups.

Also the `execute` cli is just an example framework and it can only launch one 
task group for now, you may want to enhance it if you want to launch multiple 
task groups.

On Wed, Sep 21, 2016 at 1:41 PM, 
<hubert.asa...@dlr.de<mailto:hubert.asa...@dlr.de>> wrote:
Hi,
Very cool feature.
I’ve seen some recent changes 
(MESOS-6096<https://issues.apache.org/jira/browse/MESOS-6096>) in the 
mesos-execute cli supporting grouped tasks which spawns the task-group on an 
arbitrary or specified agent node. Tested this yesterday and it works great.  
But if the task group holds a lot of one-off tasks (let’s say 50 tasks each 4 
cores,…) how could one achieve that  tasks are evenly distributed over several 
agents, depending on free resources? Don’t know if this is referred to pods, 
but do you think mesos-execute will support this?

Currently I’m using chronos & cook for batch scheduling, but I’d  like to 
achieve that every batch/group of  tasks runs in its own temporary framework 
like it does with mesos-execute

Thanks in advance & kind regards,
Hubert

——
Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
German Aerospace Center


German Remote Sensing Data Center | International Ground Segment | 
Oberpfaffenhofen | 82234 Wessling | Germany

Hubert Asamer
phone  +49 (0) 8153 28-2894<tel:%2B49%20%280%29%208153%2028-2894> | fax +49 (0) 
8153 28-1443<tel:%2B49%20%280%29%208153%2028-1443> | 
hubert.asa...@dlr.de<mailto:hubert.asa...@dlr.de>
www.DLR.de/eoc<http://www.dlr.de/eoc>




From: Vinod Kone [mailto:vinodk...@apache.org<mailto:vinodk...@apache.org>]
Sent: Dienstag, 9. August 2016 02:54
To: dev; user
Subject: Support for tasks groups aka pods in Mesos

Hi folks,

One of the most requested features in Mesos has been first class support for 
managing pod like containers. We finally have some time to focus and shepherd 
this work.

The epic tracking this work is : 
https://issues.apache.org/jira/browse/MESOS-2449

Design doc: https://issues.apache.org/jira/browse/MESOS-2449

Your feedback on the design will be most welcome. Once we get agreement on the 
design, we can start breaking down the epic into tickets.

Thanks,
Vinod & Jie

Re: Support for tasks groups aka pods in Mesos

2016-09-21 Thread Guangya Liu 
The answer is No, the taskGroup can be treated as Pod in Kubernetes, it
will be a set of containers co-located and co-managed on an agent that
share some resources (e.g., network namespace, volumes). For your case, you
may want to split your 50 tasks to small task groups.

Also the `execute` cli is just an example framework and it can only launch
one task group for now, you may want to enhance it if you want to launch
multiple task groups.

On Wed, Sep 21, 2016 at 1:41 PM,  wrote:

> Hi,
>
> Very cool feature.
>
> I’ve seen some recent changes (MESOS-6096
> ) in the mesos-execute
> cli supporting grouped tasks which spawns the task-group on an arbitrary or
> specified agent node. Tested this yesterday and it works great.  But if the
> task group holds a lot of one-off tasks (let’s say 50 tasks each 4 cores,…)
> how could one achieve that  tasks are evenly distributed over several
> agents, depending on free resources? Don’t know if this is referred to
> pods, but do you think mesos-execute will support this?
>
>
>
> Currently I’m using chronos & cook for batch scheduling, but I’d  like to
> achieve that every batch/group of  tasks runs in its own temporary
> framework like it does with mesos-execute
>
>
>
> Thanks in advance & kind regards,
>
> Hubert
>
>
>
> ——
>
> *Deutsches Zentrum für Luft- und Raumfahrt* e.V. (DLR)
>
> German Aerospace Center
>
>
>
>
>
> German Remote Sensing Data Center | International Ground Segment |
> Oberpfaffenhofen | 82234 Wessling | Germany
>
>
>
> *Hubert Asamer*
>
> phone  +49 (0) 8153 28-2894 | fax +49 (0) 8153 28-1443 |
> hubert.asa...@dlr.de
>
> www.DLR.de/eoc 
>
>
>
>
>
>
>
>
>
> *From:* Vinod Kone [mailto:vinodk...@apache.org]
> *Sent:* Dienstag, 9. August 2016 02:54
> *To:* dev; user
> *Subject:* Support for tasks groups aka pods in Mesos
>
>
>
> Hi folks,
>
>
>
> One of the most requested features in Mesos has been first class support
> for managing pod like containers. We finally have some time to focus and
> shepherd this work.
>
>
>
> The epic tracking this work is : https://issues.apache.org/
> jira/browse/MESOS-2449
>
>
>
> Design doc: https://issues.apache.org/jira/browse/MESOS-2449
>
>
>
> Your feedback on the design will be most welcome. Once we get agreement on
> the design, we can start breaking down the epic into tickets.
>
>
>
> Thanks,
>
> Vinod & Jie
>

RE: Support for tasks groups aka pods in Mesos

2016-09-20 Thread Hubert.Asamer 
Hi,
Very cool feature.
I’ve seen some recent changes 
(MESOS-6096) in the 
mesos-execute cli supporting grouped tasks which spawns the task-group on an 
arbitrary or specified agent node. Tested this yesterday and it works great.  
But if the task group holds a lot of one-off tasks (let’s say 50 tasks each 4 
cores,…) how could one achieve that  tasks are evenly distributed over several 
agents, depending on free resources? Don’t know if this is referred to pods, 
but do you think mesos-execute will support this?

Currently I’m using chronos & cook for batch scheduling, but I’d  like to 
achieve that every batch/group of  tasks runs in its own temporary framework 
like it does with mesos-execute

Thanks in advance & kind regards,
Hubert

——
Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
German Aerospace Center


German Remote Sensing Data Center | International Ground Segment | 
Oberpfaffenhofen | 82234 Wessling | Germany

Hubert Asamer
phone  +49 (0) 8153 28-2894 | fax +49 (0) 8153 28-1443 | 
hubert.asa...@dlr.de
www.DLR.de/eoc




From: Vinod Kone [mailto:vinodk...@apache.org]
Sent: Dienstag, 9. August 2016 02:54
To: dev; user
Subject: Support for tasks groups aka pods in Mesos

Hi folks,

One of the most requested features in Mesos has been first class support for 
managing pod like containers. We finally have some time to focus and shepherd 
this work.

The epic tracking this work is : 
https://issues.apache.org/jira/browse/MESOS-2449

Design doc: https://issues.apache.org/jira/browse/MESOS-2449

Your feedback on the design will be most welcome. Once we get agreement on the 
design, we can start breaking down the epic into tickets.

Thanks,
Vinod & Jie

Re: Support for tasks groups aka pods in Mesos

2016-08-11 Thread Jie Yu 
Aaron,

an important feature of kubernetes for us is the ability to share mount and
> network namespaces between containers in the pod (so containers can share
> mounts etc)


yes, this will be supported in MVP. it's described in the design doc.

- Jie

On Thu, Aug 11, 2016 at 6:32 AM, Aaron Carey <aca...@ilm.com> wrote:

> Hi Vinod,
>
> I'm not sure where this should go in the design doc, but an important
> feature of kubernetes for us is the ability to share mount and network
> namespaces between containers in the pod (so containers can share mounts
> etc). This currently isn't very well supported in docker (I think
> Kubernetes uses its own containeriser to do this?).
>
> We'd be very keen to have this ability in Mesos too.
>
> Thanks,
> Aaron
>
>
> --
>
> Aaron Carey
> Production Engineer - Cloud Pipeline
> Industrial Light & Magic
> London
> 020 3751 9150
>
> --
> *From:* Vinod Kone [vinodk...@apache.org]
> *Sent:* 09 August 2016 02:42
> *To:* Vinod Kone
> *Cc:* dev; user
> *Subject:* Re: Support for tasks groups aka pods in Mesos
>
> Sorry, sent the wrong link earlier for design doc.
>
> Design doc: https://issues.apache.org/jira/browse/MESOS-6009
>>
>
> Direct link: https://docs.google.com/document/d/1FtcyQkDfGp-
> bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz
>

Re: Support for tasks groups aka pods in Mesos

2016-08-11 Thread tommy xiao 
+1, can't wait to use it.

2016-08-11 21:32 GMT+08:00 Aaron Carey <aca...@ilm.com>:

> Hi Vinod,
>
> I'm not sure where this should go in the design doc, but an important
> feature of kubernetes for us is the ability to share mount and network
> namespaces between containers in the pod (so containers can share mounts
> etc). This currently isn't very well supported in docker (I think
> Kubernetes uses its own containeriser to do this?).
>
> We'd be very keen to have this ability in Mesos too.
>
> Thanks,
> Aaron
>
>
> --
>
> Aaron Carey
> Production Engineer - Cloud Pipeline
> Industrial Light & Magic
> London
> 020 3751 9150
>
> --
> *From:* Vinod Kone [vinodk...@apache.org]
> *Sent:* 09 August 2016 02:42
> *To:* Vinod Kone
> *Cc:* dev; user
> *Subject:* Re: Support for tasks groups aka pods in Mesos
>
> Sorry, sent the wrong link earlier for design doc.
>
> Design doc: https://issues.apache.org/jira/browse/MESOS-6009
>>
>
> Direct link: https://docs.google.com/document/d/1FtcyQkDfGp-
> bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz
>



-- 
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

RE: Support for tasks groups aka pods in Mesos

2016-08-11 Thread Aaron Carey 
Hi Vinod,

I'm not sure where this should go in the design doc, but an important feature 
of kubernetes for us is the ability to share mount and network namespaces 
between containers in the pod (so containers can share mounts etc). This 
currently isn't very well supported in docker (I think Kubernetes uses its own 
containeriser to do this?).

We'd be very keen to have this ability in Mesos too.

Thanks,
Aaron



--

Aaron Carey
Production Engineer - Cloud Pipeline
Industrial Light & Magic
London
020 3751 9150


From: Vinod Kone [vinodk...@apache.org]
Sent: 09 August 2016 02:42
To: Vinod Kone
Cc: dev; user
Subject: Re: Support for tasks groups aka pods in Mesos

Sorry, sent the wrong link earlier for design doc.

Design doc: https://issues.apache.org/jira/browse/MESOS-6009

Direct link: 
https://docs.google.com/document/d/1FtcyQkDfGp-bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz

RE: Support for tasks groups aka pods in Mesos

2016-08-09 Thread Aaron Carey 
Just had a brief look over, this is great, a huge leap forward.. we were 
considering moving to Kubernetes because of the pod support, having this in 
mesos would allow us to do a lot more!


--

Aaron Carey
Production Engineer - Cloud Pipeline
Industrial Light & Magic
London
020 3751 9150


From: Vinod Kone [vinodk...@apache.org]
Sent: 09 August 2016 02:42
To: Vinod Kone
Cc: dev; user
Subject: Re: Support for tasks groups aka pods in Mesos

Sorry, sent the wrong link earlier for design doc.

Design doc: https://issues.apache.org/jira/browse/MESOS-6009

Direct link: 
https://docs.google.com/document/d/1FtcyQkDfGp-bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz

Re: Support for tasks groups aka pods in Mesos

2016-08-08 Thread Vinod Kone 
Sorry, sent the wrong link earlier for design doc.

Design doc: https://issues.apache.org/jira/browse/MESOS-6009
>

Direct link:
https://docs.google.com/document/d/1FtcyQkDfGp-bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz