RE: Support for tasks groups aka pods in Mesos
Thanks for the clarification – I’ll try to dig into this and use execute.cpp as a starting point for my problem… Kind Regards, Hubert From: Guangya Liu [mailto:gyliu...@gmail.com] Sent: Mittwoch, 21. September 2016 09:19 To: user@mesos.apache.org Cc: dev; Vinod Kone Subject: Re: Support for tasks groups aka pods in Mesos The answer is No, the taskGroup can be treated as Pod in Kubernetes, it will be a set of containers co-located and co-managed on an agent that share some resources (e.g., network namespace, volumes). For your case, you may want to split your 50 tasks to small task groups. Also the `execute` cli is just an example framework and it can only launch one task group for now, you may want to enhance it if you want to launch multiple task groups. On Wed, Sep 21, 2016 at 1:41 PM, mailto:hubert.asa...@dlr.de>> wrote: Hi, Very cool feature. I’ve seen some recent changes (MESOS-6096<https://issues.apache.org/jira/browse/MESOS-6096>) in the mesos-execute cli supporting grouped tasks which spawns the task-group on an arbitrary or specified agent node. Tested this yesterday and it works great. But if the task group holds a lot of one-off tasks (let’s say 50 tasks each 4 cores,…) how could one achieve that tasks are evenly distributed over several agents, depending on free resources? Don’t know if this is referred to pods, but do you think mesos-execute will support this? Currently I’m using chronos & cook for batch scheduling, but I’d like to achieve that every batch/group of tasks runs in its own temporary framework like it does with mesos-execute Thanks in advance & kind regards, Hubert —— Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR) German Aerospace Center German Remote Sensing Data Center | International Ground Segment | Oberpfaffenhofen | 82234 Wessling | Germany Hubert Asamer phone +49 (0) 8153 28-2894 | fax +49 (0) 8153 28-1443 | hubert.asa...@dlr.de<mailto:hubert.asa...@dlr.de> www.DLR.de/eoc<http://www.dlr.de/eoc> From: Vinod Kone [mailto:vinodk...@apache.org<mailto:vinodk...@apache.org>] Sent: Dienstag, 9. August 2016 02:54 To: dev; user Subject: Support for tasks groups aka pods in Mesos Hi folks, One of the most requested features in Mesos has been first class support for managing pod like containers. We finally have some time to focus and shepherd this work. The epic tracking this work is : https://issues.apache.org/jira/browse/MESOS-2449 Design doc: https://issues.apache.org/jira/browse/MESOS-2449 Your feedback on the design will be most welcome. Once we get agreement on the design, we can start breaking down the epic into tickets. Thanks, Vinod & Jie
Re: Support for tasks groups aka pods in Mesos
The answer is No, the taskGroup can be treated as Pod in Kubernetes, it will be a set of containers co-located and co-managed on an agent that share some resources (e.g., network namespace, volumes). For your case, you may want to split your 50 tasks to small task groups. Also the `execute` cli is just an example framework and it can only launch one task group for now, you may want to enhance it if you want to launch multiple task groups. On Wed, Sep 21, 2016 at 1:41 PM, wrote: > Hi, > > Very cool feature. > > I’ve seen some recent changes (MESOS-6096 > <https://issues.apache.org/jira/browse/MESOS-6096>) in the mesos-execute > cli supporting grouped tasks which spawns the task-group on an arbitrary or > specified agent node. Tested this yesterday and it works great. But if the > task group holds a lot of one-off tasks (let’s say 50 tasks each 4 cores,…) > how could one achieve that tasks are evenly distributed over several > agents, depending on free resources? Don’t know if this is referred to > pods, but do you think mesos-execute will support this? > > > > Currently I’m using chronos & cook for batch scheduling, but I’d like to > achieve that every batch/group of tasks runs in its own temporary > framework like it does with mesos-execute > > > > Thanks in advance & kind regards, > > Hubert > > > > —— > > *Deutsches Zentrum für Luft- und Raumfahrt* e.V. (DLR) > > German Aerospace Center > > > > > > German Remote Sensing Data Center | International Ground Segment | > Oberpfaffenhofen | 82234 Wessling | Germany > > > > *Hubert Asamer* > > phone +49 (0) 8153 28-2894 | fax +49 (0) 8153 28-1443 | > hubert.asa...@dlr.de > > www.DLR.de/eoc <http://www.dlr.de/eoc> > > > > > > > > > > *From:* Vinod Kone [mailto:vinodk...@apache.org] > *Sent:* Dienstag, 9. August 2016 02:54 > *To:* dev; user > *Subject:* Support for tasks groups aka pods in Mesos > > > > Hi folks, > > > > One of the most requested features in Mesos has been first class support > for managing pod like containers. We finally have some time to focus and > shepherd this work. > > > > The epic tracking this work is : https://issues.apache.org/ > jira/browse/MESOS-2449 > > > > Design doc: https://issues.apache.org/jira/browse/MESOS-2449 > > > > Your feedback on the design will be most welcome. Once we get agreement on > the design, we can start breaking down the epic into tickets. > > > > Thanks, > > Vinod & Jie >
RE: Support for tasks groups aka pods in Mesos
Hi, Very cool feature. I’ve seen some recent changes (MESOS-6096<https://issues.apache.org/jira/browse/MESOS-6096>) in the mesos-execute cli supporting grouped tasks which spawns the task-group on an arbitrary or specified agent node. Tested this yesterday and it works great. But if the task group holds a lot of one-off tasks (let’s say 50 tasks each 4 cores,…) how could one achieve that tasks are evenly distributed over several agents, depending on free resources? Don’t know if this is referred to pods, but do you think mesos-execute will support this? Currently I’m using chronos & cook for batch scheduling, but I’d like to achieve that every batch/group of tasks runs in its own temporary framework like it does with mesos-execute Thanks in advance & kind regards, Hubert —— Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR) German Aerospace Center German Remote Sensing Data Center | International Ground Segment | Oberpfaffenhofen | 82234 Wessling | Germany Hubert Asamer phone +49 (0) 8153 28-2894 | fax +49 (0) 8153 28-1443 | hubert.asa...@dlr.de<mailto:hubert.asa...@dlr.de> www.DLR.de/eoc<http://www.dlr.de/eoc> From: Vinod Kone [mailto:vinodk...@apache.org] Sent: Dienstag, 9. August 2016 02:54 To: dev; user Subject: Support for tasks groups aka pods in Mesos Hi folks, One of the most requested features in Mesos has been first class support for managing pod like containers. We finally have some time to focus and shepherd this work. The epic tracking this work is : https://issues.apache.org/jira/browse/MESOS-2449 Design doc: https://issues.apache.org/jira/browse/MESOS-2449 Your feedback on the design will be most welcome. Once we get agreement on the design, we can start breaking down the epic into tickets. Thanks, Vinod & Jie
Re: Support for tasks groups aka pods in Mesos
Aaron, an important feature of kubernetes for us is the ability to share mount and > network namespaces between containers in the pod (so containers can share > mounts etc) yes, this will be supported in MVP. it's described in the design doc. - Jie On Thu, Aug 11, 2016 at 6:32 AM, Aaron Carey wrote: > Hi Vinod, > > I'm not sure where this should go in the design doc, but an important > feature of kubernetes for us is the ability to share mount and network > namespaces between containers in the pod (so containers can share mounts > etc). This currently isn't very well supported in docker (I think > Kubernetes uses its own containeriser to do this?). > > We'd be very keen to have this ability in Mesos too. > > Thanks, > Aaron > > > -- > > Aaron Carey > Production Engineer - Cloud Pipeline > Industrial Light & Magic > London > 020 3751 9150 > > -- > *From:* Vinod Kone [vinodk...@apache.org] > *Sent:* 09 August 2016 02:42 > *To:* Vinod Kone > *Cc:* dev; user > *Subject:* Re: Support for tasks groups aka pods in Mesos > > Sorry, sent the wrong link earlier for design doc. > > Design doc: https://issues.apache.org/jira/browse/MESOS-6009 >> > > Direct link: https://docs.google.com/document/d/1FtcyQkDfGp- > bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz >
Re: Support for tasks groups aka pods in Mesos
+1, can't wait to use it. 2016-08-11 21:32 GMT+08:00 Aaron Carey : > Hi Vinod, > > I'm not sure where this should go in the design doc, but an important > feature of kubernetes for us is the ability to share mount and network > namespaces between containers in the pod (so containers can share mounts > etc). This currently isn't very well supported in docker (I think > Kubernetes uses its own containeriser to do this?). > > We'd be very keen to have this ability in Mesos too. > > Thanks, > Aaron > > > -- > > Aaron Carey > Production Engineer - Cloud Pipeline > Industrial Light & Magic > London > 020 3751 9150 > > -- > *From:* Vinod Kone [vinodk...@apache.org] > *Sent:* 09 August 2016 02:42 > *To:* Vinod Kone > *Cc:* dev; user > *Subject:* Re: Support for tasks groups aka pods in Mesos > > Sorry, sent the wrong link earlier for design doc. > > Design doc: https://issues.apache.org/jira/browse/MESOS-6009 >> > > Direct link: https://docs.google.com/document/d/1FtcyQkDfGp- > bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz > -- Deshi Xiao Twitter: xds2000 E-mail: xiaods(AT)gmail.com
RE: Support for tasks groups aka pods in Mesos
Hi Vinod, I'm not sure where this should go in the design doc, but an important feature of kubernetes for us is the ability to share mount and network namespaces between containers in the pod (so containers can share mounts etc). This currently isn't very well supported in docker (I think Kubernetes uses its own containeriser to do this?). We'd be very keen to have this ability in Mesos too. Thanks, Aaron -- Aaron Carey Production Engineer - Cloud Pipeline Industrial Light & Magic London 020 3751 9150 From: Vinod Kone [vinodk...@apache.org] Sent: 09 August 2016 02:42 To: Vinod Kone Cc: dev; user Subject: Re: Support for tasks groups aka pods in Mesos Sorry, sent the wrong link earlier for design doc. Design doc: https://issues.apache.org/jira/browse/MESOS-6009 Direct link: https://docs.google.com/document/d/1FtcyQkDfGp-bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz
RE: Support for tasks groups aka pods in Mesos
Just had a brief look over, this is great, a huge leap forward.. we were considering moving to Kubernetes because of the pod support, having this in mesos would allow us to do a lot more! -- Aaron Carey Production Engineer - Cloud Pipeline Industrial Light & Magic London 020 3751 9150 From: Vinod Kone [vinodk...@apache.org] Sent: 09 August 2016 02:42 To: Vinod Kone Cc: dev; user Subject: Re: Support for tasks groups aka pods in Mesos Sorry, sent the wrong link earlier for design doc. Design doc: https://issues.apache.org/jira/browse/MESOS-6009 Direct link: https://docs.google.com/document/d/1FtcyQkDfGp-bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz
Re: Support for tasks groups aka pods in Mesos
Sorry, sent the wrong link earlier for design doc. Design doc: https://issues.apache.org/jira/browse/MESOS-6009 > Direct link: https://docs.google.com/document/d/1FtcyQkDfGp-bPHTW4pUoqQCgVlPde936bo-IIENO_ho/edit#heading=h.ip4t59nlogfz
Support for tasks groups aka pods in Mesos
Hi folks, One of the most requested features in Mesos has been first class support for managing pod like containers. We finally have some time to focus and shepherd this work. The epic tracking this work is : https://issues.apache.org/jira/browse/MESOS-2449 Design doc: https://issues.apache.org/jira/browse/MESOS-2449 Your feedback on the design will be most welcome. Once we get agreement on the design, we can start breaking down the epic into tickets. Thanks, Vinod & Jie