Re: [openstack-dev] [Magnum] Scheduling for Magnum
Thanks Sylvain, we did not work out the API requirement till now but I think the requirement should be similar with nova: we need select_destination to select the best target host based on filters and weights. There are also some discussions here https://blueprints.launchpad.net/magnum/+spec/magnum-scheduler-for-docker Thanks! 2015-02-09 16:22 GMT+08:00 Sylvain Bauza sba...@redhat.com: Hi Magnum team, Le 07/02/2015 19:24, Steven Dake (stdake) a écrit : From: Eric Windisch e...@windisch.us Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Saturday, February 7, 2015 at 10:09 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. The Gantt team explored that option by the Icehouse cycle and it failed with a lot of problems. I won't list all of those, but I'll just explain that we discovered how the Scheduler and the Nova compute manager were tighly coupled, which was meaning that a repository fork was really difficult to do without reducing the tech debt. That said, our concerns were far different from the Magnum team : it was about having feature parity and replacing the current Nova scheduler, while your team is just saying that they want to have something about containers. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch Adrian Eric, I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve. I don't want to give my opinion about which option you should take as I don't really know your needs. If I understand correctly, this is about having a scheduler providing affinity rules for containers. Do you have a document explaining which interfaces you're looking for, which kind of APIs you're wanting or what's missing with the current Nova scheduler ? MHO is that the technology shouldn't drive your decision : whatever the backend is (swarmd or an inherited nova scheduler), your interfaces should be the same. -Sylvain Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay Lau (Guangya Liu) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
From: Jay Lau jay.lau@gmail.commailto:jay.lau@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 11:31 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum Steve, So you mean we should focus on docker and k8s scheduler? I was a bit confused, why do we need to care k8s? As the k8s cluster was created by heat and once the k8s was created, the k8s has its own scheduler for creating pods/service/rcs. So seems we only need to care scheduler for native docker and ironic bay, comments? Ya scheduler only matters for native docker. Ironic bay can be k8s or docker+swarm or something similar. But yup, I understand your point. Thanks! 2015-02-10 12:32 GMT+08:00 Steven Dake (stdake) std...@cisco.commailto:std...@cisco.com: From: Joe Gordon joe.gord...@gmail.commailto:joe.gord...@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 6:41 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum On Mon, Feb 9, 2015 at 6:00 AM, Steven Dake (stdake) std...@cisco.commailto:std...@cisco.com wrote: On 2/9/15, 3:02 AM, Thierry Carrez thie...@openstack.orgmailto:thie...@openstack.org wrote: Adrian Otto wrote: [...] We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I haven't looked enough into Swarm to answer that question myself, but how much would #2 tie Magnum to Docker containers ? There is value for Magnum to support other container engines / formats (think Rocket/Appc) in the long run, so we should avoid early design choices that would prevent such support in the future. Thierry, Magnum has an object type of a bay which represents the underlying cluster architecture used. This could be kubernetes, raw docker, swarmd, or some future invention. This way Magnum can grow independently of the underlying technology and provide a satisfactory user experience dealing with the chaos that is the container development world :) While I don't disagree with anything said here, this does sound a lot like https://xkcd.com/927/ Andrew had suggested offering a unified standard user experience and API. I think that matches the 927 comic pretty well. I think we should offer each type of system using APIs that are similar in nature but that offer the native features of the system. In other words, we will offer integration across the various container landscape with OpenStack. We should strive to be conservative and pragmatic in our systems support and only support container schedulers and container managers that have become strongly emergent systems. At this point that is docker and kubernetes. Mesos might fit that definition as well. Swarmd and rocket are not yet strongly emergent, but they show promise of becoming so. As a result, they are clearly systems we should be thinking about for our roadmap. All of these systems present very similar operational models. At some point competition will choke off new system design placing an upper bound on the amount of systems we have to deal with. Regards -steve We will absolutely support relevant container technology, likely through new Bay formats (which are really just heat templates). Regards -steve -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org
Re: [openstack-dev] [Magnum] Scheduling for Magnum
Thanks Steve, just want to discuss more for this. Then per Andrew's comments, we need a generic scheduling interface, but if our focus is native docker, then does this still needed? Thanks! 2015-02-10 14:52 GMT+08:00 Steven Dake (stdake) std...@cisco.com: From: Jay Lau jay.lau@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 11:31 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum Steve, So you mean we should focus on docker and k8s scheduler? I was a bit confused, why do we need to care k8s? As the k8s cluster was created by heat and once the k8s was created, the k8s has its own scheduler for creating pods/service/rcs. So seems we only need to care scheduler for native docker and ironic bay, comments? Ya scheduler only matters for native docker. Ironic bay can be k8s or docker+swarm or something similar. But yup, I understand your point. Thanks! 2015-02-10 12:32 GMT+08:00 Steven Dake (stdake) std...@cisco.com: From: Joe Gordon joe.gord...@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 6:41 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum On Mon, Feb 9, 2015 at 6:00 AM, Steven Dake (stdake) std...@cisco.com wrote: On 2/9/15, 3:02 AM, Thierry Carrez thie...@openstack.org wrote: Adrian Otto wrote: [...] We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I haven't looked enough into Swarm to answer that question myself, but how much would #2 tie Magnum to Docker containers ? There is value for Magnum to support other container engines / formats (think Rocket/Appc) in the long run, so we should avoid early design choices that would prevent such support in the future. Thierry, Magnum has an object type of a bay which represents the underlying cluster architecture used. This could be kubernetes, raw docker, swarmd, or some future invention. This way Magnum can grow independently of the underlying technology and provide a satisfactory user experience dealing with the chaos that is the container development world :) While I don't disagree with anything said here, this does sound a lot like https://xkcd.com/927/ Andrew had suggested offering a unified standard user experience and API. I think that matches the 927 comic pretty well. I think we should offer each type of system using APIs that are similar in nature but that offer the native features of the system. In other words, we will offer integration across the various container landscape with OpenStack. We should strive to be conservative and pragmatic in our systems support and only support container schedulers and container managers that have become strongly emergent systems. At this point that is docker and kubernetes. Mesos might fit that definition as well. Swarmd and rocket are not yet strongly emergent, but they show promise of becoming so. As a result, they are clearly systems we should be thinking about for our roadmap. All of these systems present very similar operational models. At some point competition will choke off new system design placing an upper bound on the amount of systems we have to deal with. Regards -steve We will absolutely support relevant container technology, likely through new Bay formats (which are really just heat templates). Regards -steve -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
From: Andrew Melton andrew.mel...@rackspace.commailto:andrew.mel...@rackspace.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 10:38 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum I think Sylvain is getting at an important point. Magnum is trying to be as agnostic as possible when it comes to selecting a backend. Because of that, I think the biggest benefit to Magnum would be a generic scheduling interface that each pod type would implement. A pod type with a backend providing scheduling could implement a thin scheduler that simply translates the generic requests into something the backend can understand. And a pod type requiring outside scheduling could implement something more heavy. If we are careful to keep the heavy scheduling generic enough to be shared between backends requiring it, we could hopefully swap in an implementation using Gantt once that is ready. Great mid-cycle topic discussion topic. Can you add it to the planning etherpad? Thanks -steve --Andrew From: Jay Lau [jay.lau@gmail.commailto:jay.lau@gmail.com] Sent: Monday, February 09, 2015 4:36 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum Thanks Sylvain, we did not work out the API requirement till now but I think the requirement should be similar with nova: we need select_destination to select the best target host based on filters and weights. There are also some discussions here https://blueprints.launchpad.net/magnum/+spec/magnum-scheduler-for-docker Thanks! 2015-02-09 16:22 GMT+08:00 Sylvain Bauza sba...@redhat.commailto:sba...@redhat.com: Hi Magnum team, Le 07/02/2015 19:24, Steven Dake (stdake) a écrit : From: Eric Windisch e...@windisch.usmailto:e...@windisch.us Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Saturday, February 7, 2015 at 10:09 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. The Gantt team explored that option by the Icehouse cycle and it failed with a lot of problems. I won't list all of those, but I'll just explain that we discovered how the Scheduler and the Nova compute manager were tighly coupled, which was meaning that a repository fork was really difficult to do without reducing the tech debt. That said, our concerns were far different from the Magnum team : it was about having feature parity and replacing the current Nova scheduler, while your team is just saying that they want to have something about containers. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch Adrian Eric, I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve. I don't want to give my opinion about which option you should take as I don't really know your needs. If I understand correctly, this is about having a scheduler providing affinity rules for containers. Do you have a document explaining which interfaces you're looking for, which kind of APIs you're wanting or what's missing with the current Nova scheduler ? MHO is that the technology shouldn't drive your decision : whatever the backend is (swarmd or an inherited nova scheduler), your interfaces should be the same. -Sylvain Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribemailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org
Re: [openstack-dev] [Magnum] Scheduling for Magnum
Hi Magnum team, Le 07/02/2015 19:24, Steven Dake (stdake) a écrit : From: Eric Windisch e...@windisch.us mailto:e...@windisch.us Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org Date: Saturday, February 7, 2015 at 10:09 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. The Gantt team explored that option by the Icehouse cycle and it failed with a lot of problems. I won't list all of those, but I'll just explain that we discovered how the Scheduler and the Nova compute manager were tighly coupled, which was meaning that a repository fork was really difficult to do without reducing the tech debt. That said, our concerns were far different from the Magnum team : it was about having feature parity and replacing the current Nova scheduler, while your team is just saying that they want to have something about containers. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch Adrian Eric, I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve. I don't want to give my opinion about which option you should take as I don't really know your needs. If I understand correctly, this is about having a scheduler providing affinity rules for containers. Do you have a document explaining which interfaces you're looking for, which kind of APIs you're wanting or what's missing with the current Nova scheduler ? MHO is that the technology shouldn't drive your decision : whatever the backend is (swarmd or an inherited nova scheduler), your interfaces should be the same. -Sylvain Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
I think it’s fair to assert that our generic scheduling interface should be based on Gantt. When that approaches a maturity point where it’s appropriate to leverage Gantt for container use cases, we should definitely consider switching to that. We should remain engaged in Gantt design decisions along the way to provide input. In the short term we want a solution that works nicely for our Docker handler, because that’s an obvious functionality gap. The k8s handler already has a scheduler, so it can remain unchanged. Let’s not fall into a trap of over-engineering the scheduler, as that can be very tempting but yield limited value. My suggestion is that we focus on the right solution for the Docker backend for now, and keep in mind that we want a general purpose scheduler in the future that could be adapted to work with a variety of container backends. I want to recognize that Andrew’s thoughts are well considered to avoid rework and remain agnostic about container backends. Further, I think resource scheduling is the sort of problem domain that would lend itself well to a common solution with numerous use cases. If you look at the various ones that exist today, there are lots of similarities. We will find a multitude of scheduling algorithms, but probably not uniquely innovative scheduling interfaces. The interface to a scheduler will be relatively simple, and we could afford to collaborate a bit with the Gantt team to get solid ideas on the table for that. Let’s table that pursuit for now, and re-engage at our Midcycle meetup to explore that topic further. In the mean time, I’d like us to iterate on a suitable point solution for the Docker backend. A final iteration of that work may be to yank it completely, and replace it with a common scheduler at a later point. I’m willing to accept that tradeoff for a quick delivery of a Docker specific scheduler that we can learn from and iterate. Cheers, Adrian On Feb 9, 2015, at 10:57 PM, Jay Lau jay.lau@gmail.commailto:jay.lau@gmail.com wrote: Thanks Steve, just want to discuss more for this. Then per Andrew's comments, we need a generic scheduling interface, but if our focus is native docker, then does this still needed? Thanks! 2015-02-10 14:52 GMT+08:00 Steven Dake (stdake) std...@cisco.commailto:std...@cisco.com: From: Jay Lau jay.lau@gmail.commailto:jay.lau@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 11:31 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum Steve, So you mean we should focus on docker and k8s scheduler? I was a bit confused, why do we need to care k8s? As the k8s cluster was created by heat and once the k8s was created, the k8s has its own scheduler for creating pods/service/rcs. So seems we only need to care scheduler for native docker and ironic bay, comments? Ya scheduler only matters for native docker. Ironic bay can be k8s or docker+swarm or something similar. But yup, I understand your point. Thanks! 2015-02-10 12:32 GMT+08:00 Steven Dake (stdake) std...@cisco.commailto:std...@cisco.com: From: Joe Gordon joe.gord...@gmail.commailto:joe.gord...@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 6:41 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum On Mon, Feb 9, 2015 at 6:00 AM, Steven Dake (stdake) std...@cisco.commailto:std...@cisco.com wrote: On 2/9/15, 3:02 AM, Thierry Carrez thie...@openstack.orgmailto:thie...@openstack.org wrote: Adrian Otto wrote: [...] We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I haven't looked enough into Swarm to answer that question myself, but how much would #2 tie Magnum to Docker containers ? There is value for Magnum to support other container engines / formats (think Rocket/Appc) in the long run, so we should avoid early design choices that would prevent such support in the future. Thierry, Magnum has an object type of a bay which represents the underlying cluster architecture used. This could be kubernetes, raw docker, swarmd, or some future invention. This way Magnum can grow
Re: [openstack-dev] [Magnum] Scheduling for Magnum
Thanks Adrian for the clear clarification, clear now. OK, we can focus on the right solution for the Docker back-end for now. 2015-02-10 15:35 GMT+08:00 Adrian Otto adrian.o...@rackspace.com: I think it’s fair to assert that our generic scheduling interface should be based on Gantt. When that approaches a maturity point where it’s appropriate to leverage Gantt for container use cases, we should definitely consider switching to that. We should remain engaged in Gantt design decisions along the way to provide input. In the short term we want a solution that works nicely for our Docker handler, because that’s an obvious functionality gap. The k8s handler already has a scheduler, so it can remain unchanged. Let’s not fall into a trap of over-engineering the scheduler, as that can be very tempting but yield limited value. My suggestion is that we focus on the right solution for the Docker backend for now, and keep in mind that we want a general purpose scheduler in the future that could be adapted to work with a variety of container backends. I want to recognize that Andrew’s thoughts are well considered to avoid rework and remain agnostic about container backends. Further, I think resource scheduling is the sort of problem domain that would lend itself well to a common solution with numerous use cases. If you look at the various ones that exist today, there are lots of similarities. We will find a multitude of scheduling algorithms, but probably not uniquely innovative scheduling interfaces. The interface to a scheduler will be relatively simple, and we could afford to collaborate a bit with the Gantt team to get solid ideas on the table for that. Let’s table that pursuit for now, and re-engage at our Midcycle meetup to explore that topic further. In the mean time, I’d like us to iterate on a suitable point solution for the Docker backend. A final iteration of that work may be to yank it completely, and replace it with a common scheduler at a later point. I’m willing to accept that tradeoff for a quick delivery of a Docker specific scheduler that we can learn from and iterate. Cheers, Adrian On Feb 9, 2015, at 10:57 PM, Jay Lau jay.lau@gmail.com wrote: Thanks Steve, just want to discuss more for this. Then per Andrew's comments, we need a generic scheduling interface, but if our focus is native docker, then does this still needed? Thanks! 2015-02-10 14:52 GMT+08:00 Steven Dake (stdake) std...@cisco.com: From: Jay Lau jay.lau@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 11:31 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum Steve, So you mean we should focus on docker and k8s scheduler? I was a bit confused, why do we need to care k8s? As the k8s cluster was created by heat and once the k8s was created, the k8s has its own scheduler for creating pods/service/rcs. So seems we only need to care scheduler for native docker and ironic bay, comments? Ya scheduler only matters for native docker. Ironic bay can be k8s or docker+swarm or something similar. But yup, I understand your point. Thanks! 2015-02-10 12:32 GMT+08:00 Steven Dake (stdake) std...@cisco.com: From: Joe Gordon joe.gord...@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 6:41 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum On Mon, Feb 9, 2015 at 6:00 AM, Steven Dake (stdake) std...@cisco.com wrote: On 2/9/15, 3:02 AM, Thierry Carrez thie...@openstack.org wrote: Adrian Otto wrote: [...] We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I haven't looked enough into Swarm to answer that question myself, but how much would #2 tie Magnum to Docker containers ? There is value for Magnum to support other container engines / formats (think Rocket/Appc) in the long run, so we should avoid early design choices that would prevent such support in the future. Thierry, Magnum has an object type of a bay which represents the underlying cluster architecture used. This could be kubernetes, raw docker, swarmd, or some future invention. This way Magnum can grow independently of the underlying technology and provide a satisfactory user
Re: [openstack-dev] [Magnum] Scheduling for Magnum
I think Sylvain is getting at an important point. Magnum is trying to be as agnostic as possible when it comes to selecting a backend. Because of that, I think the biggest benefit to Magnum would be a generic scheduling interface that each pod type would implement. A pod type with a backend providing scheduling could implement a thin scheduler that simply translates the generic requests into something the backend can understand. And a pod type requiring outside scheduling could implement something more heavy. If we are careful to keep the heavy scheduling generic enough to be shared between backends requiring it, we could hopefully swap in an implementation using Gantt once that is ready. --Andrew From: Jay Lau [jay.lau@gmail.com] Sent: Monday, February 09, 2015 4:36 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum Thanks Sylvain, we did not work out the API requirement till now but I think the requirement should be similar with nova: we need select_destination to select the best target host based on filters and weights. There are also some discussions here https://blueprints.launchpad.net/magnum/+spec/magnum-scheduler-for-docker Thanks! 2015-02-09 16:22 GMT+08:00 Sylvain Bauza sba...@redhat.commailto:sba...@redhat.com: Hi Magnum team, Le 07/02/2015 19:24, Steven Dake (stdake) a écrit : From: Eric Windisch e...@windisch.usmailto:e...@windisch.us Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Saturday, February 7, 2015 at 10:09 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. The Gantt team explored that option by the Icehouse cycle and it failed with a lot of problems. I won't list all of those, but I'll just explain that we discovered how the Scheduler and the Nova compute manager were tighly coupled, which was meaning that a repository fork was really difficult to do without reducing the tech debt. That said, our concerns were far different from the Magnum team : it was about having feature parity and replacing the current Nova scheduler, while your team is just saying that they want to have something about containers. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch Adrian Eric, I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve. I don't want to give my opinion about which option you should take as I don't really know your needs. If I understand correctly, this is about having a scheduler providing affinity rules for containers. Do you have a document explaining which interfaces you're looking for, which kind of APIs you're wanting or what's missing with the current Nova scheduler ? MHO is that the technology shouldn't drive your decision : whatever the backend is (swarmd or an inherited nova scheduler), your interfaces should be the same. -Sylvain Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribemailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay Lau (Guangya Liu) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
Steve, So you mean we should focus on docker and k8s scheduler? I was a bit confused, why do we need to care k8s? As the k8s cluster was created by heat and once the k8s was created, the k8s has its own scheduler for creating pods/service/rcs. So seems we only need to care scheduler for native docker and ironic bay, comments? Thanks! 2015-02-10 12:32 GMT+08:00 Steven Dake (stdake) std...@cisco.com: From: Joe Gordon joe.gord...@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 6:41 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum On Mon, Feb 9, 2015 at 6:00 AM, Steven Dake (stdake) std...@cisco.com wrote: On 2/9/15, 3:02 AM, Thierry Carrez thie...@openstack.org wrote: Adrian Otto wrote: [...] We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I haven't looked enough into Swarm to answer that question myself, but how much would #2 tie Magnum to Docker containers ? There is value for Magnum to support other container engines / formats (think Rocket/Appc) in the long run, so we should avoid early design choices that would prevent such support in the future. Thierry, Magnum has an object type of a bay which represents the underlying cluster architecture used. This could be kubernetes, raw docker, swarmd, or some future invention. This way Magnum can grow independently of the underlying technology and provide a satisfactory user experience dealing with the chaos that is the container development world :) While I don't disagree with anything said here, this does sound a lot like https://xkcd.com/927/ Andrew had suggested offering a unified standard user experience and API. I think that matches the 927 comic pretty well. I think we should offer each type of system using APIs that are similar in nature but that offer the native features of the system. In other words, we will offer integration across the various container landscape with OpenStack. We should strive to be conservative and pragmatic in our systems support and only support container schedulers and container managers that have become strongly emergent systems. At this point that is docker and kubernetes. Mesos might fit that definition as well. Swarmd and rocket are not yet strongly emergent, but they show promise of becoming so. As a result, they are clearly systems we should be thinking about for our roadmap. All of these systems present very similar operational models. At some point competition will choke off new system design placing an upper bound on the amount of systems we have to deal with. Regards -steve We will absolutely support relevant container technology, likely through new Bay formats (which are really just heat templates). Regards -steve -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay Lau (Guangya Liu) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
Adrian Otto wrote: [...] We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I haven't looked enough into Swarm to answer that question myself, but how much would #2 tie Magnum to Docker containers ? There is value for Magnum to support other container engines / formats (think Rocket/Appc) in the long run, so we should avoid early design choices that would prevent such support in the future. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
On 2/9/15, 3:02 AM, Thierry Carrez thie...@openstack.org wrote: Adrian Otto wrote: [...] We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I haven't looked enough into Swarm to answer that question myself, but how much would #2 tie Magnum to Docker containers ? There is value for Magnum to support other container engines / formats (think Rocket/Appc) in the long run, so we should avoid early design choices that would prevent such support in the future. Thierry, Magnum has an object type of a bay which represents the underlying cluster architecture used. This could be kubernetes, raw docker, swarmd, or some future invention. This way Magnum can grow independently of the underlying technology and provide a satisfactory user experience dealing with the chaos that is the container development world :) We will absolutely support relevant container technology, likely through new Bay formats (which are really just heat templates). Regards -steve -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
Steve, On Feb 9, 2015, at 9:54 AM, Steven Dake (stdake) std...@cisco.commailto:std...@cisco.com wrote: From: Andrew Melton andrew.mel...@rackspace.commailto:andrew.mel...@rackspace.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 10:38 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum I think Sylvain is getting at an important point. Magnum is trying to be as agnostic as possible when it comes to selecting a backend. Because of that, I think the biggest benefit to Magnum would be a generic scheduling interface that each pod type would implement. A pod type with a backend providing scheduling could implement a thin scheduler that simply translates the generic requests into something the backend can understand. And a pod type requiring outside scheduling could implement something more heavy. If we are careful to keep the heavy scheduling generic enough to be shared between backends requiring it, we could hopefully swap in an implementation using Gantt once that is ready. Great mid-cycle topic discussion topic. Can you add it to the planning etherpad? Yes, it was listed as #5 here: https://etherpad.openstack.org/p/magnum-midcycle-topics We will arrange that further up the priority list as soon as we feel that list is complete, and ready for sorting. Adrian Thanks -steve --Andrew From: Jay Lau [jay.lau@gmail.commailto:jay.lau@gmail.com] Sent: Monday, February 09, 2015 4:36 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum Thanks Sylvain, we did not work out the API requirement till now but I think the requirement should be similar with nova: we need select_destination to select the best target host based on filters and weights. There are also some discussions here https://blueprints.launchpad.net/magnum/+spec/magnum-scheduler-for-docker Thanks! 2015-02-09 16:22 GMT+08:00 Sylvain Bauza sba...@redhat.commailto:sba...@redhat.com: Hi Magnum team, Le 07/02/2015 19:24, Steven Dake (stdake) a écrit : From: Eric Windisch e...@windisch.usmailto:e...@windisch.us Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Saturday, February 7, 2015 at 10:09 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. The Gantt team explored that option by the Icehouse cycle and it failed with a lot of problems. I won't list all of those, but I'll just explain that we discovered how the Scheduler and the Nova compute manager were tighly coupled, which was meaning that a repository fork was really difficult to do without reducing the tech debt. That said, our concerns were far different from the Magnum team : it was about having feature parity and replacing the current Nova scheduler, while your team is just saying that they want to have something about containers. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch Adrian Eric, I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve. I don't want to give my opinion about which option you should take as I don't really know your needs. If I understand correctly, this is about having a scheduler providing affinity rules for containers. Do you have a document explaining which interfaces you're looking for, which kind of APIs you're wanting or what's missing with the current Nova scheduler ? MHO is that the technology shouldn't drive your decision : whatever the backend is (swarmd or an inherited nova scheduler), your interfaces should be the same. -Sylvain Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribemailto:openstack-dev-requ
Re: [openstack-dev] [Magnum] Scheduling for Magnum
On Mon, Feb 9, 2015 at 6:00 AM, Steven Dake (stdake) std...@cisco.com wrote: On 2/9/15, 3:02 AM, Thierry Carrez thie...@openstack.org wrote: Adrian Otto wrote: [...] We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I haven't looked enough into Swarm to answer that question myself, but how much would #2 tie Magnum to Docker containers ? There is value for Magnum to support other container engines / formats (think Rocket/Appc) in the long run, so we should avoid early design choices that would prevent such support in the future. Thierry, Magnum has an object type of a bay which represents the underlying cluster architecture used. This could be kubernetes, raw docker, swarmd, or some future invention. This way Magnum can grow independently of the underlying technology and provide a satisfactory user experience dealing with the chaos that is the container development world :) While I don't disagree with anything said here, this does sound a lot like https://xkcd.com/927/ We will absolutely support relevant container technology, likely through new Bay formats (which are really just heat templates). Regards -steve -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
From: Joe Gordon joe.gord...@gmail.commailto:joe.gord...@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, February 9, 2015 at 6:41 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum On Mon, Feb 9, 2015 at 6:00 AM, Steven Dake (stdake) std...@cisco.commailto:std...@cisco.com wrote: On 2/9/15, 3:02 AM, Thierry Carrez thie...@openstack.orgmailto:thie...@openstack.org wrote: Adrian Otto wrote: [...] We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I haven't looked enough into Swarm to answer that question myself, but how much would #2 tie Magnum to Docker containers ? There is value for Magnum to support other container engines / formats (think Rocket/Appc) in the long run, so we should avoid early design choices that would prevent such support in the future. Thierry, Magnum has an object type of a bay which represents the underlying cluster architecture used. This could be kubernetes, raw docker, swarmd, or some future invention. This way Magnum can grow independently of the underlying technology and provide a satisfactory user experience dealing with the chaos that is the container development world :) While I don't disagree with anything said here, this does sound a lot like https://xkcd.com/927/ Andrew had suggested offering a unified standard user experience and API. I think that matches the 927 comic pretty well. I think we should offer each type of system using APIs that are similar in nature but that offer the native features of the system. In other words, we will offer integration across the various container landscape with OpenStack. We should strive to be conservative and pragmatic in our systems support and only support container schedulers and container managers that have become strongly emergent systems. At this point that is docker and kubernetes. Mesos might fit that definition as well. Swarmd and rocket are not yet strongly emergent, but they show promise of becoming so. As a result, they are clearly systems we should be thinking about for our roadmap. All of these systems present very similar operational models. At some point competition will choke off new system design placing an upper bound on the amount of systems we have to deal with. Regards -steve We will absolutely support relevant container technology, likely through new Bay formats (which are really just heat templates). Regards -steve -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
From: Eric Windisch e...@windisch.usmailto:e...@windisch.us Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Saturday, February 7, 2015 at 10:09 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch Adrian Eric, I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve. Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
I second that. my first instinct is the same as Steve. -- dims On Sat, Feb 7, 2015 at 1:24 PM, Steven Dake (stdake) std...@cisco.com wrote: From: Eric Windisch e...@windisch.us Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Saturday, February 7, 2015 at 10:09 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch Adrian Eric, I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve. Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
Sorry for late. I'm OK with both nova scheduler and swarm as both of them using same logic for scheduling: filter + strategy (weight), and the code structure/logic is also very similar between nova scheduler and swarm. In my understanding, even if we use swarm and translate go to python, after this scheduler is added to magnum, we may notice that it is very similar with nova scheduler. Thanks! 2015-02-08 11:01 GMT+08:00 Adrian Otto adrian.o...@rackspace.com: Ok, so if we proceed using Swarm as our first pursuit, and we want to add things to Swarm like scheduling hints, we should open a Magnum bug ticket to track each of the upstream patches, and I can help to bird dog those. We should not shy away from upstream enhancements until we get firm feedback suggesting our contributions are out of scope. Adrian Original message From: Steven Dake (stdake) Date:02/07/2015 10:27 AM (GMT-08:00) To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum From: Eric Windisch e...@windisch.us Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Saturday, February 7, 2015 at 10:09 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch Adrian Eric, I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve. Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay Lau (Guangya Liu) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
Ok, so if we proceed using Swarm as our first pursuit, and we want to add things to Swarm like scheduling hints, we should open a Magnum bug ticket to track each of the upstream patches, and I can help to bird dog those. We should not shy away from upstream enhancements until we get firm feedback suggesting our contributions are out of scope. Adrian Original message From: Steven Dake (stdake) Date:02/07/2015 10:27 AM (GMT-08:00) To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum From: Eric Windisch e...@windisch.usmailto:e...@windisch.us Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Saturday, February 7, 2015 at 10:09 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Magnum] Scheduling for Magnum 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. I see #2 as not an alternative but possibly an also. Swarm uses the Docker API, although they're only about 75% compatible at the moment. Ideally, the Docker backend would work with both single docker hosts and clusters of Docker machines powered by Swarm. It would be nice, however, if scheduler hints could be passed from Magnum to Swarm. Regards, Eric Windisch Adrian Eric, I would prefer to keep things simple and just integrate directly with swarm and leave out any cherry-picking from Nova. It would be better to integrate scheduling hints into Swarm, but I’m sure the swarm upstream is busy with requests and this may be difficult to achieve. Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Magnum] Scheduling for Magnum
On Sat, 2015-02-07 at 00:44 +, Adrian Otto wrote: Magnum Team, In our initial spec, we addressed the subject of resource scheduling. Our plan was to begin with a naive scheduler that places resources on a specified Node and can sequentially fill Nodes if one is not specified. Magnum supports multiple conductor backends[1], one of which is our Kubernetes backend. We also have a native Docker backend that we would like to enhance so that when pods or containers are created, the target nodes can be selected according to user-supplied filters. Some examples of this are: Constraint, Affinity, Anti-Affinity, Health We have multiple options for solving this challenge. Here are a few: 1) Cherry pick scheduler code from Nova, which already has a working a filter scheduler design. 2) Integrate swarmd to leverage its scheduler[2]. 3) Wait for the Gantt, when Nova Scheduler to be moved out of Nova. This is expected to happen about a year from now, possibly sooner. 4) Write our own filter scheduler, inspired by Nova. I suggest that we deserve to have a scheduling implementation for our native docker backend before Gantt is ready. It’s unrealistic that the Magnum team will be able to accelerate Gantt’s progress, as significant changes must be made in Nova for this to happen. The Nova team is best equipped to do this. It requires active participation from Nova’s core review team, which has limited bandwidth, and other priorities to focus on. I think we unanimously agree that we would prefer to use Gantt, if it were available sooner. I suggest we also rule out option 4, because it amounts to re-inventing the wheel. That leaves us with options 1 and 2 in the short term. The disadvantage of either of these approaches is that we will likely need to remove them and replace them with Gantt (or a derivative work) once it matures. The advantage of option 1 is that python code already exists for this, and we know it works well in Nova. We could cherry pick that over, and drop it directly into Magnum. The advantage of option 2 is that we leverage the talents of the developers working on Swarm, which is better than option 4. New features are likely to surface in parallel with our efforts, so we would enjoy the benefits of those without expending work in our own project. So, how do you feel about options 1 and 2? Which do you feel is more suitable for Magnum? What other options should we consider that might be better than either of these choices? I have a slight preference for option 2 - integrating with Swarm, but I could be persuaded to pick option 1, or something even more brilliant. Please discuss. Got to say that Option 1 looks far preferable. As you say, we have to switch to Gantt eventually, so it might end up being an expensive and difficult retro fit with Option 2. With Option 1, we look mostly like the Nova scheduler, so can let them take the initial hit of doing the shift to Gantt and slipstream in their wake once the major pain points are ironed out. James __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
