Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ?
Hi. At this moment, Neutron doesn't offer physical network information for now. There is a proposal for it[1] and it was discussed at the summit [2]. Although it is still early design phase[3][4], using routing protocol will surely help to discover physical network topology. [1] https://wiki.openstack.org/wiki/Topology-as-a-service [2] https://etherpad.openstack.org/p/hierarchical_network_topology [3] http://lists.openstack.org/pipermail/openstack-dev/2014-May/035868.html [4] https://review.openstack.org/#/c/91275/ thanks, Isaku Yamahata On Fri, May 30, 2014 at 11:11:46AM +0200, Mathieu Rohon mathieu.ro...@gmail.com wrote: I'm also very interested in scheduling VMs with Network requirement. This seems to be in the scope of NFV workgroup [1]. For instance, I think that scheduling should take into account bandwith/QoS requirement for a VM, or specific Nic requirement for a VM (availability of a SRIOV Vif on compute nodes). To do this, it seems that Neutron should report its available capacity (Vif availability, bandwith availability) for each compute node, and Gantt should take this reporting into account for scheduling. [1]https://etherpad.openstack.org/p/juno-nfv-bof On Fri, May 30, 2014 at 10:47 AM, jcsf jcsf31...@gmail.com wrote: Sylvain, Thank you for the background ??? I will educate myself on this work. Thanks, John *From:* Sylvain Bauza [mailto:sba...@redhat.com] *Sent:* Friday, May 30, 2014 11:31 AM *To:* OpenStack Development Mailing List (not for usage questions) *Cc:* jcsf; 'Carl Baldwin'; 'A, Keshava' *Subject:* Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ? Le 30/05/2014 10:06, jcsf a écrit : Carl, A new routing protocol is certainly of great interest. Are you working with IETF or can you share more here? WRT:Nova Schedule - There still are requirements for the Schedule to taking into consideration network as a resource. My focus is to figure out how to add network capabilities to the Scheduler’s algorithm while still maintaining clean separation of concerns between Nova and Neutron. We wouldn’t want to get back into the nova-network situation. John As it was previously mentioned, there are already different kinds of grouping for VMs in Nova that probably don't require to add new network-specific features : - aggregates and user-facing AZs allow to define a common set of capabilities for physical hosts upon which you can boot VMs - ServerGroups with Affinity/Anti-Affinity filters allow you to enforce a certain level of network proximity for VMs Once that said, there is also another effort of forking the Nova Scheduler code into a separate project so that cross-projects scheduling could happen (and consequently Neutron could use it). This project is planned to be delivered by K release, and will be called Gantt. So, could you please mention which features do you need for Nova, so we could discuss that here before issuing a spec ? -Sylvain *From:* Carl Baldwin [mailto:c...@ecbaldwin.net c...@ecbaldwin.net] *Sent:* Friday, May 30, 2014 12:05 AM *To:* A, Keshava *Cc:* jcsf31...@gmail.com; Armando M.; OpenStack Development Mailing List (not for usage questions); Kyle Mestery *Subject:* Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ? Keshava, How much of a problem is routing prefix fragmentation for you? Fragmentation causes routing table bloat and may reduce the performance of the routing table. It also increases the amount of information traded by the routing protocol. Which aspect(s) is (are) affecting you? Can you quantify this effect? A major motivation for my interest in employing a dynamic routing protocol within a datacenter is to enable IP mobility so that I don't need to worry about doing things like scheduling instances based on their IP addresses. Also, I believe that it can make floating ips more floaty so that they can cross network boundaries without having to statically configure routers. To get this mobility, it seems inevitable to accept the fragmentation in the routing prefixes. This level of fragmentation would be contained to a well-defined scope, like within a datacenter. Is it your opinion that trading off fragmentation for mobility a bad trade-off? Maybe it depends on the capabilities of the TOR switches and routers that you have. Maybe others can chime in here. Carl On Wed, May 28, 2014 at 10:11 PM, A, Keshava keshav...@hp.com wrote: Hi, Motivation behind this requirement is “ to achieve VM prefix aggregation using routing protocol ( BGP/OSPF)”. So that prefix advertised from cloud to upstream will be aggregated. I do not have idea how the current scheduler is implemented. But schedule
Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ?
Mathieu Rohon wrote: I'm also very interested in scheduling VMs with Network requirement. This seems to be in the scope of NFV workgroup [1]. For instance, I think that scheduling should take into account bandwith/QoS requirement for a VM, or specific Nic This falls in my area of interest as well. We’re working on making network quality of service guarantees by means of a combination of DSCP marking with a reservation database and separate hardware queues in physical network switches in order to ensure that the reservations granted don’t exceed the wire speed of the switches. Right now the only option if the total of requested reservations would exceed the wire speed of the switches is to deny reservations on the basis of “first come, first served” and “last come, doesn’t get served”, in other words simply issuing a failure response at reservation time to any tenants who attempt to make reservations after a particular switch port is maxed out (from a reservation perspective, not necessarily maxed out from an actual utilization perspective at any given moment.) However, with the element of chance in VM scheduling to compute node, it’s possible that a tenant could get a deny response from the reservation server because their VM landed on a particularly reservation heavy rack. If their VM happened to land on a compute node in a different rack then there might well be plenty of excess bandwidth on that rack’s uplink. But our current implementation has no way to tell Nova or the tenant that a reservation that was denied could have been granted if the VM were relocated to a less network overloaded rack. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ?
Le 30/05/2014 14:44, CARVER, PAUL a écrit : Mathieu Rohon wrote: I'm also very interested in scheduling VMs with Network requirement. This seems to be in the scope of NFV workgroup [1]. For instance, I think that scheduling should take into account bandwith/QoS requirement for a VM, or specific Nic This falls in my area of interest as well. We're working on making network quality of service guarantees by means of a combination of DSCP marking with a reservation database and separate hardware queues in physical network switches in order to ensure that the reservations granted don't exceed the wire speed of the switches. Right now the only option if the total of requested reservations would exceed the wire speed of the switches is to deny reservations on the basis of first come, first served and last come, doesn't get served, in other words simply issuing a failure response at reservation time to any tenants who attempt to make reservations after a particular switch port is maxed out (from a reservation perspective, not necessarily maxed out from an actual utilization perspective at any given moment.) Hi Paul, I don't exactly know your needs, neither your current state of work regarding the implementation of a reservation database, but please note that there is a Stackforge project aiming to provide this for OpenStack natively : http://wiki.openstack.org/wiki/Blazar (formerly Climate). During last Juno summit, some discussions in Nova were related to having a reservation system in place, and the agreement was to take a look at Climate/Blazar to see if it's a good fit. I think it would be valuable for both you and Climate folks (whose I'm in as core reviewer) to see if we can join efforts to address your requirements with Blazar so you wouldn't have to do all the things. There is a weekly meeting today at 3pm UTC in #openstack-meeting for Blazar team. If you have time, just jump in and address your questions here, that would be a good starting point. However, with the element of chance in VM scheduling to compute node, it's possible that a tenant could get a deny response from the reservation server because their VM landed on a particularly reservation heavy rack. If their VM happened to land on a compute node in a different rack then there might well be plenty of excess bandwidth on that rack's uplink. But our current implementation has no way to tell Nova or the tenant that a reservation that was denied could have been granted if the VM were relocated to a less network overloaded rack. IMHO, this strategy requires both a reservation system (in my opinion, Blazar), a resource placement system (Gantt) and possibly some AMQP notifications which would go thru the queue. That's something I also have in my scope, we can discuss that whenever you want. -Sylvain ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ?
Keshava, How much of a problem is routing prefix fragmentation for you? Fragmentation causes routing table bloat and may reduce the performance of the routing table. It also increases the amount of information traded by the routing protocol. Which aspect(s) is (are) affecting you? Can you quantify this effect? A major motivation for my interest in employing a dynamic routing protocol within a datacenter is to enable IP mobility so that I don't need to worry about doing things like scheduling instances based on their IP addresses. Also, I believe that it can make floating ips more floaty so that they can cross network boundaries without having to statically configure routers. To get this mobility, it seems inevitable to accept the fragmentation in the routing prefixes. This level of fragmentation would be contained to a well-defined scope, like within a datacenter. Is it your opinion that trading off fragmentation for mobility a bad trade-off? Maybe it depends on the capabilities of the TOR switches and routers that you have. Maybe others can chime in here. Carl On Wed, May 28, 2014 at 10:11 PM, A, Keshava keshav...@hp.com wrote: Hi, Motivation behind this requirement is “ to achieve VM prefix aggregation using routing protocol ( BGP/OSPF)”. So that prefix advertised from cloud to upstream will be aggregated. I do not have idea how the current scheduler is implemented. But schedule to maintain some kind of the ‘Network to Node mapping to VM” .. Based on that mapping to if any new VM getting hosted to give prefix in those Nodes based one input preference. It will be great help us from routing side if this is available in the infrastructure. I am available for review/technical discussion/meeting. Thanks regards, Keshava.A *From:* jcsf31...@gmail.com [mailto:jcsf31...@gmail.com] *Sent:* Thursday, May 29, 2014 9:14 AM *To:* openstack-dev@lists.openstack.org; Carl Baldwin; Kyle Mestery; OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ? Hi keshava, This is an area that I am interested in. I'd be happy to collaborate with you on a blueprint.This would require enhancements to the scheduler as you suggested. There are a number of uses cases for this. John. Sent from my smartphone. *From: *A, Keshava *Sent: *Tuesday, May 27, 2014 10:58 AM *To: *Carl Baldwin; Kyle Mestery; OpenStack Development Mailing List (not for usage questions) *Reply To: *OpenStack Development Mailing List (not for usage questions) *Subject: *[openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ? Hi, I have one of the basic question about the Nova Scheduler in the following below scenario. Whenever a new VM to be hosted is there any consideration of network attributes ? Example let us say all the VMs with 10.1.x is under TOR-1, and 20.1.xy are under TOR-2. A new CN nodes is inserted under TOR-2 and at same time a new tenant VM needs to be hosted for 10.1.xa network. Then is it possible to mandate the new VM(10.1.xa) to hosted under TOR-1 instead of it got scheduled under TOR-2 ( where there CN-23 is completely free from resource perspective ) ? This is required to achieve prefix/route aggregation and to avoid network broadcast (incase if they are scattered across different TOR/Switch) ? Thanks regards, Keshava.A ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ?
Hi Keshava, To the best of my knowledge Nova does not have an explicit way to determine VM placements based on network attributes. That said, it does have a general mechanism called host-aggregates [1] that can be leveraged to address what you are looking for. How certain hosts are grouped together to match certain network affinity rules is in the hands of the cloud operator and I believe this requires quite a bit of out-of-band management. I recall at one point that there was an effort going on to improve the usability of such a use case (using the port binding extension [2]), but my knowledge is not very current, so I'd need to fall back on some other folks listening on the ML to chime in on the latter topic. Hope this help! Armando [1] - http://docs.openstack.org/trunk/openstack-ops/content/scaling.html#segregate_cloud [2] - http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html On 27 May 2014 09:53, A, Keshava keshav...@hp.com wrote: Hi, I have one of the basic question about the Nova Scheduler in the following below scenario. Whenever a new VM to be hosted is there any consideration of network attributes ? Example let us say all the VMs with 10.1.x is under TOR-1, and 20.1.xy are under TOR-2. A new CN nodes is inserted under TOR-2 and at same time a new tenant VM needs to be hosted for 10.1.xa network. Then is it possible to mandate the new VM(10.1.xa) to hosted under TOR-1 instead of it got scheduled under TOR-2 ( where there CN-23 is completely free from resource perspective ) ? This is required to achieve prefix/route aggregation and to avoid network broadcast (incase if they are scattered across different TOR/Switch) ? Thanks regards, Keshava.A ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ?
Armando M. arma...@gmail.com wrote on 05/28/2014 07:35:52 PM: From: Armando M. arma...@gmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org, Date: 05/28/2014 07:37 PM Subject: Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ? Hi Keshava, To the best of my knowledge Nova does not have an explicit way to determine VM placements based on network attributes. That said, it does have a general mechanism called host-aggregates [1] that can be leveraged to address what you are looking for. How certain hosts are grouped together to match certain network affinity rules is in the hands of the cloud operator and I believe this requires quite a bit of out-of-band management. I recall at one point that there was an effort going on to improve the usability of such a use case (using the port binding extension [2]), but my knowledge is not very current, so I'd need to fall back on some other folks listening on the ML to chime in on the latter topic. Hope this help! Armando [1] - http://docs.openstack.org/trunk/openstack-ops/content/ scaling.html#segregate_cloud [2] - http://docs.openstack.org/api/openstack-network/2.0/content/ binding_ext_ports.html On 27 May 2014 09:53, A, Keshava keshav...@hp.com wrote: Hi, I have one of the basic question about the Nova Scheduler in the following below scenario. Whenever a new VM to be hosted is there any consideration of network attributes ? Example let us say all the VMs with 10.1.x is under TOR-1, and 20. 1.xy are under TOR-2. A new CN nodes is inserted under TOR-2 and at same time a new tenant VM needs to be hosted for 10.1.xa network. Then is it possible to mandate the new VM(10.1.xa) to hosted under TOR-1 instead of it got scheduled under TOR-2 ( where there CN-23 is completely free from resource perspective ) ? This is required to achieve prefix/route aggregation and to avoid network broadcast (incase if they are scattered across different TOR/Switch) ? The idea of taking cross-service relationships into account when scheduling has been discussed off-and-on for a while (longer than I have been involved in OpenStack), but it is not well realized today. I brought a proposal (see https://wiki.openstack.org/wiki/Heat/PolicyExtension) to the Icehouse summit. In that model, your use case would be handled by relationships that limit network hop counts between VMs. Work is underway towards this vision, but the vision remains a long way off. The current work is called Gantt, it is a preliminary step of factoring the scheduling out of nova in preparation for sharing and generalization. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ?
Yes, agree on the use case. I suggest we introduce a series of callbacks and up calls to the scheduler so that we don't bloat the scheduler with information outside of its domain. Callbacks and up calls can be used to call external systems that keep detailed topology information. We can also work on the latter together if you wish. We have a topology system already. I've been waiting for someone to work with this on. John Sent from my BlackBerry 10 smartphone. From: A, KeshavaSent: Thursday, May 29, 2014 7:13 AMTo: jcsf31...@gmail.comCc: Armando M.; OpenStack Development Mailing List (not for usage questions); Carl Baldwin; Kyle MesterySubject: RE: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ? Hi, Motivation behind this requirement is “ to achieve VM prefix aggregation using routing protocol ( BGP/OSPF)”. So that prefix advertised from cloud to upstream will be aggregated. I do not have idea how the current scheduler is implemented. But schedule to maintain some kind of the ‘Network to Node mapping to VM” .. Based on that mapping to if any new VM getting hosted to give prefix in those Nodes based one input preference. It will be great help us from routing side if this is available in the infrastructure. I am available for review/technical discussion/meeting. Thanks regards, Keshava.A From: jcsf31...@gmail.com [mailto:jcsf31...@gmail.com] Sent: Thursday, May 29, 2014 9:14 AM To: openstack-dev@lists.openstack.org; Carl Baldwin; Kyle Mestery; OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ? Hi keshava, This is an area that I am interested in. I'd be happy to collaborate with you on a blueprint. This would require enhancements to the scheduler as you suggested. There are a number of uses cases for this. John. Sent from my smartphone. From: A, Keshava Sent: Tuesday, May 27, 2014 10:58 AM To: Carl Baldwin; Kyle Mestery; OpenStack Development Mailing List (not for usage questions) Reply To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [neutron][L3] VM Scheduling v/s Network as input any consideration ? Hi, I have one of the basic question about the Nova Scheduler in the following below scenario. Whenever a new VM to be hosted is there any consideration of network attributes ? Example let us say all the VMs with 10.1.x is under TOR-1, and 20.1.xy are under TOR-2. A new CN nodes is inserted under TOR-2 and at same time a new tenant VM needs to be hosted for 10.1.xa network. Then is it possible to mandate the new VM(10.1.xa) to hosted under TOR-1 instead of it got scheduled under TOR-2 ( where there CN-23 is completely free from resource perspective ) ? This is required to achieve prefix/route aggregation and to avoid network broadcast (incase if they are scattered across different TOR/Switch) ? Thanks regards, Keshava.A ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
