Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
Thanks Kris and Sam for your replies! On 18/06/15 01:20, Kris G. Lindgren wrote: On 6/17/15, 10:59 AM, Neil Jerram neil.jer...@metaswitch.com wrote: On 17/06/15 16:17, Kris G. Lindgren wrote: See inline. Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC. On 6/17/15, 5:12 AM, Neil Jerram neil.jer...@metaswitch.com wrote: Hi Kris, Apologies in advance for questions that are probably really dumb - but there are several points here that I don't understand. On 17/06/15 03:44, Kris G. Lindgren wrote: We are doing pretty much the same thing - but in a slightly different way. We extended the nova scheduler to help choose networks (IE. don't put vm's on a network/host that doesn't have any available IP address). Why would a particular network/host not have any available IP address? If a created network has 1024 ip's on it (/22) and we provision 1020 vms, anything deployed after that will not have an additional ip address because the network doesn't have any available ip addresses (loose some ip's to the network). OK, thanks, that certainly explains the particular network possibility. So I guess this applies where your preference would be for network A, but it would be OK to fall back to network B, and so on. That sounds like it could be a useful general enhancement. (But, if a new VM absolutely _has_ to be on, say, the 'production' network, and the 'production' network is already fully used, you're fundamentally stuck, aren't you?) Yes - this would be a scheduling failure - and I am ok with that. It does no good to have a vm on a network that doesn't work. What about the /host part? Is it possible in your system for a network to have IP addresses available, but for them not to be usable on a particular host? Yes this is also a possibility. That the network allocated to a set of hosts has IP's available but no compute capacity to spin up vms on it. Again - I am ok with this. Then, we add into the host-aggregate that each HV is attached to a network metadata item which maps to the names of the neutron networks that host supports. This basically creates the mapping of which host supports what networks, so we can correctly filter hosts out during scheduling. We do allow people to choose a network if they wish and we do have the neutron end-point exposed. However, by default if they do not supply a boot command with a network, we will filter the networks down and choose one for them. That way they never hit [1]. This also works well for us, because the default UI that we provide our end-users is not horizon. Why do you define multiple networks - as opposed to just one - and why would one of your users want to choose a particular one of those? (Do you mean multiple as in public-1, public-2, ...; or multiple as in public, service, ...?) This is answered in the other email and original email as well. But basically we have multiple L2 segments that only exists on certain switches and thus are only tied to certain hosts. With the way neutron is currently structured we need to create a network for each L2. So that¹s why we define multiple networks. Thanks! Ok, just to check that I really understand this: - You have real L2 segments connecting some of your compute hosts together - and also I guess to a ToR that does L3 to the rest of the data center. Correct. - You presumably then just bridge all the TAP interfaces, on each host, to the host's outwards-facing interface. + VM | +- Host + VM | | | + VM | | + VM | | +- Host + VM | | ToR ---+ + VM | | + VM | | |- Host + VM | + VM Also correct, we are using flat provider networks (shared=true) - however provider vlan networks would work as well. - You specify each such setup as a network in the Neutron API - and hence you have multiple similar networks, for your data center as a whole. Out of interest, do you do this just because it's the Right Thing according to the current Neutron API - i.e. because a Neutron network is L2 - or also because it's needed in order to get the Neutron implementation components that you use to work correctly? For example, so that you have a DHCP agent for each L2 network (if you use the Neutron DHCP agent). Somewhat both. It was a how do I get neutron to handle this without making drastic changes to the base level neutron concepts. We currently do have dhcp-agents and nova-metadata agent running in each L2 and we specifically assign them to hosts in that L2 space. Thanks. We are currently working on ways to remove this requirement. That sounds intriguing.
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
On 17 Jun 2015, at 8:35 pm, Neil Jerram neil.jer...@metaswitch.com wrote: Hi Sam, On 17/06/15 01:31, Sam Morrison wrote: We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. This begs the question: why have you defined 10 public-N networks, instead of just one public network? I think this has all been answered but just in case. There are multiple reasons. We don’t have a single IPv4 range big enough for our cloud, don’t want the broadcast domain too be massive, the compute nodes are in different data centres etc. etc. Basically it’s not how our underlying physical network is set up and we can’t change that. Sam I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. Why the 10 subnets? Is it to do with where you actually have real L2 segments, in your deployment? Thanks, Neil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
On 18 Jun 2015, at 2:59 am, Neil Jerram neil.jer...@metaswitch.com wrote: On 17/06/15 16:17, Kris G. Lindgren wrote: See inline. Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC. On 6/17/15, 5:12 AM, Neil Jerram neil.jer...@metaswitch.com wrote: Hi Kris, Apologies in advance for questions that are probably really dumb - but there are several points here that I don't understand. On 17/06/15 03:44, Kris G. Lindgren wrote: We are doing pretty much the same thing - but in a slightly different way. We extended the nova scheduler to help choose networks (IE. don't put vm's on a network/host that doesn't have any available IP address). Why would a particular network/host not have any available IP address? If a created network has 1024 ip's on it (/22) and we provision 1020 vms, anything deployed after that will not have an additional ip address because the network doesn't have any available ip addresses (loose some ip's to the network). OK, thanks, that certainly explains the particular network possibility. So I guess this applies where your preference would be for network A, but it would be OK to fall back to network B, and so on. That sounds like it could be a useful general enhancement. (But, if a new VM absolutely _has_ to be on, say, the 'production' network, and the 'production' network is already fully used, you're fundamentally stuck, aren't you?) What about the /host part? Is it possible in your system for a network to have IP addresses available, but for them not to be usable on a particular host? Then, we add into the host-aggregate that each HV is attached to a network metadata item which maps to the names of the neutron networks that host supports. This basically creates the mapping of which host supports what networks, so we can correctly filter hosts out during scheduling. We do allow people to choose a network if they wish and we do have the neutron end-point exposed. However, by default if they do not supply a boot command with a network, we will filter the networks down and choose one for them. That way they never hit [1]. This also works well for us, because the default UI that we provide our end-users is not horizon. Why do you define multiple networks - as opposed to just one - and why would one of your users want to choose a particular one of those? (Do you mean multiple as in public-1, public-2, ...; or multiple as in public, service, ...?) This is answered in the other email and original email as well. But basically we have multiple L2 segments that only exists on certain switches and thus are only tied to certain hosts. With the way neutron is currently structured we need to create a network for each L2. So that¹s why we define multiple networks. Thanks! Ok, just to check that I really understand this: - You have real L2 segments connecting some of your compute hosts together - and also I guess to a ToR that does L3 to the rest of the data center. - You presumably then just bridge all the TAP interfaces, on each host, to the host's outwards-facing interface. + VM | +- Host + VM | | | + VM | | + VM | | +- Host + VM | | ToR ---+ + VM | | + VM | | |- Host + VM | + VM - You specify each such setup as a network in the Neutron API - and hence you have multiple similar networks, for your data center as a whole. Out of interest, do you do this just because it's the Right Thing according to the current Neutron API - i.e. because a Neutron network is L2 - or also because it's needed in order to get the Neutron implementation components that you use to work correctly? For example, so that you have a DHCP agent for each L2 network (if you use the Neutron DHCP agent). For our end users - they only care about getting a vm with a single ip address in a network which is really a zone like prod or dev or test. They stop caring after that point. So in the scheduler filter that we created we do exactly that. We will filter down from all the hosts and networks down to a combo that intersects at a host that has space, with a network that has space, And the network that was chosen is actually available to that host. Thanks, makes perfect sense now. So I think there are two possible representations, overall, of what you are looking for. 1. A 'network group' of similar L2 networks. When a VM is launched, tenant specifies the network group instead of a particular L2 network, and Nova/Neutron select a host and network with available
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
On 6/17/15, 10:59 AM, Neil Jerram neil.jer...@metaswitch.com wrote: On 17/06/15 16:17, Kris G. Lindgren wrote: See inline. Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC. On 6/17/15, 5:12 AM, Neil Jerram neil.jer...@metaswitch.com wrote: Hi Kris, Apologies in advance for questions that are probably really dumb - but there are several points here that I don't understand. On 17/06/15 03:44, Kris G. Lindgren wrote: We are doing pretty much the same thing - but in a slightly different way. We extended the nova scheduler to help choose networks (IE. don't put vm's on a network/host that doesn't have any available IP address). Why would a particular network/host not have any available IP address? If a created network has 1024 ip's on it (/22) and we provision 1020 vms, anything deployed after that will not have an additional ip address because the network doesn't have any available ip addresses (loose some ip's to the network). OK, thanks, that certainly explains the particular network possibility. So I guess this applies where your preference would be for network A, but it would be OK to fall back to network B, and so on. That sounds like it could be a useful general enhancement. (But, if a new VM absolutely _has_ to be on, say, the 'production' network, and the 'production' network is already fully used, you're fundamentally stuck, aren't you?) Yes - this would be a scheduling failure - and I am ok with that. It does no good to have a vm on a network that doesn't work. What about the /host part? Is it possible in your system for a network to have IP addresses available, but for them not to be usable on a particular host? Yes this is also a possibility. That the network allocated to a set of hosts has IP's available but no compute capacity to spin up vms on it. Again - I am ok with this. Then, we add into the host-aggregate that each HV is attached to a network metadata item which maps to the names of the neutron networks that host supports. This basically creates the mapping of which host supports what networks, so we can correctly filter hosts out during scheduling. We do allow people to choose a network if they wish and we do have the neutron end-point exposed. However, by default if they do not supply a boot command with a network, we will filter the networks down and choose one for them. That way they never hit [1]. This also works well for us, because the default UI that we provide our end-users is not horizon. Why do you define multiple networks - as opposed to just one - and why would one of your users want to choose a particular one of those? (Do you mean multiple as in public-1, public-2, ...; or multiple as in public, service, ...?) This is answered in the other email and original email as well. But basically we have multiple L2 segments that only exists on certain switches and thus are only tied to certain hosts. With the way neutron is currently structured we need to create a network for each L2. So that¹s why we define multiple networks. Thanks! Ok, just to check that I really understand this: - You have real L2 segments connecting some of your compute hosts together - and also I guess to a ToR that does L3 to the rest of the data center. Correct. - You presumably then just bridge all the TAP interfaces, on each host, to the host's outwards-facing interface. + VM | +- Host + VM | | | + VM | | + VM | | +- Host + VM | | ToR ---+ + VM | | + VM | | |- Host + VM | + VM Also correct, we are using flat provider networks (shared=true) - however provider vlan networks would work as well. - You specify each such setup as a network in the Neutron API - and hence you have multiple similar networks, for your data center as a whole. Out of interest, do you do this just because it's the Right Thing according to the current Neutron API - i.e. because a Neutron network is L2 - or also because it's needed in order to get the Neutron implementation components that you use to work correctly? For example, so that you have a DHCP agent for each L2 network (if you use the Neutron DHCP agent). Somewhat both. It was a how do I get neutron to handle this without making drastic changes to the base level neutron concepts. We currently do have dhcp-agents and nova-metadata agent running in each L2 and we specifically assign them to hosts in that L2 space. We are currently working on ways to remove this requirement. For our end users - they only care about getting a vm with a single ip address
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
Awesome! Thanks. :) Kevin From: Ihar Hrachyshka [ihrac...@redhat.com] Sent: Wednesday, June 17, 2015 9:41 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking? -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 06/17/2015 05:55 PM, Fox, Kevin M wrote: The biggest issue we have run into with multiple public networks is restricting which users can use which networks. We have the same issue, where we may have an internal public network for the datacenter, but also, say, a DMZ network we want to put some vm's on, but can't currently extend that network easily there because too many tenants will be able to launch vm's attached to the DMZ that don't have authorization. Quota's or acls or something on public networks are really needed. ...and that (acls for networks) is to be handled in Liberty with: https://blueprints.launchpad.net/neutron/+spec/rbac-networks Ihar -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJVgaNBAAoJEC5aWaUY1u57CN0IANEZh3t7da/7zpqNvgYMr10O mHwxx0HVTrchi4+xOuQm5Ibx+CtRdRS2rgGIKdjZVyuanYsbdFPDrJ32dFRU7EIJ 5oFtZvs5iev1jGpOs4jzMwAdfLN4XFmci1Vm+eNy0uatiiTOjt93RdArRMNQGQlI cJfLmzS88oG0nVoEthHd6YD4Lk8+mf2e64hHNAW8yz7ZTofHff2xRU4QdnMAOEDk aXU4R1L32zsI9lmC6ANutTzXkA+LWk14PqrA4zHzdwurDJKQQ0oq/M4jjsdG4JyX k83FvBk47ht40rV1s0kEnOQWVPR7NcAt6zDwu/l+a0uf5jTIJbl2SKE0GLwTAtY= =uzgm -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
Hi Kris, Apologies in advance for questions that are probably really dumb - but there are several points here that I don't understand. On 17/06/15 03:44, Kris G. Lindgren wrote: We are doing pretty much the same thing - but in a slightly different way. We extended the nova scheduler to help choose networks (IE. don't put vm's on a network/host that doesn't have any available IP address). Why would a particular network/host not have any available IP address? Then, we add into the host-aggregate that each HV is attached to a network metadata item which maps to the names of the neutron networks that host supports. This basically creates the mapping of which host supports what networks, so we can correctly filter hosts out during scheduling. We do allow people to choose a network if they wish and we do have the neutron end-point exposed. However, by default if they do not supply a boot command with a network, we will filter the networks down and choose one for them. That way they never hit [1]. This also works well for us, because the default UI that we provide our end-users is not horizon. Why do you define multiple networks - as opposed to just one - and why would one of your users want to choose a particular one of those? (Do you mean multiple as in public-1, public-2, ...; or multiple as in public, service, ...?) We currently only support one network per HV via this configuration, but we would like to be able to expose a network type or group via neutron in the future. I believe what you described below is also another way of phrasing the ask that we had in [2]. That you want to define multiple top level networks in neutron: 'public' and 'service'. That is made up by multiple desperate desperate? :-) I assume you probably meant separate here. L2 networks: 'public-1', 'public2,' ect which are independently constrained to a specific set of hosts/switches/datacenter. If I'm understanding correctly, this is one of those places where I get confused about the difference between Neutron-as-an-API and Neutron-as-a-software-implementation. I guess what you mean here is that your deployment hardware is really providing those L2 segments directly, and hence you aren't using Neutron's software-based simulation of L2 segments. Is that right? We have talked about working around this under our configuration one of two ways. First, is to use availability zones to provide the separation between: 'public' and 'service', or in our case: 'prod', 'pki','internal', ect, ect. Why are availability zones involved here? Assuming you had 'prod', 'pki','internal' etc. networks set up and represented as such in Neutron, why wouldn't you just say which of those networks each instance should connect to, when creating each instance? Regards, Neil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
Couple more dumb comments here - sorry that I'm processing this thread backwards! On 16/06/15 15:20, Jay Pipes wrote: Adding -dev because of the reference to the Neutron Get me a network spec. Also adding [nova] and [neutron] subject markers. Comments inline, Kris. On 05/22/2015 09:28 PM, Kris G. Lindgren wrote: During the Openstack summit this week I got to talk to a number of other operators of large Openstack deployments about how they do networking. I was happy, surprised even, to find that a number of us are using a similar type of networking strategy. That we have similar challenges around networking and are solving it in our own but very similar way. It is always nice to see that other people are doing the same things as you or see the same issues as you are and that you are not crazy. So in that vein, I wanted to reach out to the rest of the Ops Community and ask one pretty simple question. Would it be accurate to say that most of your end users want almost nothing to do with the network? That was my experience at ATT, yes. The vast majority of end users could not care less about networking, as long as the connectivity was reliable, performed well, and they could connect to the Internet (and have others connect from the Internet to their VMs) when needed. In my experience what the majority of them (both internal and external) want is to consume from Openstack a compute resource, a property of which is it that resource has an IP address. They, at most, care about which network they are on. Where a network is usually an arbitrary definition around a set of real networks, that are constrained to a location, in which the company has attached some sort of policy. For example, I want to be in the production network vs's the xyz lab network, vs's the backup network, vs's the corp network. I would say for Godaddy, 99% of our use cases would be defined as: I want a compute resource in the production network zone, or I want a compute resource in this other network zone. Kris - this looks like the answer to my question why you define multiple networks. If that's right, no need to answer that question there. The end user only cares that the IP the vm receives works in that zone, outside of that they don't care any other property of that IP. They do not care what subnet it is in, what vlan it is on, what switch it is attached to, what router its attached to, or how data flows in/out of that network. It just needs to work. Agreed. I'm not a deployer, but my team is in contact with many deployers who say similar things. Regards, Neil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
On 16 June 2015 at 22:36, Sam Morrison sorri...@gmail.com wrote: On 17 Jun 2015, at 10:56 am, Armando M. arma...@gmail.com wrote: On 16 June 2015 at 17:31, Sam Morrison sorri...@gmail.com wrote: We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. I’m not sure what the correct way of doing this. What are other people doing in the interim until this kind of use case can be done in Neutron? Would something like [1] be adequate to address your use case? If not, I'd suggest you to file an RFE bug (more details in [2]), so that we can keep the discussion focused on this specific case. HTH Armando [1] https://blueprints.launchpad.net/neutron/+spec/rbac-networks That’s not applicable in this case. We don’t care about what tenants are when in this case. [2] https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst#neutron-request-for-feature-enhancements The bug Kris mentioned outlines all I want too I think. I don't know what you're referring to. Sam Cheers, Sam [1] https://github.com/NeCTAR-RC/nova/commit/1bc2396edc684f83ce471dd9dc9219c4635afb12 On 17 Jun 2015, at 12:20 am, Jay Pipes jaypi...@gmail.com wrote: Adding -dev because of the reference to the Neutron Get me a network spec. Also adding [nova] and [neutron] subject markers. Comments inline, Kris. On 05/22/2015 09:28 PM, Kris G. Lindgren wrote: During the Openstack summit this week I got to talk to a number of other operators of large Openstack deployments about how they do networking. I was happy, surprised even, to find that a number of us are using a similar type of networking strategy. That we have similar challenges around networking and are solving it in our own but very similar way. It is always nice to see that other people are doing the same things as you or see the same issues as you are and that you are not crazy. So in that vein, I wanted to reach out to the rest of the Ops Community and ask one pretty simple question. Would it be accurate to say that most of your end users want almost nothing to do with the network? That was my experience at ATT, yes. The vast majority of end users could not care less about networking, as long as the connectivity was reliable, performed well, and they could connect to the Internet (and have others connect from the Internet to their VMs) when needed. In my experience what the majority of them (both internal and external) want is to consume from Openstack a compute resource, a property of which is it that resource has an IP address. They, at most, care about which network they are on. Where a network is usually an arbitrary definition around a set of real networks, that are constrained to a location, in which the company has attached some sort of policy. For example, I want to be in the production network vs's the xyz lab network, vs's the backup network, vs's the corp network. I would say for Godaddy, 99% of our use cases would be defined as: I want a compute resource in the production network zone, or I want a compute resource in this other network zone. The end user only cares that the IP the vm receives works in that zone, outside of that they don't care any other property of that IP. They do not care what subnet it is in, what vlan it is on, what switch it is attached to, what router its attached to, or how data flows in/out
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
Hi Sam, On 17/06/15 01:31, Sam Morrison wrote: We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. This begs the question: why have you defined 10 public-N networks, instead of just one public network? I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. Why the 10 subnets? Is it to do with where you actually have real L2 segments, in your deployment? Thanks, Neil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
[Sorry - unintentionally dropped -operators below; adding it back in this copy.] On 17/06/15 11:35, Neil Jerram wrote: Hi Sam, On 17/06/15 01:31, Sam Morrison wrote: We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. This begs the question: why have you defined 10 public-N networks, instead of just one public network? I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. Why the 10 subnets? Is it to do with where you actually have real L2 segments, in your deployment? Thanks, Neil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
On Wed, Jun 17, 2015 at 1:59 AM, Armando M. arma...@gmail.com wrote: On 16 June 2015 at 22:36, Sam Morrison sorri...@gmail.com wrote: On 17 Jun 2015, at 10:56 am, Armando M. arma...@gmail.com wrote: On 16 June 2015 at 17:31, Sam Morrison sorri...@gmail.com wrote: We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. I’m not sure what the correct way of doing this. What are other people doing in the interim until this kind of use case can be done in Neutron? Would something like [1] be adequate to address your use case? If not, I'd suggest you to file an RFE bug (more details in [2]), so that we can keep the discussion focused on this specific case. HTH Armando [1] https://blueprints.launchpad.net/neutron/+spec/rbac-networks That’s not applicable in this case. We don’t care about what tenants are when in this case. [2] https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst#neutron-request-for-feature-enhancements The bug Kris mentioned outlines all I want too I think. I don't know what you're referring to. Armando, I think this is the bug he's referring to: https://bugs.launchpad.net/neutron/+bug/1458890 This is something I'd like to look at next week during the mid-cycle, especially since Carl is there and his spec for routed networks [2] covers a lot of these use cases. [2] https://review.openstack.org/#/c/172244/ Sam Cheers, Sam [1] https://github.com/NeCTAR-RC/nova/commit/1bc2396edc684f83ce471dd9dc9219c4635afb12 On 17 Jun 2015, at 12:20 am, Jay Pipes jaypi...@gmail.com wrote: Adding -dev because of the reference to the Neutron Get me a network spec. Also adding [nova] and [neutron] subject markers. Comments inline, Kris. On 05/22/2015 09:28 PM, Kris G. Lindgren wrote: During the Openstack summit this week I got to talk to a number of other operators of large Openstack deployments about how they do networking. I was happy, surprised even, to find that a number of us are using a similar type of networking strategy. That we have similar challenges around networking and are solving it in our own but very similar way. It is always nice to see that other people are doing the same things as you or see the same issues as you are and that you are not crazy. So in that vein, I wanted to reach out to the rest of the Ops Community and ask one pretty simple question. Would it be accurate to say that most of your end users want almost nothing to do with the network? That was my experience at ATT, yes. The vast majority of end users could not care less about networking, as long as the connectivity was reliable, performed well, and they could connect to the Internet (and have others connect from the Internet to their VMs) when needed. In my experience what the majority of them (both internal and external) want is to consume from Openstack a compute resource, a property of which is it that resource has an IP address. They, at most, care about which network they are on. Where a network is usually an arbitrary definition around a set of real networks, that are constrained to a location, in which the company has attached some sort of policy. For example, I want to be in the production network vs's the xyz lab network, vs's the backup network, vs's the corp network. I would say for Godaddy, 99% of our use cases would be defined as: I want a
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
While I didn't know about the Neutron mid-cycle being next week. I do happen to live in Fort Collins, so I could easy become available if you want to talk face-to-face about https://bugs.launchpad.net/neutron/+bug/1458890. Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC. From: Kyle Mestery mest...@mestery.commailto:mest...@mestery.com Date: Wednesday, June 17, 2015 at 7:08 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Cc: openstack-operat...@lists.openstack.orgmailto:openstack-operat...@lists.openstack.org openstack-operat...@lists.openstack.orgmailto:openstack-operat...@lists.openstack.org Subject: Re: [Openstack-operators] [openstack-dev] [nova] [neutron] Re: How do your end users use networking? On Wed, Jun 17, 2015 at 1:59 AM, Armando M. arma...@gmail.commailto:arma...@gmail.com wrote: On 16 June 2015 at 22:36, Sam Morrison sorri...@gmail.commailto:sorri...@gmail.com wrote: On 17 Jun 2015, at 10:56 am, Armando M. arma...@gmail.commailto:arma...@gmail.com wrote: On 16 June 2015 at 17:31, Sam Morrison sorri...@gmail.commailto:sorri...@gmail.com wrote: We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. I’m not sure what the correct way of doing this. What are other people doing in the interim until this kind of use case can be done in Neutron? Would something like [1] be adequate to address your use case? If not, I'd suggest you to file an RFE bug (more details in [2]), so that we can keep the discussion focused on this specific case. HTH Armando [1] https://blueprints.launchpad.net/neutron/+spec/rbac-networks That’s not applicable in this case. We don’t care about what tenants are when in this case. [2] https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst#neutron-request-for-feature-enhancements The bug Kris mentioned outlines all I want too I think. I don't know what you're referring to. Armando, I think this is the bug he's referring to: https://bugs.launchpad.net/neutron/+bug/1458890 This is something I'd like to look at next week during the mid-cycle, especially since Carl is there and his spec for routed networks [2] covers a lot of these use cases. [2] https://review.openstack.org/#/c/172244/ Sam Cheers, Sam [1] https://github.com/NeCTAR-RC/nova/commit/1bc2396edc684f83ce471dd9dc9219c4635afb12 On 17 Jun 2015, at 12:20 am, Jay Pipes jaypi...@gmail.commailto:jaypi...@gmail.com wrote: Adding -dev because of the reference to the Neutron Get me a network spec. Also adding [nova] and [neutron] subject markers. Comments inline, Kris. On 05/22/2015 09:28 PM, Kris G. Lindgren wrote: During the Openstack summit this week I got to talk to a number of other operators of large Openstack deployments about how they do networking. I was happy, surprised even, to find that a number of us are using a similar type of networking strategy. That we have similar challenges around networking and are solving it in our own but very similar way. It is always nice to see that other people are doing the same things as you or see the same issues as you are and that you are not crazy. So in that vein, I wanted to reach out to the rest of the Ops Community and ask one pretty simple question. Would it be accurate to say that most of your end users want almost
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 06/17/2015 05:55 PM, Fox, Kevin M wrote: The biggest issue we have run into with multiple public networks is restricting which users can use which networks. We have the same issue, where we may have an internal public network for the datacenter, but also, say, a DMZ network we want to put some vm's on, but can't currently extend that network easily there because too many tenants will be able to launch vm's attached to the DMZ that don't have authorization. Quota's or acls or something on public networks are really needed. ...and that (acls for networks) is to be handled in Liberty with: https://blueprints.launchpad.net/neutron/+spec/rbac-networks Ihar -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJVgaNBAAoJEC5aWaUY1u57CN0IANEZh3t7da/7zpqNvgYMr10O mHwxx0HVTrchi4+xOuQm5Ibx+CtRdRS2rgGIKdjZVyuanYsbdFPDrJ32dFRU7EIJ 5oFtZvs5iev1jGpOs4jzMwAdfLN4XFmci1Vm+eNy0uatiiTOjt93RdArRMNQGQlI cJfLmzS88oG0nVoEthHd6YD4Lk8+mf2e64hHNAW8yz7ZTofHff2xRU4QdnMAOEDk aXU4R1L32zsI9lmC6ANutTzXkA+LWk14PqrA4zHzdwurDJKQQ0oq/M4jjsdG4JyX k83FvBk47ht40rV1s0kEnOQWVPR7NcAt6zDwu/l+a0uf5jTIJbl2SKE0GLwTAtY= =uzgm -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
On 17/06/15 16:17, Kris G. Lindgren wrote: See inline. Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC. On 6/17/15, 5:12 AM, Neil Jerram neil.jer...@metaswitch.com wrote: Hi Kris, Apologies in advance for questions that are probably really dumb - but there are several points here that I don't understand. On 17/06/15 03:44, Kris G. Lindgren wrote: We are doing pretty much the same thing - but in a slightly different way. We extended the nova scheduler to help choose networks (IE. don't put vm's on a network/host that doesn't have any available IP address). Why would a particular network/host not have any available IP address? If a created network has 1024 ip's on it (/22) and we provision 1020 vms, anything deployed after that will not have an additional ip address because the network doesn't have any available ip addresses (loose some ip's to the network). OK, thanks, that certainly explains the particular network possibility. So I guess this applies where your preference would be for network A, but it would be OK to fall back to network B, and so on. That sounds like it could be a useful general enhancement. (But, if a new VM absolutely _has_ to be on, say, the 'production' network, and the 'production' network is already fully used, you're fundamentally stuck, aren't you?) What about the /host part? Is it possible in your system for a network to have IP addresses available, but for them not to be usable on a particular host? Then, we add into the host-aggregate that each HV is attached to a network metadata item which maps to the names of the neutron networks that host supports. This basically creates the mapping of which host supports what networks, so we can correctly filter hosts out during scheduling. We do allow people to choose a network if they wish and we do have the neutron end-point exposed. However, by default if they do not supply a boot command with a network, we will filter the networks down and choose one for them. That way they never hit [1]. This also works well for us, because the default UI that we provide our end-users is not horizon. Why do you define multiple networks - as opposed to just one - and why would one of your users want to choose a particular one of those? (Do you mean multiple as in public-1, public-2, ...; or multiple as in public, service, ...?) This is answered in the other email and original email as well. But basically we have multiple L2 segments that only exists on certain switches and thus are only tied to certain hosts. With the way neutron is currently structured we need to create a network for each L2. So that¹s why we define multiple networks. Thanks! Ok, just to check that I really understand this: - You have real L2 segments connecting some of your compute hosts together - and also I guess to a ToR that does L3 to the rest of the data center. - You presumably then just bridge all the TAP interfaces, on each host, to the host's outwards-facing interface. + VM | +- Host + VM | | | + VM | | + VM | | +- Host + VM | | ToR ---+ + VM | | + VM | | |- Host + VM | + VM - You specify each such setup as a network in the Neutron API - and hence you have multiple similar networks, for your data center as a whole. Out of interest, do you do this just because it's the Right Thing according to the current Neutron API - i.e. because a Neutron network is L2 - or also because it's needed in order to get the Neutron implementation components that you use to work correctly? For example, so that you have a DHCP agent for each L2 network (if you use the Neutron DHCP agent). For our end users - they only care about getting a vm with a single ip address in a network which is really a zone like prod or dev or test. They stop caring after that point. So in the scheduler filter that we created we do exactly that. We will filter down from all the hosts and networks down to a combo that intersects at a host that has space, with a network that has space, And the network that was chosen is actually available to that host. Thanks, makes perfect sense now. So I think there are two possible representations, overall, of what you are looking for. 1. A 'network group' of similar L2 networks. When a VM is launched, tenant specifies the network group instead of a particular L2 network, and Nova/Neutron select a host and network with available compute power and IP addressing. This sounds like what you've described above. 2. A new kind of network whose ports are partitioned into various L2
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
Great! I'll reach out to you in unicast mode on this Kris, thanks! On Wed, Jun 17, 2015 at 10:27 AM, Kris G. Lindgren klindg...@godaddy.com wrote: While I didn't know about the Neutron mid-cycle being next week. I do happen to live in Fort Collins, so I could easy become available if you want to talk face-to-face about https://bugs.launchpad.net/neutron/+bug/1458890. Kris Lindgren Senior Linux Systems Engineer GoDaddy, LLC. From: Kyle Mestery mest...@mestery.com Date: Wednesday, June 17, 2015 at 7:08 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Cc: openstack-operat...@lists.openstack.org openstack-operat...@lists.openstack.org Subject: Re: [Openstack-operators] [openstack-dev] [nova] [neutron] Re: How do your end users use networking? On Wed, Jun 17, 2015 at 1:59 AM, Armando M. arma...@gmail.com wrote: On 16 June 2015 at 22:36, Sam Morrison sorri...@gmail.com wrote: On 17 Jun 2015, at 10:56 am, Armando M. arma...@gmail.com wrote: On 16 June 2015 at 17:31, Sam Morrison sorri...@gmail.com wrote: We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. I’m not sure what the correct way of doing this. What are other people doing in the interim until this kind of use case can be done in Neutron? Would something like [1] be adequate to address your use case? If not, I'd suggest you to file an RFE bug (more details in [2]), so that we can keep the discussion focused on this specific case. HTH Armando [1] https://blueprints.launchpad.net/neutron/+spec/rbac-networks That’s not applicable in this case. We don’t care about what tenants are when in this case. [2] https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst#neutron-request-for-feature-enhancements The bug Kris mentioned outlines all I want too I think. I don't know what you're referring to. Armando, I think this is the bug he's referring to: https://bugs.launchpad.net/neutron/+bug/1458890 This is something I'd like to look at next week during the mid-cycle, especially since Carl is there and his spec for routed networks [2] covers a lot of these use cases. [2] https://review.openstack.org/#/c/172244/ Sam Cheers, Sam [1] https://github.com/NeCTAR-RC/nova/commit/1bc2396edc684f83ce471dd9dc9219c4635afb12 On 17 Jun 2015, at 12:20 am, Jay Pipes jaypi...@gmail.com wrote: Adding -dev because of the reference to the Neutron Get me a network spec. Also adding [nova] and [neutron] subject markers. Comments inline, Kris. On 05/22/2015 09:28 PM, Kris G. Lindgren wrote: During the Openstack summit this week I got to talk to a number of other operators of large Openstack deployments about how they do networking. I was happy, surprised even, to find that a number of us are using a similar type of networking strategy. That we have similar challenges around networking and are solving it in our own but very similar way. It is always nice to see that other people are doing the same things as you or see the same issues as you are and that you are not crazy. So in that vein, I wanted to reach out to the rest of the Ops Community and ask one pretty simple question. Would it be accurate to say that most of your end users want almost nothing to do with the network? That was my
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
The biggest issue we have run into with multiple public networks is restricting which users can use which networks. We have the same issue, where we may have an internal public network for the datacenter, but also, say, a DMZ network we want to put some vm's on, but can't currently extend that network easily there because too many tenants will be able to launch vm's attached to the DMZ that don't have authorization. Quota's or acls or something on public networks are really needed. Thanks, Kevin From: Neil Jerram [neil.jer...@metaswitch.com] Sent: Wednesday, June 17, 2015 4:28 AM To: Jay Pipes; openstack-operat...@lists.openstack.org; openstack-dev@lists.openstack.org; Kyle Mestery Subject: Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking? Couple more dumb comments here - sorry that I'm processing this thread backwards! On 16/06/15 15:20, Jay Pipes wrote: Adding -dev because of the reference to the Neutron Get me a network spec. Also adding [nova] and [neutron] subject markers. Comments inline, Kris. On 05/22/2015 09:28 PM, Kris G. Lindgren wrote: During the Openstack summit this week I got to talk to a number of other operators of large Openstack deployments about how they do networking. I was happy, surprised even, to find that a number of us are using a similar type of networking strategy. That we have similar challenges around networking and are solving it in our own but very similar way. It is always nice to see that other people are doing the same things as you or see the same issues as you are and that you are not crazy. So in that vein, I wanted to reach out to the rest of the Ops Community and ask one pretty simple question. Would it be accurate to say that most of your end users want almost nothing to do with the network? That was my experience at ATT, yes. The vast majority of end users could not care less about networking, as long as the connectivity was reliable, performed well, and they could connect to the Internet (and have others connect from the Internet to their VMs) when needed. In my experience what the majority of them (both internal and external) want is to consume from Openstack a compute resource, a property of which is it that resource has an IP address. They, at most, care about which network they are on. Where a network is usually an arbitrary definition around a set of real networks, that are constrained to a location, in which the company has attached some sort of policy. For example, I want to be in the production network vs's the xyz lab network, vs's the backup network, vs's the corp network. I would say for Godaddy, 99% of our use cases would be defined as: I want a compute resource in the production network zone, or I want a compute resource in this other network zone. Kris - this looks like the answer to my question why you define multiple networks. If that's right, no need to answer that question there. The end user only cares that the IP the vm receives works in that zone, outside of that they don't care any other property of that IP. They do not care what subnet it is in, what vlan it is on, what switch it is attached to, what router its attached to, or how data flows in/out of that network. It just needs to work. Agreed. I'm not a deployer, but my team is in contact with many deployers who say similar things. Regards, Neil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
On 16 June 2015 at 17:31, Sam Morrison sorri...@gmail.com wrote: We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. I’m not sure what the correct way of doing this. What are other people doing in the interim until this kind of use case can be done in Neutron? Would something like [1] be adequate to address your use case? If not, I'd suggest you to file an RFE bug (more details in [2]), so that we can keep the discussion focused on this specific case. HTH Armando [1] https://blueprints.launchpad.net/neutron/+spec/rbac-networks [2] https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst#neutron-request-for-feature-enhancements Cheers, Sam [1] https://github.com/NeCTAR-RC/nova/commit/1bc2396edc684f83ce471dd9dc9219c4635afb12 On 17 Jun 2015, at 12:20 am, Jay Pipes jaypi...@gmail.com wrote: Adding -dev because of the reference to the Neutron Get me a network spec. Also adding [nova] and [neutron] subject markers. Comments inline, Kris. On 05/22/2015 09:28 PM, Kris G. Lindgren wrote: During the Openstack summit this week I got to talk to a number of other operators of large Openstack deployments about how they do networking. I was happy, surprised even, to find that a number of us are using a similar type of networking strategy. That we have similar challenges around networking and are solving it in our own but very similar way. It is always nice to see that other people are doing the same things as you or see the same issues as you are and that you are not crazy. So in that vein, I wanted to reach out to the rest of the Ops Community and ask one pretty simple question. Would it be accurate to say that most of your end users want almost nothing to do with the network? That was my experience at ATT, yes. The vast majority of end users could not care less about networking, as long as the connectivity was reliable, performed well, and they could connect to the Internet (and have others connect from the Internet to their VMs) when needed. In my experience what the majority of them (both internal and external) want is to consume from Openstack a compute resource, a property of which is it that resource has an IP address. They, at most, care about which network they are on. Where a network is usually an arbitrary definition around a set of real networks, that are constrained to a location, in which the company has attached some sort of policy. For example, I want to be in the production network vs's the xyz lab network, vs's the backup network, vs's the corp network. I would say for Godaddy, 99% of our use cases would be defined as: I want a compute resource in the production network zone, or I want a compute resource in this other network zone. The end user only cares that the IP the vm receives works in that zone, outside of that they don't care any other property of that IP. They do not care what subnet it is in, what vlan it is on, what switch it is attached to, what router its attached to, or how data flows in/out of that network. It just needs to work. We have also found that by giving the users a floating ip address that can be moved between vm's (but still constrained within a network zone) we can solve almost all of our users asks. Typically, the internal need for a floating ip is when a compute resource needs to talk to another protected internal
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
On 17 Jun 2015, at 10:56 am, Armando M. arma...@gmail.com wrote: On 16 June 2015 at 17:31, Sam Morrison sorri...@gmail.com mailto:sorri...@gmail.com wrote: We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. I’m not sure what the correct way of doing this. What are other people doing in the interim until this kind of use case can be done in Neutron? Would something like [1] be adequate to address your use case? If not, I'd suggest you to file an RFE bug (more details in [2]), so that we can keep the discussion focused on this specific case. HTH Armando [1] https://blueprints.launchpad.net/neutron/+spec/rbac-networks https://blueprints.launchpad.net/neutron/+spec/rbac-networks That’s not applicable in this case. We don’t care about what tenants are when in this case. [2] https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst#neutron-request-for-feature-enhancements https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst#neutron-request-for-feature-enhancements The bug Kris mentioned outlines all I want too I think. Sam Cheers, Sam [1] https://github.com/NeCTAR-RC/nova/commit/1bc2396edc684f83ce471dd9dc9219c4635afb12 https://github.com/NeCTAR-RC/nova/commit/1bc2396edc684f83ce471dd9dc9219c4635afb12 On 17 Jun 2015, at 12:20 am, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: Adding -dev because of the reference to the Neutron Get me a network spec. Also adding [nova] and [neutron] subject markers. Comments inline, Kris. On 05/22/2015 09:28 PM, Kris G. Lindgren wrote: During the Openstack summit this week I got to talk to a number of other operators of large Openstack deployments about how they do networking. I was happy, surprised even, to find that a number of us are using a similar type of networking strategy. That we have similar challenges around networking and are solving it in our own but very similar way. It is always nice to see that other people are doing the same things as you or see the same issues as you are and that you are not crazy. So in that vein, I wanted to reach out to the rest of the Ops Community and ask one pretty simple question. Would it be accurate to say that most of your end users want almost nothing to do with the network? That was my experience at ATT, yes. The vast majority of end users could not care less about networking, as long as the connectivity was reliable, performed well, and they could connect to the Internet (and have others connect from the Internet to their VMs) when needed. In my experience what the majority of them (both internal and external) want is to consume from Openstack a compute resource, a property of which is it that resource has an IP address. They, at most, care about which network they are on. Where a network is usually an arbitrary definition around a set of real networks, that are constrained to a location, in which the company has attached some sort of policy. For example, I want to be in the production network vs's the xyz lab network, vs's the backup network, vs's the corp network. I would say for Godaddy, 99% of our use cases would be defined as: I want a compute resource in the production network zone, or I want a compute resource in this other network zone. The end user only cares
Re: [openstack-dev] [Openstack-operators] [nova] [neutron] Re: How do your end users use networking?
We at NeCTAR are starting the transition to neutron from nova-net and neutron almost does what we want. We have 10 “public networks and 10 “service networks and depending on which compute node you land on you get attached to one of them. In neutron speak we have multiple shared externally routed provider networks. We don’t have any tenant networks or any other fancy stuff yet. How I’ve currently got this set up is by creating 10 networks and subsequent subnets eg. public-1, public-2, public-3 … and service-1, service-2, service-3 and so on. In nova we have made a slight change in allocate for instance [1] whereby the compute node has a designated hardcoded network_ids for the public and service network it is physically attached to. We have also made changes in the nova API so users can’t select a network and the neutron endpoint is not registered in keystone. That all works fine but ideally I want a user to be able to choose if they want a public and or service network. We can’t let them as we have 10 public networks, we almost need something in neutron like a network group” or something that allows a user to select “public” and it allocates them a port in one of the underlying public networks. I tried going down the route of having 1 public and 1 service network in neutron then creating 10 subnets under each. That works until you get to things like dhcp-agent and metadata agent although this looks like it could work with a few minor changes. Basically I need a dhcp-agent to be spun up per subnet and ensure they are spun up in the right place. I’m not sure what the correct way of doing this. What are other people doing in the interim until this kind of use case can be done in Neutron? Cheers, Sam [1] https://github.com/NeCTAR-RC/nova/commit/1bc2396edc684f83ce471dd9dc9219c4635afb12 On 17 Jun 2015, at 12:20 am, Jay Pipes jaypi...@gmail.com wrote: Adding -dev because of the reference to the Neutron Get me a network spec. Also adding [nova] and [neutron] subject markers. Comments inline, Kris. On 05/22/2015 09:28 PM, Kris G. Lindgren wrote: During the Openstack summit this week I got to talk to a number of other operators of large Openstack deployments about how they do networking. I was happy, surprised even, to find that a number of us are using a similar type of networking strategy. That we have similar challenges around networking and are solving it in our own but very similar way. It is always nice to see that other people are doing the same things as you or see the same issues as you are and that you are not crazy. So in that vein, I wanted to reach out to the rest of the Ops Community and ask one pretty simple question. Would it be accurate to say that most of your end users want almost nothing to do with the network? That was my experience at ATT, yes. The vast majority of end users could not care less about networking, as long as the connectivity was reliable, performed well, and they could connect to the Internet (and have others connect from the Internet to their VMs) when needed. In my experience what the majority of them (both internal and external) want is to consume from Openstack a compute resource, a property of which is it that resource has an IP address. They, at most, care about which network they are on. Where a network is usually an arbitrary definition around a set of real networks, that are constrained to a location, in which the company has attached some sort of policy. For example, I want to be in the production network vs's the xyz lab network, vs's the backup network, vs's the corp network. I would say for Godaddy, 99% of our use cases would be defined as: I want a compute resource in the production network zone, or I want a compute resource in this other network zone. The end user only cares that the IP the vm receives works in that zone, outside of that they don't care any other property of that IP. They do not care what subnet it is in, what vlan it is on, what switch it is attached to, what router its attached to, or how data flows in/out of that network. It just needs to work. We have also found that by giving the users a floating ip address that can be moved between vm's (but still constrained within a network zone) we can solve almost all of our users asks. Typically, the internal need for a floating ip is when a compute resource needs to talk to another protected internal or external resource. Where it is painful (read: slow) to have the acl's on that protected resource updated. The external need is from our hosting customers who have a domain name (or many) tied to an IP address and changing IP's/DNS is particularly painful. This is precisely my experience as well. Since the vast majority of our end users don't care about any of the technical network stuff, we spend a large amount of time/effort in abstracting or hiding the technical stuff from the users view. Which
