Re: [openstack-dev] [nova] [neutron] Todays' meeting log: PCI pass-through network support
Somehow I missed this email until today. See inline… thanks, Robert On 12/24/13 1:31 AM, Irena Berezovsky ire...@mellanox.commailto:ire...@mellanox.com wrote: Please, see inline From: Ian Wells [mailto:ijw.ubu...@cack.org.uk] Sent: Tuesday, December 24, 2013 1:38 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] [neutron] Todays' meeting log: PCI pass-through network support On autodiscovery and configuration, we agree that each compute node finds out what it has based on some sort of list of match expressions; we just disagree on where they should live. I know we've talked APIs for setting that matching expression, but I would prefer that compute nodes are responsible for their own physical configuration - generally this seems wiser on the grounds that configuring new hardware correctly is a devops problem and this pushes the problem into the installer, clear devops territory. It also makes the (I think likely) assumption that the config may differ per compute node without having to add more complexity to the API with host aggregates and so on. And it means that a compute node can start working without consulting the central database or reporting its entire device list back to the central controller. [IrenaB] Totally agree on this. For both auto-discovery and configuration, we need to close the format and content that is will be available to nova. [Robert] Agreed. With auto-discovery, if all the PCI passthrough networking devices on a compute node are there for guest use, then no whitelist needs to be configured on the node at all. Also, for deployments that do require explicit whitelist, I'd think that we may go with configuration-based whitelist first, and then if needed, add the API based. My concern here if there is a way to provide auto-discovery based on network connectivity (something like what neutron has i.e. ‘physical_interface_mappings’) [Robert] Can you elaborate on this? A specific plugin can have its own configuration defined. What we want to make sure is that the port binding passed between nova and neutron contains sufficient information for a plugin to operate. For configuration, maybe worth to provide some reference flow for managing it by installer. On PCI groups, I think it is a good idea to have them declared centrally (their name, not their content). Now, I would use config to define them and maybe an API for the tenant to list their names, personally; that's simpler and easier to implement and doesn't preclude adding an (admin) API in the future. But I don't imagine the list of groups will change frequently so any update API would be very infrequently used, and if someone really feels they want to implement it I'm not going to stop them. [IrenaB] The issue we need to resolve is nova scheduler taking its decision that satisfies network connectivity On nova boot, I completely agree that we need a new argument to --nic to specify the PCI group of the NIC. The rest of the arguments - I'm wondering if we could perhaps do this in two stages: 1. Neutron will read those arguments (attachment type, additional stuff like port group where relevant) from the port during an attach and pass relevant information to the plugging driver in Nova [IrenaB] Do you mean via ‘neutron port-create before ‘nova boot’? Hopefully we can close the details during the discussion today. [Robert] this has been proposed in the google doc. 2. We add a feature to nova so that you can specify other properties in the --nic section line and they're passed straight to the port-create called from within nova. [IrenaB] I like this option. This should also allow to request virtio versus SR-IOV nic. It should be possible to have both options available on the same Host. [Robert] Currently, with neutron plugin, the properties contained in the —nic option is directly passed to neutron. However, nova will perform some validation checks by calling neutron. All the properties are organized in the requested_network dictionary. This means, adding new properties into the requested_network dictionary requires nova code change so that they can be added in the dictionary. It also means all the places that use the dictionary may need to be changed. If I understand correctly, you suggest that some properties be grouped in a dict field of the requested_network dictionary. It's a good idea. Future expansion of the —nic option, if any, can be made possible by just changing the nova client and the nova code interfacing with the nova client. So sounds like that the newly proposed properties in —nic are ok? I'd think that we should support nova boot first since it will not alter the existing application flow to create instances. Support of port-create with the new properties is an easy task once support for 'nova boot' is done. This is not specific to passthrough at all, just a useful general purpose feature
Re: [openstack-dev] [nova] [neutron] Todays' meeting log: PCI pass-through network support
On autodiscovery and configuration, we agree that each compute node finds out what it has based on some sort of list of match expressions; we just disagree on where they should live. I know we've talked APIs for setting that matching expression, but I would prefer that compute nodes are responsible for their own physical configuration - generally this seems wiser on the grounds that configuring new hardware correctly is a devops problem and this pushes the problem into the installer, clear devops territory. It also makes the (I think likely) assumption that the config may differ per compute node without having to add more complexity to the API with host aggregates and so on. And it means that a compute node can start working without consulting the central database or reporting its entire device list back to the central controller. On PCI groups, I think it is a good idea to have them declared centrally (their name, not their content). Now, I would use config to define them and maybe an API for the tenant to list their names, personally; that's simpler and easier to implement and doesn't preclude adding an (admin) API in the future. But I don't imagine the list of groups will change frequently so any update API would be very infrequently used, and if someone really feels they want to implement it I'm not going to stop them. On nova boot, I completely agree that we need a new argument to --nic to specify the PCI group of the NIC. The rest of the arguments - I'm wondering if we could perhaps do this in two stages: 1. Neutron will read those arguments (attachment type, additional stuff like port group where relevant) from the port during an attach and pass relevant information to the plugging driver in Nova 2. We add a feature to nova so that you can specify other properties in the --nic section line and they're passed straight to the port-create called from within nova. This is not specific to passthrough at all, just a useful general purpose feature. However, it would simplify both the problem and design here, because these parameters, whatever they are, are now entirely the responsibility of Neutron and Nova's simply transporting them into it. A PCI aware Neutron will presumably understand the attachment type, the port group and so on, or will reject them if they're meaningless to it, and we've even got room for future expansion without changing Nova or Neutron, just the plugin. We can propose it now and independently, put in a patch and have it ready before we need it. I think anything that helps to clarify and divide the responsibilities of Nova and Neutron will be helpful, because then we don't end up with too many cross-project-interrelated patches. I'm going to ignore the allocation problem for now. If a single user can allocate all the NICs in the cluster to himself, we still have a more useful solution than the one now where he can't use them, so it's not the top of our list. Time seems to be running out for Icehouse. We need to come to agreement ASAP. I will be out from wednesday until after new year. I'm thinking that to move it forward after the new year, we may need to have the IRC meeting in a daily basis until we reach agreement. This should be one of our new year's resolutions? Whatever gets it done. -- Ian. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [neutron] Todays' meeting log: PCI pass-through network support
Please, see inline From: Ian Wells [mailto:ijw.ubu...@cack.org.uk] Sent: Tuesday, December 24, 2013 1:38 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] [neutron] Todays' meeting log: PCI pass-through network support On autodiscovery and configuration, we agree that each compute node finds out what it has based on some sort of list of match expressions; we just disagree on where they should live. I know we've talked APIs for setting that matching expression, but I would prefer that compute nodes are responsible for their own physical configuration - generally this seems wiser on the grounds that configuring new hardware correctly is a devops problem and this pushes the problem into the installer, clear devops territory. It also makes the (I think likely) assumption that the config may differ per compute node without having to add more complexity to the API with host aggregates and so on. And it means that a compute node can start working without consulting the central database or reporting its entire device list back to the central controller. [IrenaB] Totally agree on this. For both auto-discovery and configuration, we need to close the format and content that is will be available to nova. My concern here if there is a way to provide auto-discovery based on network connectivity (something like what neutron has i.e. 'physical_interface_mappings') For configuration, maybe worth to provide some reference flow for managing it by installer. On PCI groups, I think it is a good idea to have them declared centrally (their name, not their content). Now, I would use config to define them and maybe an API for the tenant to list their names, personally; that's simpler and easier to implement and doesn't preclude adding an (admin) API in the future. But I don't imagine the list of groups will change frequently so any update API would be very infrequently used, and if someone really feels they want to implement it I'm not going to stop them. [IrenaB] The issue we need to resolve is nova scheduler taking its decision that satisfies network connectivity On nova boot, I completely agree that we need a new argument to --nic to specify the PCI group of the NIC. The rest of the arguments - I'm wondering if we could perhaps do this in two stages: 1. Neutron will read those arguments (attachment type, additional stuff like port group where relevant) from the port during an attach and pass relevant information to the plugging driver in Nova [IrenaB] Do you mean via 'neutron port-create before 'nova boot'? Hopefully we can close the details during the discussion today. 2. We add a feature to nova so that you can specify other properties in the --nic section line and they're passed straight to the port-create called from within nova. [IrenaB] I like this option. This should also allow to request virtio versus SR-IOV nic. It should be possible to have both options available on the same Host. This is not specific to passthrough at all, just a useful general purpose feature. However, it would simplify both the problem and design here, because these parameters, whatever they are, are now entirely the responsibility of Neutron and Nova's simply transporting them into it. A PCI aware Neutron will presumably understand the attachment type, the port group and so on, or will reject them if they're meaningless to it, and we've even got room for future expansion without changing Nova or Neutron, just the plugin. We can propose it now and independently, put in a patch and have it ready before we need it. I think anything that helps to clarify and divide the responsibilities of Nova and Neutron will be helpful, because then we don't end up with too many cross-project-interrelated patches. [IrenaB] +2 I'm going to ignore the allocation problem for now. If a single user can allocate all the NICs in the cluster to himself, we still have a more useful solution than the one now where he can't use them, so it's not the top of our list. [IrenaB] Agree Time seems to be running out for Icehouse. We need to come to agreement ASAP. I will be out from wednesday until after new year. I'm thinking that to move it forward after the new year, we may need to have the IRC meeting in a daily basis until we reach agreement. This should be one of our new year's resolutions? Whatever gets it done. [IrenaB] Fine with me. If we reach required decisions today regarding neutron, I can start to dive into the details of SR-IOV mechanism driver assuming ML2 plugin. BR, Irena -- Ian. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
