Hi Yunhong,

I haven't looked at Mellanox in much detail. I think that we'll get more 
details from Irena down the road. Regarding your question, I can only answer 
based on my experience with Cisco's VM-FEX. In a nutshell:
     -- a vNIC is connected to an external switch. Once the host is booted up, 
all the PFs and VFs provisioned on the vNIC will be created, as well as all the 
corresponding ethernet interfaces .
     -- As far as Neutron is concerned, a neutron port can be associated with a 
VF. One way to do so is to specify this requirement in the —nic option, 
providing information such as:
               . PCI alias (this is the same alias as defined in your nova 
               . direct pci-passthrough/macvtap
               . port profileid that is compliant with 802.1Qbh
     -- similar to how you translate the nova flavor with PCI requirements to 
PCI requests for scheduling purpose, Nova API (the nova api component) can 
translate the above to PCI requests for scheduling purpose. I can give more 
detail later on this.

Regarding your last question, since the vNIC is already connected with the 
external switch, the vNIC driver will be responsible for communicating the port 
profile to the external switch. As you have already known, libvirt provides 
several ways to specify a VM to be booted up with SRIOV. For example, in the 
following interface definition:

  <interface type='hostdev' managed='yes'>
        <address type='pci' domain='0' bus='0x09' slot='0x0' function='0x01'/>
      <mac address='01:23:45:67:89:ab' />
      <virtualport type='802.1Qbh'>
        <parameters profileid='my-port-profile' />

The SRIOV VF (bus 0x09, VF 0x01) will be allocated, and the port profile 
'my-port-profile' will be used to provision this VF. Libvirt will be 
responsible for invoking the vNIC driver to configure this VF with the port 
profile my-port-porfile. The driver will talk to the external switch using the 
802.1qbh standards to complete the VF's configuration and binding with the VM.

Now that nova PCI passthrough is responsible for 
discovering/scheduling/allocating a VF, the rest of the puzzle is to associate 
this PCI device with the feature that's going to use it, and the feature will 
be responsible for configuring it. You can also see from the above example, in 
one implementation of SRIOV, the feature (in this case neutron) may not need to 
do much in terms of working with the external switch, the work is actually done 
by libvirt behind the scene.

Now the questions are:

        -- how the port profile gets defined/managed

        -- how the port profile gets associated with a neutron network

The first question will be specific to the particular product, and therefore a 
particular neutron plugin has to mange that.

There may be several approaches to address the second question. For example, in 
the simplest case, a port profile can be associated with a neutron network. 
This has some significant drawbacks. Since the port profile defines features 
for all the ports that use it, the one port profile to one neutron network 
mapping would mean all the ports on the network will have exactly the same 
features (for example, QoS characteristics). To make it flexible, the binding 
of a port profile to a port may be done at the port creation time.

Let me know if the above answered your question.



On 10/29/13 3:03 AM, "Jiang, Yunhong" 
<yunhong.ji...@intel.com<mailto:yunhong.ji...@intel.com>> wrote:

Robert, is it possible to have a IRC meeting? I’d prefer to IRC meeting because 
it’s more openstack style and also can keep the minutes clearly.

To your flow, can you give more detailed example. For example, I can consider 
user specify the instance with –nic option specify a network id, and then how 
nova device the requirement to the PCI device? I assume the network id should 
define the switches that the device can connect to , but how is that 
information translated to the PCI property requirement? Will this translation 
happen before the nova scheduler make host decision?


From: Robert Li (baoli) [mailto:ba...@cisco.com]
Sent: Monday, October 28, 2013 12:22 PM
To: Irena Berezovsky; 
prashant.upadhy...@aricent.com<mailto:prashant.upadhy...@aricent.com>; Jiang, 
Yunhong; chris.frie...@windriver.com<mailto:chris.frie...@windriver.com>; He, 
Yongli; Itzik Brown
Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle Mestery 
(kmestery); Sandhya Dasu (sadasu)
Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network support

Hi Irena,

Thank you very much for your comments. See inline.


On 10/27/13 3:48 AM, "Irena Berezovsky" 
<ire...@mellanox.com<mailto:ire...@mellanox.com>> wrote:

Hi Robert,
Thank you very much for sharing the information regarding your efforts. Can you 
please share your idea of the end to end flow? How do you suggest  to bind Nova 
and Neutron?

The end to end flow is actually encompassed in the blueprints in a nutshell. I 
will reiterate it in below. The binding between Nova and Neutron occurs with 
the neutron v2 API that nova invokes in order to provision the neutron 
services. The vif driver is responsible for plugging in an instance onto the 
networking setup that neutron has created on the host.

Normally, one will invoke "nova boot" api with the —nic options to specify the 
nic with which the instance will be connected to the network. It currently 
allows net-id, fixed ip and/or port-id to be specified for the option. However, 
it doesn't allow one to specify special networking requirements for the 
instance. Thanks to the nova pci-passthrough work, one can specify PCI 
passthrough device(s) in the nova flavor. But it doesn't provide means to tie 
up these PCI devices in the case of ethernet adpators with networking services. 
Therefore the idea is actually simple as indicated by the blueprint titles, to 
provide means to tie up SRIOV devices with neutron services. A work flow would 
roughly look like this for 'nova boot':

      -- Specifies networking requirements in the —nic option. Specifically for 
SRIOV, allow the following to be specified in addition to the existing required 
               . PCI alias
               . direct pci-passthrough/macvtap
               . port profileid that is compliant with 802.1Qbh

        The above information is optional. In the absence of them, the existing 
behavior remains.

     -- if special networking requirements exist, Nova api creates PCI requests 
in the nova instance type for scheduling purpose

     -- Nova scheduler schedules the instance based on the requested flavor 
plus the PCI requests that are created for networking.

     -- Nova compute invokes neutron services with PCI passthrough information 
if any

     --  Neutron performs its normal operations based on the request, such as 
allocating a port, assigning ip addresses, etc. Specific to SRIOV, it should 
validate the information such as profileid, and stores them in its db. It's 
also possible to associate a port profileid with a neutron network so that port 
profileid becomes optional in the —nic option. Neutron returns  nova the port 
information, especially for PCI passthrough related information in the port 
binding object. Currently, the port binding object contains the following 

    -- nova constructs the domain xml and plug in the instance by calling the 
vif driver. The vif driver can build up the interface xml based on the port 
binding information.

The blueprints you registered make sense. On Nova side, there is a need to bind 
between requested virtual network and PCI device/interface to be allocated as 
On the Neutron side, there is a need to  support networking configuration of 
the vNIC. Neutron should be able to identify the PCI device/macvtap interface 
in order to apply configuration. I think it makes sense to provide neutron 
integration via dedicated Modular Layer 2 Mechanism Driver to allow PCI 
pass-through vNIC support along with other networking technologies.

I haven't sorted through this yet. A neutron port could be associated with a 
PCI device or not, which is a common feature, IMHO. However, a ML2 driver may 
be needed specific to a particular SRIOV technology.

During the Havana Release, we introduced Mellanox Neutron plugin that enables 
networking via SRIOV pass-through devices or macvtap interfaces.
We want to integrate our solution with PCI pass-through Nova support.  I will 
be glad to share more details if you are interested.

Good to know that you already have a SRIOV implementation. I found out some 
information online about the mlnx plugin, but need more time to get to know it 
better. And certainly I'm interested in knowing its details.

The PCI pass-through networking support is planned to be discussed during the 
summit: http://summit.openstack.org/cfp/details/129. I think it’s worth to 
drill down into more detailed proposal and present it during the summit, 
especially since it impacts both nova and neutron projects.

I agree. Maybe we can steal some time in that discussion.

Would you be interested in collaboration on this effort? Would you be 
interested to exchange more emails or set an IRC/WebEx meeting during this week 
before the summit?

Sure. If folks want to discuss it before the summit, we can schedule a webex 
later this week. Or otherwise, we can continue the discussion with email.


From: Robert Li (baoli) [mailto:ba...@cisco.com]
Sent: Friday, October 25, 2013 11:16 PM
To: prashant.upadhy...@aricent.com<mailto:prashant.upadhy...@aricent.com>; 
Irena Berezovsky; yunhong.ji...@intel.com<mailto:yunhong.ji...@intel.com>; 
Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle Mestery 
(kmestery); Sandhya Dasu (sadasu)
Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network support

Hi Irena,

This is Robert Li from Cisco Systems. Recently, I was tasked to investigate 
such support for Cisco's systems that support VM-FEX, which is a SRIOV 
technology supporting 802-1Qbh. I was able to bring up nova instances with 
SRIOV interfaces, and establish networking in between the instances that 
employes the SRIOV interfaces. Certainly, this was accomplished with hacking 
and some manual intervention. Based on this experience and my study with the 
two existing nova pci-passthrough blueprints that have been implemented and 
committed into Havana 
(https://blueprints.launchpad.net/nova/+spec/pci-passthrough-base and
https://blueprints.launchpad.net/nova/+spec/pci-passthrough-libvirt),  I 
registered a couple of blueprints (one on Nova side, the other on the Neutron 


in order to address SRIOV support in openstack.

Please take a look at them and see if they make sense, and let me know any 
comments and questions. We can also discuss this in the summit, I suppose.

I noticed that there is another thread on this topic, so copy those folks  from 
that thread as well.


On 10/16/13 4:32 PM, "Irena Berezovsky" 
<ire...@mellanox.com<mailto:ire...@mellanox.com>> wrote:

As one of the next steps for PCI pass-through I would like to discuss is the 
support for PCI pass-through vNIC.
While nova takes care of PCI pass-through device resources  management and VIF 
settings, neutron should manage their networking configuration.
I would like to register asummit proposal to discuss the support for PCI 
pass-through networking.
I am not sure what would be the right topic to discuss the PCI pass-through 
networking, since it involve both nova and neutron.
There is already a session registered by Yongli on nova topic to discuss the 
PCI pass-through next steps.
I think PCI pass-through networking is quite a big topic and it worth to have a 
separate discussion.
Is there any other people who are interested to discuss it and share their 
thoughts and experience?


OpenStack-dev mailing list

Reply via email to