> > Hi Ian,
> >
> > > -----Original Message-----
> > > From: Stokes, Ian <[email protected]>
> > > Sent: Wednesday, January 16, 2019 12:37 AM
> > > To: Kevin Traynor <[email protected]>; Ophir Munk
> > > <[email protected]>; [email protected]
> > > Cc: Olga Shern <[email protected]>; Ilya Maximets
> > > <[email protected]>
> > > Subject: RE: [PATCH v5] netdev-dpdk: support port representors
> > >
> > > > On 01/15/2019 09:47 AM, Ophir Munk wrote:
> > > > > Dpdk port representors were introduced in dpdk versions 18.xx.
> > > > > Prior to port representors there was a one-to-one relationship
> > > > > between an rte device (e.g. PCI bus) and an eth device
> > > > > (referenced as dpdk port id in OVS). With port representors the
> > > > > relationship becomes one-to-many rte device to eth devices.
> > > > > For example in [3] there are two devices (representors) using
> > > > > the same PCI physical address 0000:08:00.0:
> > > > > "0000:08:00.0,representor=[3]" and "0000:08:00.0,representor=[5]".
> > > > > This commit handles the new one-to-many relationship. For
> > > > > example, when one of the device port representors in [3] is
> > > > > closed - the PCI bus cannot be detached until the other device
> > > > > port representor is closed as well. OVS remains backward
> > > > > compatible by supporting dpdk legacy PCI ports which do not
> include port representors.
> > > > > Dpdk port representors related commits are listed in [1]. Dpdk
> > > > > port representors documentation appears in [2]. A sample
> > > > > configuration which uses two representors ports (the output of
> > > > > "ovs-
> > vsctl show"
> > > > > command) is shown in [3].
> > > > >
> > > >
> > > > Hi Ophir, I had a scan through and there isn't any
> > > > documentation/examples for this outside the commit message. I
> > > > think a user would need something basic, or at least reference to
> > > > know that this
> > > exists and how to use it.
> > >
> > > +1, although I can confirm the backwards compatibility with the
> > > +legacy pci
> > > port methodology I'm seeing issues around the use of representors
> > > which I'm not entirely sure are user configuration related or
> > > specific to the patch implementation, will need more time to
> > > investigate and
> > confirm.
> > >
> > > Ian
> >
> > Can you please inform what exact issues you are seeing with
> representors?
>
> Hi Ophir, in my case trying to add an i40e port representor was failing
> with invalid port.
>
> I want to confirm this is not an issue in DPDK, or a configuration issue
> on my side, or that it's just not supported (although I would have thought
> it was).
>
> > What is the configuration?
> PF I40e address with 05:00.0 is bound to igb_uio.
> VF then created for PF with 'echo 1 >
> /sys/bus/pci/devices/0000\:05\:00.0/max_vfs'.
> VF is created with address 05:02.0
> VF then bound to igb_uio driver.
> OVS DPDK then stated (no new commands are used here when launching).
> Add bridge br0
> Attempt to add VF with port representor, use the PF address and
> representor of VF
>
> ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> options:dpdk-devargs=0000:05:00.0,representor=[0]
>
> Possibly it's above where I'm going wrong. Either on the address passed,
> or the representor value used (though I've tried a range of represntor
> values besides 0, no luck, how do you typically identify the value?)
>
> > What is the NIC under test?
> The NIC is Intel X710 (Ethernet Controller X710 for 10GbE SFP+ 1572).
> The VF created is recognized as Ethernet Virtual Function 700 Series 154c
>
> > What is the expected result?
> I was attempting to see if the port representor could be used to add a VF
> for the i40e PF.
> There is no issue if I add the VF using the legacy method ' ovs-vsctl add-
> port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-
> devargs=0000:05:02.0'.
>
> > What is the actual result?
>
> 2019-01-16T10:18:38Z|00062|dpdk|INFO|EAL: PCI device 0000:05:00.0 on NUMA
> socket 0
> 2019-01-16T10:18:38Z|00063|dpdk|ERR|EAL: Failed to attach device on
> primary process 2019-01-16T10:18:38Z|00064|netdev_dpdk|WARN|Error
> attaching device '0000:05:00.0,representor=[0]' to DPDK
> 2019-01-16T10:18:38Z|00065|netdev|WARN|dpdk0: could not set configuration
> (Invalid argument) 2019-01-16T10:18:38Z|00066|dpdk|ERR|Invalid port_id=32
> >
> > Without those details I do not know how the "issues" can be addressed.
>
> As I said this could be a user configuration issue, I spoke with Awal co-
> authored the i40e port representor code which is how I arrived at the
> commands above but these could be wrong.
Hi Ophir,
Quick update on this. I've managed to get the port representors working to an
extent for i40e and ixgbe devices with this patch but there are a few issues
causing failures.
ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
options:dpdk-devargs=0000:05:00.0,representor=[0]
will succeed if both the PF and the VF are hotplugged to the IGB_UIO driver
after OVS has launched.
However it will fail if PF has already been bound to the IGB_UIO driver before
launching OVS or after being probed by a previous call to add another
representor that shares the same PF.
This has to do with the logic around netdev_dpdk_probe_port_by_devargs().
The port_id in this will initially be set to INVALID as it will not be found as
part of the following iteration
RTE_ETH_FOREACH_MATCHING_DEV (port_id, devargs, &iterator) {
/* If a break is done - must call rte_eth_iterator_cleanup. */
rte_eth_iterator_cleanup(&iterator);
break;
}
This makes sense as it will not have been added as a DPDK device yet in both
hotplug and non hotplug cases.
It could be the case that the device was not probed yet, in which it is probed
and port ID checked again.
/* The port may be invalid if it was not probed */
if (port_id == DPDK_ETH_PORT_ID_INVALID) {
if (rte_dev_probe(devargs)) {
port_id = DPDK_ETH_PORT_ID_INVALID;
} else {
RTE_ETH_FOREACH_MATCHING_DEV (port_id, devargs, &iterator) {
/* If a break is done - must call rte_eth_iterator_cleanup. */
rte_eth_iterator_cleanup(&iterator);
break;
}
}
>From my testing I see the call above to rte_dev_probe(devargs) flagging a DPDK
>error for attaching to a primary process (this is because 'local_dev_probe(')
>will return EEXIST for devices that were already probed in EAL init for OVS
>DPDK.). This does not occur in the hotplug case.
Although the error is flagged, DPDK then attempts to attach the port
representor via secondary process with 'eal_dev_hotplug_request_to_secondary()'.
This seems to succeed as no error is returned here. However after returning
from rte_dev_probe(devargs) (which returns 0), when we iterate over the ETH
list again with
} else {
RTE_ETH_FOREACH_MATCHING_DEV (port_id, devargs, &iterator) {
/* If a break is done - must call rte_eth_iterator_cleanup. */
rte_eth_iterator_cleanup(&iterator);
break;
}
It seems the secondary device (representor in this case) although added is not
picked up in this iteration, so port_id remains as invalid.
As I said this only occurs when the PF has been probed on init (non hotplug use
case).
But even for the hotplug case it will also be a problem if you wish to add
another representor that uses the same PF.
You may have resolved the issue in the v6 but I thought it was wise to flag
what the root cause of the problem I was hitting.
Ian
>
> Ian
> >
> > Regards,
> > Ophir
> _______________________________________________
> dev mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
