Hi, I eventually managed to get it working by using "uio_pci_generic" as a DPDK-enabled driver for Intel 82599 ports. I had to disable IOMMU (e.g. intel_iommu=off in the kernel boot parameters), otherwise uio_pci_generic does not work. Is this documented somewhere? If not, I think it should be.
In any case, I confirm that DPDK with OVS 2.7.0 does not work when vfio-pci is used as DPDK enabled driver; I keep getting "Error attaching device". This is weird, because DPDK pktgen works fine with vfio-pci DPDK driver (intel_iommu must be on in this case). Thanks, Vincenzo 2017-04-20 18:21 GMT+02:00 Vincenzo Maffione <[email protected] >: > vfio-pci is loaded and bound to the NIC, as shown by dpdk-devbind. > > $ sudo tools/dpdk-devbind.py --status > > Network devices using DPDK-compatible driver > ============================================ > 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=vfio-pci > unused=ixgbe > > Network devices using kernel driver > =================================== > 0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e > unused=vfio-pci > 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp1s0f1 > drv=ixgbe unused=vfio-pci > 0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller' > if=enp6s0 drv=r8169 unused=vfio-pci > 0000:07:00.0 '82574L Gigabit Network Connection' if=enp7s0 drv=e1000e > unused=vfio-pci > > [...] > > And I get: > > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: PCI device > 0000:01:00.0 on NUMA socket -1 > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: probe driver: > 8086:10fb net_ixgbe > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Requested device > 0000:01:00.0 cannot be used > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Driver cannot attach > the device (0000:01:00.0) > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: > 2017-04-20T16:19:22Z|00027|netdev_dpdk|WARN|Error > attaching device '0000:01:00.0' to DPDK > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: > 2017-04-20T16:19:22Z|00028|netdev|WARN|dpdk-p1: > could not set configuration (Invalid argumen > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: PCI device > 0000:01:00.0 on NUMA socket -1 > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: probe driver: > 8086:10fb net_ixgbe > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Requested device > 0000:01:00.0 cannot be used > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: EAL: Driver cannot attach > the device (0000:01:00.0) > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: > ovs|00027|netdev_dpdk|WARN|Error > attaching device '0000:01:00.0' to DPDK > Apr 20 18:19:22 arch-server ovs-vswitchd[650]: > ovs|00028|netdev|WARN|dpdk-p1: could not set configuration (Invalid > argument) > > > Thanks, > Vincenzo > > > 2017-04-20 18:13 GMT+02:00 Shivaram Mysore <[email protected]>: > >> I have a feeling that vfio-pci is not installed. Note that you will have >> to edit /etc/modules to load on boot. >> >> You can also check by >> # lsmod | grep vfio >> >> To load on demand - use >> # modprobe vfio-pci >> >> >> >> On Thu, Apr 20, 2017 at 9:03 AM, Vincenzo Maffione <[email protected]> >> wrote: >> >>> Hi, >>> I installed OVS 2.7.0 and use uio_pci_generic in place of vfio-pci, >>> and the error disappeared. >>> >>> Thanks, >>> Vincenzo >>> >>> 2017-04-15 17:56 GMT+02:00 Shivaram Mysore <[email protected]>: >>> >>>> You should look at emails from -(Hint: search DPDK in subject) >>>> >>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Marc >>>> h/thread.html >>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Apri >>>> l/thread.html >>>> https://mail.openvswitch.org/pipermail/ovs-discuss/2017-Apri >>>> l/044155.html >>>> >>>> If possible, I would recommend using Ubuntu 17.04 with OVS 2.7 with >>>> DPDK packages from http://packages.wand.net.nz/ which can simplify >>>> your deployment. >>>> >>>> /Shivaram >>>> >>>> On Fri, Apr 14, 2017 at 4:04 AM, Vincenzo Maffione < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> I would like to attach one of my ixgbe NIC ports to an OpenVSwitch >>>>> instance, but I keep getting an unexpected error, see below. >>>>> >>>>> I've successfully bound both my ixgbe NIC ports to DPDK, using VFIO: >>>>> >>>>> # tools/dpdk-devbind.py --status >>>>> >>>>> Network devices using DPDK-compatible driver >>>>> ============================================ >>>>> 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' >>>>> drv=vfio-pci >>>>> unused=ixgbe >>>>> 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' >>>>> drv=vfio-pci >>>>> unused=ixgbe >>>>> >>>>> Network devices using kernel driver >>>>> =================================== >>>>> 0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e >>>>> unused=vfio-pci >>>>> 0000:06:00.0 'RTL8111/8168/8411 PCI Express Gigabit Ethernet >>>>> Controller' >>>>> if=enp6s0 drv=r8169 unused=vfio-pci >>>>> 0000:07:00.0 '82574L Gigabit Network Connection' if=enp7s0 drv=e1000e >>>>> unused=vfio-pci >>>>> >>>>> Other network devices >>>>> ===================== >>>>> <none> >>>>> >>>>> Crypto devices using DPDK-compatible driver >>>>> =========================================== >>>>> <none> >>>>> >>>>> Crypto devices using kernel driver >>>>> ================================== >>>>> <none> >>>>> >>>>> Other crypto devices >>>>> ==================== >>>>> <none> >>>>> >>>>> >>>>> Then I created a bridge >>>>> >>>>> # ovs-vsctl add-br obr1 -- set bridge obr1 datapath_type=netdev >>>>> >>>>> But then when trying to add the dpdk NIC: >>>>> >>>>> # ovs-vsctl add-port obr1 dpdk-p0 -- set Interface dpdk-p0 type=dpdk >>>>> options:dpdk-devargs=0000:01:00.0 >>>>> >>>>> the command fails, and I get this in the log: >>>>> >>>>> ovs-vswitchd[353]: 2017-04-14T10:54:06Z|00026|bridge|WARN|could not >>>>> open >>>>> network device dpdk-p0 (No such device) >>>>> ovs-vswitchd[353]: ovs|00026|bridge|WARN|could not open network device >>>>> dpdk-p0 (No such device) >>>>> >>>>> Note that I've successfully managed to setup an OVS instance with two >>>>> DPDK >>>>> vhost-user ports (each one assigned to a QEMU VM): this is why I'm >>>>> confident my OVS is built and configured correctly to work with DPDK. >>>>> Also, >>>>> DPDK by itself is working as expected, I can use dpdk pktgen. >>>>> Moreover, OVS log shows that DPDK is recognized >>>>> >>>>> [...] >>>>> ovs|00005|dpdk|INFO|DPDK Enabled, initializing >>>>> [...] >>>>> >>>>> Note that I'm carefully following the documentation here >>>>> http://docs.openvswitch.org/en/latest/intro/install/dpdk/ and here >>>>> http://docs.openvswitch.org/en/latest/howto/dpdk/ >>>>> >>>>> More info: >>>>> >>>>> DPDK version: 16.11 >>>>> OVS version: 2.6.1, configured with --with-dpdk >>>>> Host: Archlinux >>>>> /proc/cmdline: BOOT_IMAGE=../vmlinuz-linux-vfio >>>>> root=UUID=1c777330-3f00-4ab8-8d5d-d4b2b5d860b0 rw intel_iommu=on >>>>> hugepages=2048 pci=assign-busses pcie_acs_override=downstream >>>>> initrd=../initramfs-linux-vfio.img >>>>> >>>>> Can anybody help? I'm at a loss here. >>>>> >>>>> Thanks, >>>>> Vincenzo >>>>> >>>> >>>> >>> >>> >>> -- >>> Vincenzo Maffione >>> >> >> >
