Sent from my iPhone
On Nov 14, 2017, at 2:59 AM, Johnson, Brian <[email protected]<mailto:[email protected]>> wrote: Do you have Intel VT-d enabled in the BIOS and intel_iommu=on iommu=pt in grub? I have had some issues with enabling this where other devices such as RAID controllers cause the OS to no boot after applying these settings. Removing the grub line and upgrading the RAID firmware fixed the issue. IOMMU<https://dpdk-guide.gitlab.io/dpdk-guide/setup/iommu.html#iommu> An input-output memory management unit (IOMMU) is required for safely driving DMA-capable hardware from userspace and because of that it is a prerequisite for using VFIO. Not all systems have one though, so you’ll need to check that the hardware supports it and that it is enabled in the BIOS settings (VT-d or Virtualization Technology for Directed I/Oon Intel systems) Finally, IOMMU needs to be excplitly enabled in the kernel as well. To do so, pass either intel_iommu=on (for Intel systems) or amd_iommu=on (for AMD systems) added to the kernel command line. In addition it is recommended to use iommu=pt option which improves IO performance for devices in the host. Once the system boots up, check the contents of /sys/kernel/iommu_groups/ directory. If it is non-empty, you have successfully set up IOMMU. To permanently add this to the kernel commandline, append it to GRUB_CMDLINE_LINUX in /etc/default/grub and then execute: # grub2-mkconfig -o /boot/grub2/grub.cfg Sent from my iPhone On Nov 14, 2017, at 12:38 AM, James Bensley <[email protected]<mailto:[email protected]>> wrote: Hi guys, Thank you all for your responses! I initially thought this was unlikely to be a Pktgen specific issue however I thought it worth mentioning I am using Pktgen just in case. On 14 November 2017 at 04:49, Muhammad Zain-ul-Abideen <[email protected]<mailto:[email protected]>> wrote: where is -p argument Do you mean upper case "-P"? Lower case “p” isn’t an option in “Pktgen –-help” output Pktgen once used -p Portman value and is not used anymore I still see that option being used in some command lines. On 13 November 2017 at 19:35, James Bensley <[email protected]<mailto:[email protected]>> wrote: $ sudo ./app/x86_64-native-linuxapp-gcc/pktgen -l 2-6 -n 1 -w 09:00.0 -w 09:00.1 -v -- -P -m [3-4].0 [5-6].1 On 13 November 2017 at 21:49, Wiles, Keith <[email protected]<mailto:[email protected]>> wrote: For the hard lockup problem try using testpmd application and see if that the same problem if not then it will be next week before I can look at. Let me know if testpmd works or not. Thanks for taking the time to reply whilst traveling Keith! Sadly I’m getting the same behaviour with testpmd: $ ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused= 0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused= Network devices using kernel driver =================================== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=igb_uio *Active* 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb unused=igb_uio 0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0 drv=tg3 unused=igb_uio 0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1 drv=tg3 unused=igb_uio 0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2 drv=tg3 unused=igb_uio *Active* 0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3 drv=tg3 unused=igb_uio $ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2 EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e << HARD LOCK-UP >> $ ./dpdk-devbind.py --status-dev net Network devices using DPDK-compatible driver ============================================ 0000:09:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=igb_uio 0000:09:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=vfio-pci unused=igb_uio Network devices using kernel driver =================================== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=igb_uio,vfio-pci *Active* 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eno2 drv=igb unused=igb_uio,vfio-pci 0000:06:00.0 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f0 drv=tg3 unused=igb_uio,vfio-pci 0000:06:00.1 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f1 drv=tg3 unused=igb_uio,vfio-pci 0000:06:00.2 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f2 drv=tg3 unused=igb_uio,vfio-pci *Active* 0000:06:00.3 'NetXtreme BCM5719 Gigabit Ethernet PCIe 1657' if=ens3f3 drv=tg3 unused=igb_uio,vfio-pci $ sudo ./testpmd -l 0-3 -n 4 -- -i --portmask=0x1 --nb-cores=2 EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:02:00.1 on NUMA socket 0 EAL: probe driver: 8086:1521 net_e1000_igb EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.0 failed to select IOMMU type EAL: Requested device 0000:09:00.0 cannot be used EAL: PCI device 0000:09:00.1 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.1 failed to select IOMMU type EAL: Requested device 0000:09:00.1 cannot be used EAL: No probed ethernet devices Interactive-mode selected USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 Done On 14 November 2017 at 06:29, Rosen, Rami <[email protected]<mailto:[email protected]>> wrote: Second, I think the root cause for not finding the ports is around this message ("failed to select IOMMU type") in: ... EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.0 failed to select IOMMU type EAL: Requested device 0000:09:00.0 cannot be used EAL: PCI device 0000:09:00.1 on NUMA socket 0 EAL: probe driver: 8086:1572 net_i40e EAL: 0000:09:00.1 failed to select IOMMU type EAL: Requested device 0000:09:00.1 cannot be used ... What does " find /sys/kernel/iommu_groups/ -type l" give ? Could it be that 0000:09:00.0 and 0000:09:00.1 belongs to an IOMMU group in which there are other devices ? To me (a layman) this looks correct, do you agree? $ find /sys/kernel/iommu_groups/ -type l | grep 09 /sys/kernel/iommu_groups/36/devices/0000:09:00.0 /sys/kernel/iommu_groups/37/devices/0000:09:00.1 $ find /sys/kernel/iommu_groups/ -type l | grep -E "36|37" /sys/kernel/iommu_groups/36/devices/0000:09:00.0 /sys/kernel/iommu_groups/37/devices/0000:09:00.1 Cheers, James.
