Hi I am trying to run latest DPDK on SR-IOV interface in VM environment. Interfaces are detected well in VM environment.
When I started traffic through testpmd, it indicates both tx and rx are zero. I tried debugging. Wondering if you could point me some steps which I might be missing? Thanks in advance. Here are the steps that I followed. SR-IOV interfaces are detected in VM environment. Lspci indicate below interfaces in VM environment and ethtool indicates links are up and link speeds are fine. 00:08.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 00:09.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) root at regress-KVM:/dev/abc/dpdk/tools# ethtool eth2 Settings for eth2: Supported ports: [ ] Supported link modes: 10000baseT/Full Supported pause frame use: No Supports auto-negotiation: No Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Speed: 10000Mb/s Duplex: Full Port: Other PHYAD: 0 Transceiver: Unknown! Auto-negotiation: off Current message level: 0x00000007 (7) drv probe link Link detected: yes root at regress-KVM:/dev/abc/dpdk/tools# ethtool eth3 Settings for eth3: Supported ports: [ ] Supported link modes: 10000baseT/Full Supported pause frame use: No Supports auto-negotiation: No Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Speed: 10000Mb/s Duplex: Full Port: Other PHYAD: 0 Transceiver: Unknown! Auto-negotiation: off Current message level: 0x00000007 (7) drv probe link Link detected: yes Loaded uio module. root at regress-KVM:/dev/abc/dpdk/tools# modprobe uio root at regress-KVM:/dev/abc/dpdk/tools# insmod ../x86_64-native-linuxapp-gcc/kmod/igb_uio.ko Bound interfaces to igb_uio. root at regress-KVM:/dev/abc/dpdk/tools# python dpdk_nic_bind.py --bind=igb_uio 00:09.0 root at regress-KVM:/dev/abc/dpdk/tools# python dpdk_nic_bind.py --bind=igb_uio 00:08.0 Verified that interfaces are bound to igb_uio. root at regress-KVM:/dev/abc/dpdk/tools# python dpdk_nic_bind.py --status Network devices using DPDK-compatible driver ============================================ 0000:00:08.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=vfio-pci 0000:00:09.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=vfio-pci Network devices using kernel driver =================================== 0000:00:03.0 'Virtio network device' if= drv=virtio-pci unused=igb_uio,vfio-pci 0000:00:07.0 '82599 Ethernet Controller Virtual Function' if=eth1 drv=ixgbevf unused=igb_uio,vfio-pci Other network devices ===================== <none> root at regress-KVM:/dev/abc/dpdk/tools# Built test-pmd and executed and links detected. root at regress-KVM:/dev/abc/dpdk/app/test-pmd# ./testpmd -c 0xf -n 4 -- -i --nb-cores=2 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Detected lcore 2 as core 0 on socket 0 EAL: Detected lcore 3 as core 0 on socket 0 EAL: Support maximum 64 logical core(s) by configuration. EAL: Detected 4 lcore(s) EAL: unsupported IOMMU type! EAL: VFIO support could not be initialized EAL: Setting up memory... EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd8e1200000 (size = 0x200000) EAL: Ask a virtual area of 0x7f000000 bytes EAL: Virtual area found at 0x7fd862000000 (size = 0x7f000000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd861c00000 (size = 0x200000) EAL: Ask a virtual area of 0x400000 bytes EAL: Virtual area found at 0x7fd861600000 (size = 0x400000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd861200000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd860e00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd860a00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd860600000 (size = 0x200000) EAL: Requesting 1024 pages of size 2MB from socket 0 EAL: TSC frequency is ~1999999 KHz EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: Master core 0 is ready (tid=e281a840) EAL: Core 3 is ready (tid=5e9f0700) EAL: Core 2 is ready (tid=5f1f1700) EAL: Core 1 is ready (tid=5f9f2700) EAL: PCI device 0000:00:03.0 on NUMA socket -1 EAL: probe driver: 1af4:1000 rte_virtio_pmd EAL: 0000:00:03.0 not managed by UIO driver, skipping EAL: PCI device 0000:00:07.0 on NUMA socket -1 EAL: probe driver: 8086:10ed rte_ixgbevf_pmd EAL: 0000:00:07.0 not managed by UIO driver, skipping EAL: PCI device 0000:00:08.0 on NUMA socket -1 EAL: probe driver: 8086:10ed rte_ixgbevf_pmd EAL: PCI memory mapped at 0x7fd8e2828000 EAL: PCI memory mapped at 0x7fd8e2824000 EAL: PCI device 0000:00:09.0 on NUMA socket -1 EAL: probe driver: 8086:10ed rte_ixgbevf_pmd EAL: PCI memory mapped at 0x7fd8e2820000 EAL: PCI memory mapped at 0x7fd8e27e0000 Interactive-mode selected Configuring Port 0 (socket 0) Port 0: 52:54:00:E2:90:42 Configuring Port 1 (socket 0) Port 1: 52:54:00:CA:0D:17 Checking link statuses... Port 0 Link Up - speed 10000 Mbps - full-duplex Port 1 Link Up - speed 10000 Mbps - full-duplex Done testpmd> When I start and stop traffic, stats show that zero traffic xmitted. testpmd> start io packet forwarding - CRC stripping disabled - packets/burst=32 nb forwarding cores=2 - nb forwarding ports=2 RX queues=1 - RX desc=128 - RX free threshold=0 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=1 - TX desc=512 - TX free threshold=0 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=0 - TXQ flags=0x0 testpmd> stop Telling cores to stop... Waiting for lcores to finish... ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 0 TX-dropped: 0 TX-total: 0 ---------------------------------------------------------------------------- ---------------------- Forward statistics for port 1 ---------------------- RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 0 TX-dropped: 0 TX-total: 0 ---------------------------------------------------------------------------- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 0 TX-dropped: 0 TX-total: 0 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Done. testpmd> I debugged this to some extent and below are my observations. For some reason I am not able to compile old dpdk code from the website. Can you point me to your code? When I start traffic in testpmd, I see two thread taking 100%cpu utilization. Ideally I would expect one thread to be rx and another to be tx. One thread does the tx and while other thread does rx, is my understanding correct? But what I see is both the treads are in rx path!? Is that expected? Am I missing something? I believe you must have automated these steps in scripts for vmx. Can you please point me to them? PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8496 root 20 0 2291m 5844 5532 R 99.9 0.0 2:27.13 testpmd 8495 root 20 0 2291m 5844 5532 R 99.8 0.0 2:27.12 testpmd 8896 root 20 0 24972 1712 1124 R 0.3 0.0 0:05.60 top 1 root 20 0 27352 3200 1416 S 0.0 0.0 0:01.42 init (gdb) info threads Id Target Id Frame 5 Thread 0x7f7d029f3700 (LWP 8494) "testpmd" 0x00007f7d84234083 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 * 4 Thread 0x7f7d021f2700 (LWP 8495) "testpmd" rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f7d021f0d10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 3 Thread 0x7f7d019f1700 (LWP 8496) "testpmd" rte_eth_rx_burst (port_id=2 '\002', queue_id=0, rx_pkts=0x7f7d019efd10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 2 Thread 0x7f7d011f0700 (LWP 8497) "testpmd" 0x00007f7d8450fc9d in read () at ../sysdeps/unix/syscall-template.S:81 1 Thread 0x7f7d85037840 (LWP 8493) "testpmd" 0x00007f7d8450fc9d in read () at ../sysdeps/unix/syscall-template.S:81 (gdb) thread 4 [Switching to thread 4 (Thread 0x7f7d021f2700 (LWP 8495))] #0 rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f7d021f0d10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 2304} (gdb) bt #0 rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f7d021f0d10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 #1 0x0000000000438464 in pkt_burst_io_forward (fs=0x7f7d052f6200) at iofwd.c:97 #2 0x0000000000427596 in run_pkt_fwd_on_lcore (fc=0x7f7d052f7dc0, pkt_fwd=0x43841f <pkt_burst_io_forward>) at testpmd.c:952 #3 0x00000000004275df in start_pkt_forward_on_core (fwd_arg=0x7f7d052f7dc0) at testpmd.c:959 #4 0x0000000000498759 in eal_thread_loop (arg=<optimized out>) at /dev/abc/dpdk/lib/librte_eal/linuxapp/eal/eal_thread.c:224 #5 0x00007f7d84508f6e in start_thread (arg=0x7f7d021f2700) at pthread_create.c:311 #6 0x00007f7d842339cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb) thread 3 [Switching to thread 3 (Thread 0x7f7d019f1700 (LWP 8496))] #0 rte_eth_rx_burst (port_id=2 '\002', queue_id=0, rx_pkts=0x7f7d019efd10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 2304} (gdb) bt #0 rte_eth_rx_burst (port_id=2 '\002', queue_id=0, rx_pkts=0x7f7d019efd10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 #1 0x0000000000438464 in pkt_burst_io_forward (fs=0x7f7d052f6100) at iofwd.c:97 #2 0x0000000000427596 in run_pkt_fwd_on_lcore (fc=0x7f7d052f7d40, pkt_fwd=0x43841f <pkt_burst_io_forward>) at testpmd.c:952 #3 0x00000000004275df in start_pkt_forward_on_core (fwd_arg=0x7f7d052f7d40) at testpmd.c:959 #4 0x0000000000498759 in eal_thread_loop (arg=<optimized out>) at /dev/abc/dpdk/lib/librte_eal/linuxapp/eal/eal_thread.c:224 #5 0x00007f7d84508f6e in start_thread (arg=0x7f7d019f1700) at pthread_create.c:311 #6 0x00007f7d842339cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb)