[dpdk-users] Vhost PMD Performance Doesn't Scale as Expected
I'm examining performance between two VMs connected with a vhost interface on DPDK 20.08 and testpmd. Each VM (client-0, server-0) has 4 VCPUs, 4 RX/TX queues per port, 4GB RAM, and runs 8 containers, each with an instance of qperf running the tcp_bw test. The configuration is targeting all CPU/memory activity for NUMA node 1. When I look at the cumulative throughput as I increase the number of qperf pairs I'm noticing that the performance doesn't appear to scale as I had hoped. Here's a table with some results: concurrent qperf pairs msg_size 1 2 4 8 8,19212.74 Gb/s 21.68 Gb/s 27.89 Gb/s 30.94 Gb/s 16,384 13.84 Gb/s 24.06 Gb/s 28.51 Gb/s 30.47 Gb/s 32,768 16.13 Gb/s 24.49 Gb/s 28.89 Gb/s 30.23 Gb/s 65,536 16.19 Gb/s 22.53 Gb/s 29.79 Gb/s 30.46 Gb/s 131,072 15.37 Gb/s 23.89 Gb/s 29.65 Gb/s 30.88 Gb/s 262,144 14.73 Gb/s 22.97 Gb/s 29.54 Gb/s 31.28 Gb/s 524,288 14.62 Gb/s 23.39 Gb/s 28.70 Gb/s 30.98 Gb/s Can anyone suggest a possible configuration change that might improve performance or is this generally what is expected? I was expecting performance to nearly double as I move from 1 to 2 to 4 queues. Even single queue performance is below Intel's published performance results (see https://fast.dpdk.org/doc/perf/DPDK_20_05_Intel_virtio_performance_report.pdf), though I was unable to get the vhost-switch example application to run due to an mbuf allocation error for the i40e PMD and had to revert to the testpmd app. Configuration details below. Dave /proc/cmdline: -- BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-147.el8.x86_64 root=/dev/mapper/rhel-root ro intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=64 crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet =1 nohz=on nohz_full=8-15,24-31 rcu_nocbs=8-15,24-31 tuned.non_isolcpus=00ff00ff intel_pstate=disable nosoftlockup testpmd command-line: - ~/src/dpdk/build/app/dpdk-testpmd -l 7,24-31 -n 4 --no-pci --vdev 'net_vhost0,iface=/tmp/vhost-dpdk-server-0,dequeue-zero-copy=1,tso=1,queues=4' --vdev 'net_vhost1,iface=/tmp/vhost-dpdk-client-0,dequeue-zero-copy=1,tso=1,queues=4' -- -i --nb-cores=8 --numa --rxq=4 --txq=4 testpmd forwarding core mapping: Start automatic packet forwarding io packet forwarding - ports=2 - cores=8 - streams=8 - NUMA support enabled, MP allocation mode: native Logical Core 24 (socket 1) forwards packets on 1 streams: RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01 Logical Core 25 (socket 1) forwards packets on 1 streams: RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00 Logical Core 26 (socket 1) forwards packets on 1 streams: RX P=0/Q=1 (socket 0) -> TX P=1/Q=1 (socket 0) peer=02:00:00:00:00:01 Logical Core 27 (socket 1) forwards packets on 1 streams: RX P=1/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00 Logical Core 28 (socket 1) forwards packets on 1 streams: RX P=0/Q=2 (socket 0) -> TX P=1/Q=2 (socket 0) peer=02:00:00:00:00:01 Logical Core 29 (socket 1) forwards packets on 1 streams: RX P=1/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00 Logical Core 30 (socket 1) forwards packets on 1 streams: RX P=0/Q=3 (socket 0) -> TX P=1/Q=3 (socket 0) peer=02:00:00:00:00:01 Logical Core 31 (socket 1) forwards packets on 1 streams: RX P=1/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00 io packet forwarding packets/burst=32 nb forwarding cores=8 - nb forwarding ports=2 port 0: RX queue number: 4 Tx queue number: 4 Rx offloads=0x0 Tx offloads=0x0 RX queue: 0 RX desc=0 - RX free threshold=0 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=0 - TX free threshold=0 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x0 - TX RS bit threshold=0 port 1: RX queue number: 4 Tx queue number: 4 Rx offloads=0x0 Tx offloads=0x0 RX queue: 0 RX desc=0 - RX free threshold=0 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=0 - TX free threshold=0 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x0 - TX RS bit threshold=0 lscpu: -- Architecture:x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 2 NUMA node(s):2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz Stepping:4 CPU MHz: 2400.075 BogoMIPS:4200.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache:
Re: [dpdk-users] DPDK application is hunged in eal_hugepage_info_init
On Wed, 12 Aug 2020 16:59:33 +0530 Jatin Sahu wrote: > Hi, > > We are using DPDK libraries with huge pages enabled. > We observed that the application rte_eal_init() function is stuck in > flock() call. > > Please advise how to resolve this. > > DPDK Version: dpdk-stable-18.11.6/ > > [root@loadtestsrv helloworld]# ./build/helloworld -l 0-3 -n 4 > EAL: Detected 72 lcore(s) > EAL: Detected 2 NUMA nodes > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > Stack Trace: > Thread 3 (Thread 0x7f783c34e700 (LWP 1034)): > #0 0x7f783c664113 in epoll_wait () from /lib64/libc.so.6 > #1 0x00558724 in eal_intr_thread_main () > #2 0x7f783c939dd5 in start_thread () from /lib64/libpthread.so.0 > #3 0x7f783c663b3d in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7f783bb4d700 (LWP 1035)): > #0 0x7f783c940bfd in recvmsg () from /lib64/libpthread.so.0 > #1 0x00565bae in mp_handle () > #2 0x7f783c939dd5 in start_thread () from /lib64/libpthread.so.0 > #3 0x7f783c663b3d in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7f783d66dc00 (LWP 1033)): > #0 0x7f783c655167 in flock () from /lib64/libc.so.6 > #1 0x0054e068 in eal_hugepage_info_init () > #2 0x0054ccfe in rte_eal_init () > #3 0x004712a6 in main () > [root@loadtestsrv user]# > > HugePage details: > > [root@loadtestsrv roamware]# cat /proc/meminfo | grep -i huge > AnonHugePages:112640 kB > HugePages_Total: 32 > HugePages_Free: 32 > HugePages_Rsvd:0 > HugePages_Surp:0 > Hugepagesize:1048576 kB > > [roamware@loadtestsrv ~]$ cat > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages > 512 > [roamware@loadtestsrv ~]$ cat > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages > 512 > > [root@loadtestsrv roamware]# cat /proc/mounts | grep -i huge > cgroup /sys/fs/cgroup/hugetlb cgroup > rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 > nodev /mnt/huge hugetlbfs rw,seclabel,relatime 0 0 > nodev /mnt/huge hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 > nodev /mnt/huge hugetlbfs rw,seclabel,relatime 0 0 > > Regards, > Jatin Maybe in this case Flock blocks if other process has the file locked? Is there another process which has flocked the file? Or you have a buggy kernel?
Re: [dpdk-users] Resolved Issues per release
12/08/2020 15:32, Jim Holland (jimholla): > The release notes at https://doc.dpdk.org/guides-20.02/rel_notes/index.html > don't list the resolved issues/bugs. Is there a way to find what bugs are > resolved for each dpdk release? When looking at the git history, every commit fixing a bug should contain a line starting with "Fixes:" identifying the root cause. Catching such commits will give you all details of bugs fixed.
[dpdk-users] Resolved Issues per release
The release notes at https://doc.dpdk.org/guides-20.02/rel_notes/index.html don't list the resolved issues/bugs. Is there a way to find what bugs are resolved for each dpdk release? Thanks...Jim
[dpdk-users] DPDK application is hunged in eal_hugepage_info_init
Hi, We are using DPDK libraries with huge pages enabled. We observed that the application rte_eal_init() function is stuck in flock() call. Please advise how to resolve this. DPDK Version: dpdk-stable-18.11.6/ [root@loadtestsrv helloworld]# ./build/helloworld -l 0-3 -n 4 EAL: Detected 72 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket Stack Trace: Thread 3 (Thread 0x7f783c34e700 (LWP 1034)): #0 0x7f783c664113 in epoll_wait () from /lib64/libc.so.6 #1 0x00558724 in eal_intr_thread_main () #2 0x7f783c939dd5 in start_thread () from /lib64/libpthread.so.0 #3 0x7f783c663b3d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f783bb4d700 (LWP 1035)): #0 0x7f783c940bfd in recvmsg () from /lib64/libpthread.so.0 #1 0x00565bae in mp_handle () #2 0x7f783c939dd5 in start_thread () from /lib64/libpthread.so.0 #3 0x7f783c663b3d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f783d66dc00 (LWP 1033)): #0 0x7f783c655167 in flock () from /lib64/libc.so.6 #1 0x0054e068 in eal_hugepage_info_init () #2 0x0054ccfe in rte_eal_init () #3 0x004712a6 in main () [root@loadtestsrv user]# HugePage details: [root@loadtestsrv roamware]# cat /proc/meminfo | grep -i huge AnonHugePages:112640 kB HugePages_Total: 32 HugePages_Free: 32 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize:1048576 kB [roamware@loadtestsrv ~]$ cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages 512 [roamware@loadtestsrv ~]$ cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages 512 [root@loadtestsrv roamware]# cat /proc/mounts | grep -i huge cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 nodev /mnt/huge hugetlbfs rw,seclabel,relatime 0 0 nodev /mnt/huge hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 nodev /mnt/huge hugetlbfs rw,seclabel,relatime 0 0 Regards, Jatin
