Re: [ovs-discuss] [ovs-dpdk] bandwidth issue of vhostuserclient virtio ovs-dpdk
On 29/11/2018 08:24, LIU Yulong wrote: > Hi, > > We recently tested ovs-dpdk, but we met some bandwidth issue. The bandwidth > from VM to VM was not close to the physical NIC, it's about 4.3Gbps on a > 10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3 test can easily > reach 9.3Gbps. We enabled the virtio multiqueue for all guest VMs. In the > dpdk vhostuser guest, we noticed that the interrupts are centralized to > only one queue. But for no dpdk VM, interrupts can hash to all queues. > For those dpdk vhostuser VMs, we also noticed that the PMD usages were > also centralized to one no matter server(tx) or client(rx). And no matter > one PMD or multiple PMDs, this behavior always exists. > > Furthuremore, my colleague add some systemtap hook on the openvswitch > function, he found something interesting. The function > __netdev_dpdk_vhost_send will send all the packets to one virtionet-queue. > Seems that there are some algorithm/hash table/logic does not do the hash > very well. > Hi, When you say "no dpdk VMs", you mean that within your VM you're relying on the Kernel to get the packets, using virtio-net. And when you say "dpdk vhostuser guest", you mean you're using DPDK inside the VM to get the packets. Is this correct? If so, could you also tell us which DPDK app you're using inside of those VMs? Is it testpmd? If so, how are you setting the `--rxq` and `--txq` args? Otherwise, how are you setting those in your app when initializing DPDK? The information below is useful in telling us how you're setting your configurations in OvS, but we are still missing the configurations inside the VM. This should help us in getting more information, Tiago. > So I'd like to find some help from the community. Maybe I'm missing some > configrations. > > Thanks. > > > Here is the list of the environment and some configrations: > # uname -r > 3.10.0-862.11.6.el7.x86_64 > # rpm -qa|grep dpdk > dpdk-17.11-11.el7.x86_64 > # rpm -qa|grep openvswitch > openvswitch-2.9.0-3.el7.x86_64 > # ovs-vsctl list open_vswitch > _uuid : a6a3d9eb-28a8-4bf0-a8b4-94577b5ffe5e > bridges : [531e4bea-ce12-402a-8a07-7074c31b978e, > 5c1675e2-5408-4c1f-88bc-6d9c9b932d47] > cur_cfg : 1305 > datapath_types : [netdev, system] > db_version : "7.15.1" > external_ids : {hostname="cq01-compute-10e112e5e140", > rundir="/var/run/openvswitch", > system-id="e2cc84fe-a3c8-455f-8c64-260741c141ee"} > iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, > geneve, gre, internal, lisp, patch, stt, system, tap, vxlan] > manager_options : [43803994-272b-49cb-accc-ab672d1eefc8] > next_cfg : 1305 > other_config : {dpdk-init="true", dpdk-lcore-mask="0x1", > dpdk-socket-mem="1024,1024", pmd-cpu-mask="0x10", > vhost-iommu-support="true"} > ovs_version : "2.9.0" > ssl : [] > statistics : {} > system_type : centos > system_version : "7" > # lsmod |grep vfio > vfio_pci 41312 2 > vfio_iommu_type1 22300 1 > vfio 32695 7 vfio_iommu_type1,vfio_pci > irqbypass 13503 23 kvm,vfio_pci > > # ovs-appctl dpif/show > netdev@ovs-netdev: hit:759366335 missed:754283 > br-ex: > bond1108 4/6: (tap) > br-ex 65534/3: (tap) > nic-10G-1 5/4: (dpdk: configured_rx_queues=8, > configured_rxq_descriptors=2048, configured_tx_queues=2, > configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, > requested_rxq_descriptors=2048, requested_tx_queues=2, > requested_txq_descriptors=2048, rx_csum_offload=true) > nic-10G-2 6/5: (dpdk: configured_rx_queues=8, > configured_rxq_descriptors=2048, configured_tx_queues=2, > configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, > requested_rxq_descriptors=2048, requested_tx_queues=2, > requested_txq_descriptors=2048, rx_csum_offload=true) > phy-br-ex 3/none: (patch: peer=int-br-ex) > br-int: > br-int 65534/2: (tap) > int-br-ex 1/none: (patch: peer=phy-br-ex) > vhu76f9a623-9f 2/1: (dpdkvhostuserclient: configured_rx_queues=8, > configured_tx_queues=8, mtu=1500, requested_rx_queues=8, > requested_tx_queues=8) > > # ovs-appctl dpctl/show -s > netdev@ovs-netdev: > lookups: hit:759366335 missed:754283 lost:72 > flows: 186 > port 0: ovs-netdev (tap) > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 aborted:0 carrier:0 > collisions:0 > RX bytes:0 TX bytes:0 > port 1: vhu76f9a623-9f (dpdkvhostuserclient: configured_rx_queues=8, > configured_tx_queues=8, mtu=1500, requested_rx_queues=8, > requested_tx_queues=8) > RX packets:718391758 errors:0 dropped:0 overruns:? frame:? > TX packets:30372410 errors:? dropped:719200 aborted:? carrier:? > collisions:? > RX bytes:1086995317051 (1012.3 GiB) TX bytes:2024893540 (1.9 GiB) > port 2: br-int (tap) > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:1393992 errors:0 dropped:4 aborted:0 carrier:0 > collisions:0 > RX bytes:0 TX bytes:2113
[ovs-discuss] [ovs-dpdk] bandwidth issue of vhostuserclient virtio ovs-dpdk
Hi, We recently tested ovs-dpdk, but we met some bandwidth issue. The bandwidth from VM to VM was not close to the physical NIC, it's about 4.3Gbps on a 10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3 test can easily reach 9.3Gbps. We enabled the virtio multiqueue for all guest VMs. In the dpdk vhostuser guest, we noticed that the interrupts are centralized to only one queue. But for no dpdk VM, interrupts can hash to all queues. For those dpdk vhostuser VMs, we also noticed that the PMD usages were also centralized to one no matter server(tx) or client(rx). And no matter one PMD or multiple PMDs, this behavior always exists. Furthuremore, my colleague add some systemtap hook on the openvswitch function, he found something interesting. The function __netdev_dpdk_vhost_send will send all the packets to one virtionet-queue. Seems that there are some algorithm/hash table/logic does not do the hash very well. So I'd like to find some help from the community. Maybe I'm missing some configrations. Thanks. Here is the list of the environment and some configrations: # uname -r 3.10.0-862.11.6.el7.x86_64 # rpm -qa|grep dpdk dpdk-17.11-11.el7.x86_64 # rpm -qa|grep openvswitch openvswitch-2.9.0-3.el7.x86_64 # ovs-vsctl list open_vswitch _uuid : a6a3d9eb-28a8-4bf0-a8b4-94577b5ffe5e bridges : [531e4bea-ce12-402a-8a07-7074c31b978e, 5c1675e2-5408-4c1f-88bc-6d9c9b932d47] cur_cfg : 1305 datapath_types : [netdev, system] db_version : "7.15.1" external_ids: {hostname="cq01-compute-10e112e5e140", rundir="/var/run/openvswitch", system-id="e2cc84fe-a3c8-455f-8c64-260741c141ee"} iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, geneve, gre, internal, lisp, patch, stt, system, tap, vxlan] manager_options : [43803994-272b-49cb-accc-ab672d1eefc8] next_cfg: 1305 other_config: {dpdk-init="true", dpdk-lcore-mask="0x1", dpdk-socket-mem="1024,1024", pmd-cpu-mask="0x10", vhost-iommu-support="true"} ovs_version : "2.9.0" ssl : [] statistics : {} system_type : centos system_version : "7" # lsmod |grep vfio vfio_pci 41312 2 vfio_iommu_type1 22300 1 vfio 32695 7 vfio_iommu_type1,vfio_pci irqbypass 13503 23 kvm,vfio_pci # ovs-appctl dpif/show netdev@ovs-netdev: hit:759366335 missed:754283 br-ex: bond1108 4/6: (tap) br-ex 65534/3: (tap) nic-10G-1 5/4: (dpdk: configured_rx_queues=8, configured_rxq_descriptors=2048, configured_tx_queues=2, configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, requested_rxq_descriptors=2048, requested_tx_queues=2, requested_txq_descriptors=2048, rx_csum_offload=true) nic-10G-2 6/5: (dpdk: configured_rx_queues=8, configured_rxq_descriptors=2048, configured_tx_queues=2, configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, requested_rxq_descriptors=2048, requested_tx_queues=2, requested_txq_descriptors=2048, rx_csum_offload=true) phy-br-ex 3/none: (patch: peer=int-br-ex) br-int: br-int 65534/2: (tap) int-br-ex 1/none: (patch: peer=phy-br-ex) vhu76f9a623-9f 2/1: (dpdkvhostuserclient: configured_rx_queues=8, configured_tx_queues=8, mtu=1500, requested_rx_queues=8, requested_tx_queues=8) # ovs-appctl dpctl/show -s netdev@ovs-netdev: lookups: hit:759366335 missed:754283 lost:72 flows: 186 port 0: ovs-netdev (tap) RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 aborted:0 carrier:0 collisions:0 RX bytes:0 TX bytes:0 port 1: vhu76f9a623-9f (dpdkvhostuserclient: configured_rx_queues=8, configured_tx_queues=8, mtu=1500, requested_rx_queues=8, requested_tx_queues=8) RX packets:718391758 errors:0 dropped:0 overruns:? frame:? TX packets:30372410 errors:? dropped:719200 aborted:? carrier:? collisions:? RX bytes:1086995317051 (1012.3 GiB) TX bytes:2024893540 (1.9 GiB) port 2: br-int (tap) RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:1393992 errors:0 dropped:4 aborted:0 carrier:0 collisions:0 RX bytes:0 TX bytes:2113616736 (2.0 GiB) port 3: br-ex (tap) RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:6660091 errors:0 dropped:967 aborted:0 carrier:0 collisions:0 RX bytes:0 TX bytes:2451440870 (2.3 GiB) port 4: nic-10G-1 (dpdk: configured_rx_queues=8, configured_rxq_descriptors=2048, configured_tx_queues=2, configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, requested_rxq_descriptors=2048, requested_tx_queues=2, requested_txq_descriptors=2048, rx_csum_offload=true) RX packets:36409466 errors:0 dropped:0 overruns:? frame:? TX packets:718371472 errors:0 dropped:20276 aborted:? carrier:? collisions:? RX bytes:2541593983 (2.4 GiB) TX bytes:1089838136919 (1015.0 GiB) port 5: nic-10G-2 (dpdk: configured_rx_queues=8, configured_rxq_descriptors=2048, configured_tx_queues=2, configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, requested_rxq_descriptors=2048, requested_tx_queues=2, requested_txq_descrip