Hi, Thanks for the reply, please see my inline comments below.
On Thu, Nov 29, 2018 at 6:00 PM Lam, Tiago <tiago....@intel.com> wrote: > On 29/11/2018 08:24, LIU Yulong wrote: > > Hi, > > > > We recently tested ovs-dpdk, but we met some bandwidth issue. The > bandwidth > > from VM to VM was not close to the physical NIC, it's about 4.3Gbps on a > > 10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3 test can easily > > reach 9.3Gbps. We enabled the virtio multiqueue for all guest VMs. In the > > dpdk vhostuser guest, we noticed that the interrupts are centralized to > > only one queue. But for no dpdk VM, interrupts can hash to all queues. > > For those dpdk vhostuser VMs, we also noticed that the PMD usages were > > also centralized to one no matter server(tx) or client(rx). And no matter > > one PMD or multiple PMDs, this behavior always exists. > > > > Furthuremore, my colleague add some systemtap hook on the openvswitch > > function, he found something interesting. The function > > __netdev_dpdk_vhost_send will send all the packets to one > virtionet-queue. > > Seems that there are some algorithm/hash table/logic does not do the hash > > very well. > > > > Hi, > > When you say "no dpdk VMs", you mean that within your VM you're relying > on the Kernel to get the packets, using virtio-net. And when you say > "dpdk vhostuser guest", you mean you're using DPDK inside the VM to get > the packets. Is this correct? Sorry for the inaccurate description. I'm really new to DPDK. No DPDK inside VM, all these settings are for host only. (`host` means the hypervisor physical machine in the perspective of virtualization. On the other hand `guest` means the virtual machine.) "no dpdk VMs" means the host does not setup DPDK (ovs is working in traditional way), the VMs were boot on that. Maybe a new name `VMs-on-NO-DPDK-host`? If so, could you also tell us which DPDK app you're using inside of > those VMs? Is it testpmd? If so, how are you setting the `--rxq` and > `--txq` args? Otherwise, how are you setting those in your app when > initializing DPDK? > Inside VM, there is no DPDK app, VM kernel also does not set any config related to DPDK. `iperf3` is the tool for bandwidth testing. The information below is useful in telling us how you're setting your > configurations in OvS, but we are still missing the configurations > inside the VM. > > This should help us in getting more information, > > Maybe you have noticed that, we only setup one PMD in the pasted configurations. But VM has 8 queues. Should the pmd quantity match the queues? Tiago. > > > So I'd like to find some help from the community. Maybe I'm missing some > > configrations. > > > > Thanks. > > > > > > Here is the list of the environment and some configrations: > > # uname -r > > 3.10.0-862.11.6.el7.x86_64 > > # rpm -qa|grep dpdk > > dpdk-17.11-11.el7.x86_64 > > # rpm -qa|grep openvswitch > > openvswitch-2.9.0-3.el7.x86_64 > > # ovs-vsctl list open_vswitch > > _uuid : a6a3d9eb-28a8-4bf0-a8b4-94577b5ffe5e > > bridges : [531e4bea-ce12-402a-8a07-7074c31b978e, > > 5c1675e2-5408-4c1f-88bc-6d9c9b932d47] > > cur_cfg : 1305 > > datapath_types : [netdev, system] > > db_version : "7.15.1" > > external_ids : {hostname="cq01-compute-10e112e5e140", > > rundir="/var/run/openvswitch", > > system-id="e2cc84fe-a3c8-455f-8c64-260741c141ee"} > > iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, > > geneve, gre, internal, lisp, patch, stt, system, tap, vxlan] > > manager_options : [43803994-272b-49cb-accc-ab672d1eefc8] > > next_cfg : 1305 > > other_config : {dpdk-init="true", dpdk-lcore-mask="0x1", > > dpdk-socket-mem="1024,1024", pmd-cpu-mask="0x100000", > > vhost-iommu-support="true"} > > ovs_version : "2.9.0" > > ssl : [] > > statistics : {} > > system_type : centos > > system_version : "7" > > # lsmod |grep vfio > > vfio_pci 41312 2 > > vfio_iommu_type1 22300 1 > > vfio 32695 7 vfio_iommu_type1,vfio_pci > > irqbypass 13503 23 kvm,vfio_pci > > > > # ovs-appctl dpif/show > > netdev@ovs-netdev: hit:759366335 missed:754283 > > br-ex: > > bond1108 4/6: (tap) > > br-ex 65534/3: (tap) > > nic-10G-1 5/4: (dpdk: configured_rx_queues=8, > > configured_rxq_descriptors=2048, configured_tx_queues=2, > > configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, > > requested_rxq_descriptors=2048, requested_tx_queues=2, > > requested_txq_descriptors=2048, rx_csum_offload=true) > > nic-10G-2 6/5: (dpdk: configured_rx_queues=8, > > configured_rxq_descriptors=2048, configured_tx_queues=2, > > configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, > > requested_rxq_descriptors=2048, requested_tx_queues=2, > > requested_txq_descriptors=2048, rx_csum_offload=true) > > phy-br-ex 3/none: (patch: peer=int-br-ex) > > br-int: > > br-int 65534/2: (tap) > > int-br-ex 1/none: (patch: peer=phy-br-ex) > > vhu76f9a623-9f 2/1: (dpdkvhostuserclient: configured_rx_queues=8, > > configured_tx_queues=8, mtu=1500, requested_rx_queues=8, > > requested_tx_queues=8) > > > > # ovs-appctl dpctl/show -s > > netdev@ovs-netdev: > > lookups: hit:759366335 missed:754283 lost:72 > > flows: 186 > > port 0: ovs-netdev (tap) > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:0 errors:0 dropped:0 aborted:0 carrier:0 > > collisions:0 > > RX bytes:0 TX bytes:0 > > port 1: vhu76f9a623-9f (dpdkvhostuserclient: configured_rx_queues=8, > > configured_tx_queues=8, mtu=1500, requested_rx_queues=8, > > requested_tx_queues=8) > > RX packets:718391758 errors:0 dropped:0 overruns:? frame:? > > TX packets:30372410 errors:? dropped:719200 aborted:? carrier:? > > collisions:? > > RX bytes:1086995317051 (1012.3 GiB) TX bytes:2024893540 (1.9 GiB) > > port 2: br-int (tap) > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:1393992 errors:0 dropped:4 aborted:0 carrier:0 > > collisions:0 > > RX bytes:0 TX bytes:2113616736 (2.0 GiB) > > port 3: br-ex (tap) > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:6660091 errors:0 dropped:967 aborted:0 carrier:0 > > collisions:0 > > RX bytes:0 TX bytes:2451440870 (2.3 GiB) > > port 4: nic-10G-1 (dpdk: configured_rx_queues=8, > > configured_rxq_descriptors=2048, configured_tx_queues=2, > > configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, > > requested_rxq_descriptors=2048, requested_tx_queues=2, > > requested_txq_descriptors=2048, rx_csum_offload=true) > > RX packets:36409466 errors:0 dropped:0 overruns:? frame:? > > TX packets:718371472 errors:0 dropped:20276 aborted:? carrier:? > > collisions:? > > RX bytes:2541593983 (2.4 GiB) TX bytes:1089838136919 (1015.0 GiB) > > port 5: nic-10G-2 (dpdk: configured_rx_queues=8, > > configured_rxq_descriptors=2048, configured_tx_queues=2, > > configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8, > > requested_rxq_descriptors=2048, requested_tx_queues=2, > > requested_txq_descriptors=2048, rx_csum_offload=true) > > RX packets:5319466 errors:0 dropped:0 overruns:? frame:? > > TX packets:0 errors:0 dropped:0 aborted:? carrier:? > > collisions:? > > RX bytes:344903551 (328.9 MiB) TX bytes:0 > > port 6: bond1108 (tap) > > RX packets:228 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:5460 errors:0 dropped:18 aborted:0 carrier:0 > > collisions:0 > > RX bytes:21459 (21.0 KiB) TX bytes:341087 (333.1 KiB) > > > > # ovs-appctl dpif-netdev/pmd-stats-show > > pmd thread numa_id 0 core_id 20: > > packets received: 760120690 > > packet recirculations: 0 > > avg. datapath passes per packet: 1.00 > > emc hits: 750787577 > > megaflow hits: 8578758 > > avg. subtable lookups per megaflow hit: 1.05 > > miss with success upcall: 754283 > > miss with failed upcall: 72 > > avg. packets per output batch: 2.21 > > idle cycles: 210648140144730 (99.13%) > > processing cycles: 1846745927216 (0.87%) > > avg cycles per packet: 279554.14 (212494886071946/760120690) > > avg processing cycles per packet: 2429.54 (1846745927216/760120690) > > main thread: > > packets received: 0 > > packet recirculations: 0 > > avg. datapath passes per packet: 0.00 > > emc hits: 0 > > megaflow hits: 0 > > avg. subtable lookups per megaflow hit: 0.00 > > miss with success upcall: 0 > > miss with failed upcall: 0 > > avg. packets per output batch: 0.00 > > > > # ovs-appctl dpif-netdev/pmd-rxq-show > > pmd thread numa_id 0 core_id 20: > > isolated : false > > port: nic-10G-1 queue-id: 0pmd usage: 0 % > > port: nic-10G-1 queue-id: 1pmd usage: 0 % > > port: nic-10G-1 queue-id: 2pmd usage: 0 % > > port: nic-10G-1 queue-id: 3pmd usage: 0 % > > port: nic-10G-1 queue-id: 4pmd usage: 0 % > > port: nic-10G-1 queue-id: 5pmd usage: 0 % > > port: nic-10G-1 queue-id: 6pmd usage: 0 % > > port: nic-10G-1 queue-id: 7pmd usage: 0 % > > port: nic-10G-2 queue-id: 0pmd usage: 0 % > > port: nic-10G-2 queue-id: 1pmd usage: 0 % > > port: nic-10G-2 queue-id: 2pmd usage: 0 % > > port: nic-10G-2 queue-id: 3pmd usage: 0 % > > port: nic-10G-2 queue-id: 4pmd usage: 0 % > > port: nic-10G-2 queue-id: 5pmd usage: 0 % > > port: nic-10G-2 queue-id: 6pmd usage: 0 % > > port: nic-10G-2 queue-id: 7pmd usage: 0 % > > port: vhu76f9a623-9f queue-id: 0pmd usage: 0 % > > port: vhu76f9a623-9f queue-id: 1pmd usage: 0 % > > port: vhu76f9a623-9f queue-id: 2pmd usage: 0 % > > port: vhu76f9a623-9f queue-id: 3pmd usage: 0 % > > port: vhu76f9a623-9f queue-id: 4pmd usage: 0 % > > port: vhu76f9a623-9f queue-id: 5pmd usage: 0 % > > port: vhu76f9a623-9f queue-id: 6pmd usage: 0 % > > port: vhu76f9a623-9f queue-id: 7pmd usage: 0 % > > > > > > # virsh dumpxml instance-5c5191ff-c1a2-4429-9a8b-93ddd939583d > > ... > > <interface type='vhostuser'> > > <mac address='fa:16:3e:77:ab:fb'/> > > <source type='unix' path='/var/lib/vhost_sockets/vhu76f9a623-9f' > > mode='server'/> > > <target dev='vhu76f9a623-9f'/> > > <model type='virtio'/> > > <driver name='vhost' queues='8'/> > > <alias name='net0'/> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > > function='0x0'/> > > </interface> > > ... > > > > # ovs-vsctl show > > a6a3d9eb-28a8-4bf0-a8b4-94577b5ffe5e > > Manager "ptcp:6640:127.0.0.1" > > is_connected: true > > Bridge br-int > > Controller "tcp:127.0.0.1:6633 <http://127.0.0.1:6633>" > > is_connected: true > > fail_mode: secure > > Port int-br-ex > > Interface int-br-ex > > type: patch > > options: {peer=phy-br-ex} > > Port br-int > > Interface br-int > > type: internal > > Port "vhu76f9a623-9f" > > tag: 1 > > Interface "vhu76f9a623-9f" > > type: dpdkvhostuserclient > > options: {n_rxq="8", > > vhost-server-path="/var/lib/vhost_sockets/vhu76f9a623-9f"} > > Bridge br-ex > > Controller "tcp:127.0.0.1:6633 <http://127.0.0.1:6633>" > > is_connected: true > > fail_mode: secure > > Port dpdkbond > > Interface "nic-10G-1" > > type: dpdk > > options: {dpdk-devargs="0000:01:00.0", n_rxq="8", > n_txq="8"} > > Interface "nic-10G-2" > > type: dpdk > > options: {dpdk-devargs="0000:05:00.1", n_rxq="8", > n_txq="8"} > > Port phy-br-ex > > Interface phy-br-ex > > type: patch > > options: {peer=int-br-ex} > > Port br-ex > > Interface br-ex > > type: internal > > > > # numactl --hardware > > available: 2 nodes (0-1) > > node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 > > node 0 size: 130978 MB > > node 0 free: 7539 MB > > node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 > > node 1 size: 131072 MB > > node 1 free: 6886 MB > > node distances: > > node 0 1 > > 0: 10 21 > > 1: 21 10 > > > > # grep HugePages_ /proc/meminfo > > HugePages_Total: 232 > > HugePages_Free: 10 > > HugePages_Rsvd: 0 > > HugePages_Surp: 0 > > > > > > # cat /proc/cmdline > > BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 > > root=UUID=220ee106-5e00-4809-91a0-641e045a4c21 ro > > intel_idle.max_cstate=0 crashkernel=auto rhgb quiet > > default_hugepagesz=1G hugepagesz=1G hugepages=232 iommu=pt intel_iommu=on > > > > > > Best regards, > > LIU Yulong > > > > _______________________________________________ > > discuss mailing list > > disc...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss