> > >Yes that test gives much better results without DPDK – I saw 18Gbps. The >reason is that OVS- >DPDK does not support TSO. But DPDK does better with UDP and packets/second >tests. Some >patches were proposed - google ovs dpdk tso. >
Hi, As Darragh mentioned, DPDK does not currently support TSO or LRO (both of which are typically available for kernel-based interfaces), which is why you observe a performance deficit. I recently pushed an RFC patch for TSO support for OvS v2.6.1 - this may help boost your numbers: https://patchwork.ozlabs.org/patch/749564/ Thanks, Mark >You could try using more queues - >http://docs.openvswitch.org/en/latest/topics/dpdk/vhost- >user/#adding-vhost-user-ports-to-the-guest-qemu > >Darragh. > >From: [email protected] >[mailto:[email protected]] On >Behalf Of [email protected] >Sent: 20 April 2017 08:35 >To: ovs-discuss <[email protected]> >Subject: [ovs-discuss] OVS-DPDK performance problem in Openstack Ocata > >Hi, > >I tested ovs- >dpdk(compiled using ovs 2.6.1) under openstack ocata, and I found the >performance is low. >Can someone give me some suggestions？ > >iperf3 througput test between 2 VMs on the same compute node gives bandwidth: >6.82 Gbps. Inte >l 82599ES 10-Gigabit SFI/SFP+ NICs with igb_uio are used. >==================================== >Transfer Bandwith Retr >7.94 GBytes 6.82 Gbits/sec 17119 sender >7.94 GBytes 6.82 Gbits/sec receiver > >Envionment: >1 compute node with 2 VM(dpdk), iperf3 are running under the 2 VMs. >compute node hardware: Huawei RH2288H V3, E5-2640 [email protected] GHZ*2, DDR3*16G*8 > > >[root@EXTENV-10-254-9-7 ~]# top >top - 16:17:10 up 30 min, 5 users, load average: 5.36, 5.53, 4.26 >Tasks: 387 total, 1 running, 386 sleeping, 0 stopped, 0 zombie >%Cpu(s): 18.4 us, 0.3 sy, 0.0 ni, 81.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st >KiB Mem : 26357619+total, 346508 free, 26285744+used, 372236 buff/cache >KiB Swap: 13421772+total, 13421772+free, 0 used. 372340 avail Mem > > 3841 root 10 -10 8081120 366680 12928 S 402.0 0.1 76:02.16 ovs- >vswitchd > > 4080 root 20 0 1984008 233264 11880 S 100.0 0.1 10:50.82 qemu- >kvm > > 4068 root 20 0 1981960 266368 11880 S 94.7 0.1 10:23.92 qemu- >kvm > > 249 root 25 5 0 0 0 S 3.7 0.0 3:10.67 ksmd > > > 1094 root 20 0 4368 676 520 S 0.7 0.0 0:02.61 rngd > > > 3986 neutron 20 0 350596 85664 5180 S 0.3 0.0 0:05.27 neutron-op > > >[root@EXTENV-10-254-9-7 ~]# sudo top -p `pidof ovs-vswitchd` -H -d1 >top - 16:31:18 up 44 min, 5 users, load average: 4.12, 4.34, 4.39 >Threads: 41 total, 4 running, 37 sleeping, 0 stopped, 0 zombie >%Cpu(s): 12.5 us, 0.3 sy, 0.0 ni, 87.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st >KiB Mem : 26357619+total, 347080 free, 26285494+used, 374160 buff/cache >KiB Swap: 13421772+total, 13421772+free, 0 used. 373876 avail Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 3953 root 10 - >10 8081120 366680 12928 R 99.9 0.1 32:54.06 pmd93 > > 3950 root 10 - >10 8081120 366680 12928 R 99.9 0.1 32:54.07 pmd90 > > 3951 root 10 - >10 8081120 366680 12928 R 99.9 0.1 32:54.07 pmd91 > > 3952 root 10 - >10 8081120 366680 12928 R 99.9 0.1 32:54.07 pmd92 > > 3841 root 10 -10 8081120 366680 12928 S 0.0 0.1 1:03.48 ovs- >vswitchd > > 3842 root 10 -10 8081120 366680 12928 S 0.0 0.1 0:00.00 vfio-sync > > >[root@EXTENV-10-254-9-7 ~]# ovs-appctl dpif-netdev/pmd-stats-show >pmd thread numa_id 0 core_id 6: > emc hits:555098657 > megaflow hits:27 > avg. subtable lookups per hit:1.19 > miss:43 > lost:0 > polling cycles:2617584158048 (74.68%) > processing cycles:887414434068 (25.32%) > avg cycles per packet: 6314.19 (3504998592116/555098759) > avg processing cycles per packet: 1598.66 (887414434068/555098759) >pmd thread numa_id 0 core_id 7: > emc hits:0 > megaflow hits:0 > avg. subtable lookups per hit:0.00 > miss:0 > lost:0 > polling cycles:2024072202084 (100.00%) > processing cycles:0 (0.00%) >pmd thread numa_id 0 core_id 22: > emc hits:0 > megaflow hits:0 > avg. subtable lookups per hit:0.00 > miss:0 > lost:0 > polling cycles:2695473929852 (100.00%) > processing cycles:0 (0.00%) >pmd thread numa_id 0 core_id 23: > emc hits:68276661 > megaflow hits:27 > avg. subtable lookups per hit:1.15 > miss:42 > lost:0 > polling cycles:3094829433060 (94.27%) > processing cycles:188263651948 (5.73%) > avg cycles per packet: 48085.07 (3283093085008/68276761) > avg processing cycles per packet: 2757.36 (188263651948/68276761) >main thread: > emc hits:0 > megaflow hits:0 > avg. subtable lookups per hit:0.00 > miss:0 > lost:0 > polling cycles:65844260 (100.00%) > processing cycles:0 (0.00%) >[root@EXTENV-10-254-9-7 ~]# > > ># ovs-appctl dpif-netdev/pmd-rxq-show >pmd thread numa_id 0 core_id 6: > isolated : false > port: vhu6e13f9fe-a6 queue-id: 0 >pmd thread numa_id 0 core_id 7: > isolated : false > port: vhu69e40c2f-d0 queue-id: 0 >pmd thread numa_id 0 core_id 22: > isolated : false > port: dpdk0 queue-id: 0 >pmd thread numa_id 0 core_id 23: > isolated : false > port: vhu0161a0d7-ae queue-id: 0 > > ># dpdk-devbind -s >Network devices using DPDK-compatible driver >============================================ >0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio >unused=ixgbe,vfio- >pci >0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio >unused=ixgbe,vfio- >pci > > >[root@EXTENV-10-254-9-7 ~]# uname -a >Linux EXTENV-10-254-9-7 3.10.0- >327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 >GNU/Linux > >cat /proc/cmdline >BOOT_IMAGE=/vmlinuz-3.10.0-327.el7.x86_64 root=/dev/mapper/bclinux- >root ro crashkernel=auto rd.lvm.lv=bclinux/root rd.lvm.lv=bclinux/swap rhgb >quiet default_hug >epagesz=1GB hugepagesz=1G hugepages=200 isolcpus=2,3,26,27 > ># cat /proc/meminfo | grep Huge >AnonHugePages: 710656 kB >HugePages_Total: 248 >HugePages_Free: 245 >HugePages_Rsvd: 0 >HugePages_Surp: 0 > > ># ovs-vsctl list Open_vSwitch >_uuid : b000a321-12ea-47c6-9e96-3b76368b0336 >bridges : [5b784499-4af6-4354-92e1-e20712339bd3, >a353f240-92f7-4f2d-b5e8- >4aa32c6a5a24] >cur_cfg : 894 >datapath_types : [netdev, system] >db_version : "7.14.0" >external_ids : {hostname="EXTENV-10-254-9-7", >system-id="c4e79302-273a-4afe-b77c- >397e383a3fa5"} >iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, >geneve, gre, internal >, ipsec_gre, lisp, patch, stt, system, tap, vxlan] >manager_options : [1d16f251-b844-44cd-a044-8cffbfd7ece2] >next_cfg : 894 >other_config : {dpdk-alloc-mem="2048", dpdk-init="true", >dpdk-socket-mem="1024", pmd- >cpu-mask="c000c0"} >ovs_version : "2.6.1-dpdk1" >ssl : [] >statistics : {} >system_type : centos >system_version : "7" > > ># cpu_layout.py >============================================================ >Core and Socket Information (as reported by '/proc/cpuinfo') >============================================================ > >cores = [0, 1, 2, 3, 4, 5, 6, 7] >sockets = [0, 1] > > Socket 0 Socket 1 > -------- -------- >Core 0 [0, 16] [8, 24] >Core 1 [1, 17] [9, 25] >Core 2 [2, 18] [10, 26] >Core 3 [3, 19] [11, 27] >Core 4 [4, 20] [12, 28] >Core 5 [5, 21] [13, 29] >Core 6 [6, 22] [14, 30] >Core 7 [7, 23] [15, 31] > > >[root@EXTENV-10-254-9-7 ~]# ovs-vsctl show >b000a321-12ea-47c6-9e96-3b76368b0336 > Manager "ptcp:6640:127.0.0.1" > is_connected: true > Bridge br-dpdk > Controller "tcp:127.0.0.1:6633" > is_connected: true > fail_mode: secure > Port br-dpdk > tag: 787 > Interface br-dpdk > type: internal > Port "dpdk0" > Interface "dpdk0" > type: dpdk > options: {dpdk-devargs="0000:02:00.0"} > Port phy-br-dpdk > Interface phy-br-dpdk > type: patch > options: {peer=int-br-dpdk} > Bridge br-int > Controller "tcp:127.0.0.1:6633" > is_connected: true > fail_mode: secure > Port patch-tun > Interface patch-tun > type: patch > options: {peer=patch-int} > Port "vhu69e40c2f-d0" > tag: 1 > Interface "vhu69e40c2f-d0" > type: dpdkvhostuserclient > options: >{vhost-server-path="/var/run/openvswitch/vhu69e40c2f-d0"} > Port "vhu0161a0d7-ae" > tag: 1 > Interface "vhu0161a0d7-ae" > type: dpdkvhostuserclient > options: >{vhost-server-path="/var/run/openvswitch/vhu0161a0d7-ae"} > Port int-br-dpdk > Interface int-br-dpdk > type: patch > options: {peer=phy-br-dpdk} > Port br-int > Interface br-int > type: internal > Port "vhu6e13f9fe-a6" > tag: 1 > Interface "vhu6e13f9fe-a6" > type: dpdkvhostuserclient > options: >{vhost-server-path="/var/run/openvswitch/vhu6e13f9fe-a6"} > ovs_version: "2.6.1-dpdk1" >[root@EXTENV-10-254-9-7 ~]# > >________________________________________ >[email protected] _______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
