Re: [ovs-discuss] poor ovs-dpdk performance vs ovs

[Ian Stokes]

On 11/16/2018 6:52 PM, Robert Brooks wrote:

Hi Tiago,

you are correct, this is a virtualization environment. The tests numbers 
I am quoting are running directly on the hypervisor, rather than guests.


My ultimate aim is to get good performance for guests talking over 
VXLAN. Using OvS I get around 5Gb/s between guests talking over VXLAN 
and near wire speed performance where the guests talk to a bridge 
without VXLAN. I'd like to improve on the VXLAN performance.


I believe VXLAN is offloaded by tx-udp_tnl-segmentation? (...VXLAN is 
being configured at the interface level by OpenNebula, so handled by the 
kernel module).


Regarding the NIC, I see:

[root@v41 ~]# ethtool -k p2p1 | grep tcp-segmentation-offload

*tcp-segmentation-offload*: on


Are the TSO patches for OvS-DPDK suitable to apply and use, if so 
anything I should know? eg version to apply to, source to obtain current 
patches from?




Hi Robert,

You can download Tiagos patch set from below:

https://patchwork.ozlabs.org/project/openvswitch/list/?series=70068=*

Cover note can be found at

https://mail.openvswitch.org/pipermail/ovs-dev/2018-October/352889.html

Note the first 3 patches of the series have been upstreamed to master 
already, the rest may not apply cleanly though at this point.


If you are interested in testing the series you can checkout the 
following commit on master that the series was rebased to.


35fe9ef ("dpif-netdev: Add vlan to mask for flow_put operation.")

Thanks
Ian

Many thanks,

Rob

On Fri, Nov 16, 2018 at 2:06 AM Lam, Tiago > wrote:


Hi,

I have a couple of questions, but at a first glance this seems to be a
result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are
in the works of addressing that.

The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar
to the ones we have been getting with and without our TSO series [1]
applied.

On the description below, is this a virtualized environment by any
change? I.e. are you running the iperf tests on top of VMs running on
those Centos hosts, or directly on them?

Can you also confirm you are using TSO when using OvS with the Kernel by
reporting what you get with $ethtool -k $iface | grep
"tcp-segmentation-offload:". Is it reported as "on"?

Regards,
Tiago.

[1]
https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html

On 15/11/2018 22:17, Robert Brooks wrote:
 > I have a pair of Centos 7 hosts on 10gig switch.
 >
 > Using OVS without dpdk enabled I can get 9.4Gb/s with a simple
iperf test.
 >
 > After switching the receiving host to ovs-dpdk following guidance
here:
 > http://docs.openvswitch.org/en/latest/intro/install/dpdk/
 >
 > I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.
 >
 > Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus,
256GB
 > RAM, Intel 92599ES NIC.
 >
 > I have confirmed the doucmented kernel boot options are set and 1GB
 > hugepages are in use.
 >
 > # cat /proc/cmdline
 > BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64
 > root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto
 > rd.lvm.lv  =vg01/swap
rd.lvm.lv 
 > =vg01/usr console=ttyS0,115200 iommu=pt
intel_iommu=on
 > default_hugepagesz=1G hugepagesz=1G hugepages=8
 >
 > Software is openvswitch 2.10.1 built using the provided rpm spec
file. I
 > have used both the Centos provided dpdk 17.11 and a re-compile using
 > latest LTS of 17.11.4.
 >
 > I have tried various performance tweaks, including cpu pinning and
 > isolated cpus.
 >
 > Switching back to regular ovs returns the performance to close to
wire
 > speed.
 >
 > Under load the following is logged:
 >
 > 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to
[POLLIN] on
 > fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98%
 > CPU usage)
 >
 > Config dump:
 >
 > # ovsdb-client dump Open_vSwitch
 >
 > AutoAttach table
 >
 > _uuid mappings system_description system_name
 >
 > -  -- ---
 >
 >
 > Bridge table
 >
 > _uuid                                auto_attach controller
datapath_id
 >       datapath_type datapath_version external_ids fail_mode
flood_vlans
 > flow_tables ipfix mcast_snooping_enable mirrors name  netflow
 > other_config ports
 >
 >
 >  --- --
 > -- -  
-
 > --- --- - - --- -
 > --- 
 > 

Re: [ovs-discuss] poor ovs-dpdk performance vs ovs

[Robert Brooks]

Hi Tiago,

you are correct, this is a virtualization environment. The tests numbers I
am quoting are running directly on the hypervisor, rather than guests.

My ultimate aim is to get good performance for guests talking over VXLAN.
Using OvS I get around 5Gb/s between guests talking over VXLAN and near
wire speed performance where the guests talk to a bridge without VXLAN. I'd
like to improve on the VXLAN performance.

I believe VXLAN is offloaded by tx-udp_tnl-segmentation? (...VXLAN is being
configured at the interface level by OpenNebula, so handled by the kernel
module).

Regarding the NIC, I see:

[root@v41 ~]# ethtool -k p2p1 | grep tcp-segmentation-offload

*tcp-segmentation-offload*: on


Are the TSO patches for OvS-DPDK suitable to apply and use, if so anything
I should know? eg version to apply to, source to obtain current patches
from?

Many thanks,

Rob

On Fri, Nov 16, 2018 at 2:06 AM Lam, Tiago  wrote:

> Hi,
>
> I have a couple of questions, but at a first glance this seems to be a
> result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are
> in the works of addressing that.
>
> The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar
> to the ones we have been getting with and without our TSO series [1]
> applied.
>
> On the description below, is this a virtualized environment by any
> change? I.e. are you running the iperf tests on top of VMs running on
> those Centos hosts, or directly on them?
>
> Can you also confirm you are using TSO when using OvS with the Kernel by
> reporting what you get with $ethtool -k $iface | grep
> "tcp-segmentation-offload:". Is it reported as "on"?
>
> Regards,
> Tiago.
>
> [1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html
>
> On 15/11/2018 22:17, Robert Brooks wrote:
> > I have a pair of Centos 7 hosts on 10gig switch.
> >
> > Using OVS without dpdk enabled I can get 9.4Gb/s with a simple iperf
> test.
> >
> > After switching the receiving host to ovs-dpdk following guidance here:
> > http://docs.openvswitch.org/en/latest/intro/install/dpdk/
> >
> > I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.
> >
> > Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus, 256GB
> > RAM, Intel 92599ES NIC.
> >
> > I have confirmed the doucmented kernel boot options are set and 1GB
> > hugepages are in use.
> >
> > # cat /proc/cmdline
> > BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64
> > root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto
> > rd.lvm.lv =vg01/swap rd.lvm.lv
> > =vg01/usr console=ttyS0,115200 iommu=pt intel_iommu=on
> > default_hugepagesz=1G hugepagesz=1G hugepages=8
> >
> > Software is openvswitch 2.10.1 built using the provided rpm spec file. I
> > have used both the Centos provided dpdk 17.11 and a re-compile using
> > latest LTS of 17.11.4.
> >
> > I have tried various performance tweaks, including cpu pinning and
> > isolated cpus.
> >
> > Switching back to regular ovs returns the performance to close to wire
> > speed.
> >
> > Under load the following is logged:
> >
> > 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to [POLLIN] on
> > fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98%
> > CPU usage)
> >
> > Config dump:
> >
> > # ovsdb-client dump Open_vSwitch
> >
> > AutoAttach table
> >
> > _uuid mappings system_description system_name
> >
> > -  -- ---
> >
> >
> > Bridge table
> >
> > _uuidauto_attach controller datapath_id
> >   datapath_type datapath_version external_ids fail_mode flood_vlans
> > flow_tables ipfix mcast_snooping_enable mirrors name  netflow
> > other_config ports
> >
> >
> >  --- --
> > -- -   -
> > --- --- - - --- -
> > --- 
> > -
> >
> > 006f021a-b14d-49fb-a11d-48df2fa2bca1 []  []
> > "001b21a6ddc4" netdev"" {}   []
> > []  {}  []false []  "br0" []
> >   {}   [3bd79dba-777c-40d0-b573-bf9e027326f4,
> > 60b0661f-2177-4283-a6cb-a80336
> >
> >
> > Controller table
> >
> > _uuid connection_mode controller_burst_limit controller_rate_limit
> > enable_async_messages external_ids inactivity_probe is_connected
> > local_gateway local_ip local_netmask max_backoff other_config role
> > status target
> >
> > - --- -- -
> > -   
> > -  - ---  
> > -- --
> >
> >
> > Flow_Sample_Collector_Set table
> >
> > _uuid bridge external_ids id ipfix
> >
> > - --  -- -
> >
> >
> > Flow_Table table
> >
> > 

Re: [ovs-discuss] poor ovs-dpdk performance vs ovs

[Lam, Tiago]

Hi,

I have a couple of questions, but at a first glance this seems to be a
result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are
in the works of addressing that.

The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar
to the ones we have been getting with and without our TSO series [1]
applied.

On the description below, is this a virtualized environment by any
change? I.e. are you running the iperf tests on top of VMs running on
those Centos hosts, or directly on them?

Can you also confirm you are using TSO when using OvS with the Kernel by
reporting what you get with $ethtool -k $iface | grep
"tcp-segmentation-offload:". Is it reported as "on"?

Regards,
Tiago.

[1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html

On 15/11/2018 22:17, Robert Brooks wrote:
> I have a pair of Centos 7 hosts on 10gig switch.
> 
> Using OVS without dpdk enabled I can get 9.4Gb/s with a simple iperf test.
> 
> After switching the receiving host to ovs-dpdk following guidance here:
> http://docs.openvswitch.org/en/latest/intro/install/dpdk/
> 
> I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.
> 
> Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus, 256GB
> RAM, Intel 92599ES NIC.
> 
> I have confirmed the doucmented kernel boot options are set and 1GB
> hugepages are in use.
> 
> # cat /proc/cmdline 
> BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64
> root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto
> rd.lvm.lv =vg01/swap rd.lvm.lv
> =vg01/usr console=ttyS0,115200 iommu=pt intel_iommu=on
> default_hugepagesz=1G hugepagesz=1G hugepages=8
> 
> Software is openvswitch 2.10.1 built using the provided rpm spec file. I
> have used both the Centos provided dpdk 17.11 and a re-compile using
> latest LTS of 17.11.4.
> 
> I have tried various performance tweaks, including cpu pinning and
> isolated cpus.
> 
> Switching back to regular ovs returns the performance to close to wire
> speed.
> 
> Under load the following is logged:
> 
> 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to [POLLIN] on
> fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98%
> CPU usage)
> 
> Config dump:
> 
> # ovsdb-client dump Open_vSwitch
> 
> AutoAttach table
> 
> _uuid mappings system_description system_name
> 
> -  -- ---
> 
> 
> Bridge table
> 
> _uuid                                auto_attach controller datapath_id 
>       datapath_type datapath_version external_ids fail_mode flood_vlans
> flow_tables ipfix mcast_snooping_enable mirrors name  netflow
> other_config ports                                                      
>          
> 
>  --- --
> -- -   -
> --- --- - - --- -
> --- 
> -
> 
> 006f021a-b14d-49fb-a11d-48df2fa2bca1 []          []        
> "001b21a6ddc4" netdev        ""     {}           []       
> []          {}          []    false                 []      "br0" []   
>   {}           [3bd79dba-777c-40d0-b573-bf9e027326f4,
> 60b0661f-2177-4283-a6cb-a80336
> 
> 
> Controller table
> 
> _uuid connection_mode controller_burst_limit controller_rate_limit
> enable_async_messages external_ids inactivity_probe is_connected
> local_gateway local_ip local_netmask max_backoff other_config role
> status target
> 
> - --- -- -
> -   
> -  - ---  
> -- --
> 
> 
> Flow_Sample_Collector_Set table
> 
> _uuid bridge external_ids id ipfix
> 
> - --  -- -
> 
> 
> Flow_Table table
> 
> _uuid external_ids flow_limit groups name overflow_policy prefixes
> 
> -  -- --  --- 
> 
> 
> IPFIX table
> 
> _uuid cache_active_timeout cache_max_flows external_ids obs_domain_id
> obs_point_id other_config sampling targets
> 
> -  ---  -
>    ---
> 
> 
> Interface table
> 
> _uuid                                admin_state bfd bfd_status
> cfm_fault cfm_fault_status cfm_flap_count cfm_health cfm_mpid
> cfm_remote_mpids cfm_remote_opstate duplex error external_ids ifindex 
> ingress_policing_burst ingress_policing_rate lacp_current link_resets
> link_speed  link_stat 
> 
>  --- --- --
> -  -- -- 
>  -- -- -  
> -- -  ---
> --- --
> 
>