Re: [ovs-discuss] [ovs-dev] Packet Drop Issue in OVS-DPDK L2FWD Application
On Mon, Nov 19, 2018 at 5:36 AM Ian Stokes wrote:
> On 11/18/2018 8:16 PM, vkrishnabhat k wrote:
> > Hi Team,
> >
> > I am new to OVS and DPDK. While I am using l2fwd application with OVS and
> > DPDK I am seeing packet drop issue in OVS bridge.
> >
> > Topology : My topology has Ubuntu machine (Ubuntu 18.04 LTS). I have
> > installed Qemu-KVM 2.11.1 version. Also I am using OVS-DPDK. Please find
> > the detailed topology attached with this mail. I have bound two NICs
> (Intel
> > 82599ES 10-gigabit ) to dpdk IGB_UIO driver and also have added same
> ports
> > in to OVS bridge "br0". I am trying to send the bidirectional traffic
> from
> > both the port and measure the throughput value for the l2fwd application
>
I also saw drops in the br0 using ovs-dpdk with 82599ES cards, see "poor
ovs-dpdk performance vs ovs", unfortunately the stats got truncated, so I
re-ran the testing and I see them here:
AutoAttach table
_uuid mappings system_description system_name
- -- ---
Bridge table
_uuidauto_attach controller datapath_id
datapath_type datapath_version external_ids fail_mode flood_vlans
flow_tables ipfix mcast_snooping_enable mirrors name netflow other_config
ports
protocols rstp_enable rstp_status sflow status stp_enable
--- --
-- - -
--- --- - - --- - ---
- --- --- - -- --
28911b6f-4f85-4a77-982c-d16b0e284e1a [] [] "001b21a6ddc4"
netdev"" {} [][] {}
[]false [] "br0" [] {}
[257ad852-9078-4378-a996-3cbb7772457e,
2cff7d6e-2f3a-4aec-8f1c-f29125760771] []false {} []
{} false
Controller table
_uuid connection_mode controller_burst_limit controller_rate_limit
enable_async_messages external_ids inactivity_probe is_connected
local_gateway local_ip local_netmask max_backoff other_config role status
target
- --- -- -
-
- - --- --
--
Flow_Sample_Collector_Set table
_uuid bridge external_ids id ipfix
- -- -- -
Flow_Table table
_uuid external_ids flow_limit groups name overflow_policy prefixes
- -- -- ---
IPFIX table
_uuid cache_active_timeout cache_max_flows external_ids obs_domain_id
obs_point_id other_config sampling targets
- --- -
---
Interface table
_uuidadmin_state bfd bfd_status cfm_fault
cfm_fault_status cfm_flap_count cfm_health cfm_mpid cfm_remote_mpids
cfm_remote_opstate duplex error external_ids ifindex ingress_policing_burst
ingress_policing_rate lacp_current link_resets link_speed link_state lldp
mac mac_in_use mtu mtu_request name ofport ofport_request
options other_config statistics
status
type
--- --- -- -
-- --
-- -- -
-- - ---
--- -- --- --- --- --
-- -- -
Re: [ovs-discuss] poor ovs-dpdk performance vs ovs
Hi Tiago,
you are correct, this is a virtualization environment. The tests numbers I
am quoting are running directly on the hypervisor, rather than guests.
My ultimate aim is to get good performance for guests talking over VXLAN.
Using OvS I get around 5Gb/s between guests talking over VXLAN and near
wire speed performance where the guests talk to a bridge without VXLAN. I'd
like to improve on the VXLAN performance.
I believe VXLAN is offloaded by tx-udp_tnl-segmentation? (...VXLAN is being
configured at the interface level by OpenNebula, so handled by the kernel
module).
Regarding the NIC, I see:
[root@v41 ~]# ethtool -k p2p1 | grep tcp-segmentation-offload
*tcp-segmentation-offload*: on
Are the TSO patches for OvS-DPDK suitable to apply and use, if so anything
I should know? eg version to apply to, source to obtain current patches
from?
Many thanks,
Rob
On Fri, Nov 16, 2018 at 2:06 AM Lam, Tiago wrote:
> Hi,
>
> I have a couple of questions, but at a first glance this seems to be a
> result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are
> in the works of addressing that.
>
> The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar
> to the ones we have been getting with and without our TSO series [1]
> applied.
>
> On the description below, is this a virtualized environment by any
> change? I.e. are you running the iperf tests on top of VMs running on
> those Centos hosts, or directly on them?
>
> Can you also confirm you are using TSO when using OvS with the Kernel by
> reporting what you get with $ethtool -k $iface | grep
> "tcp-segmentation-offload:". Is it reported as "on"?
>
> Regards,
> Tiago.
>
> [1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html
>
> On 15/11/2018 22:17, Robert Brooks wrote:
> > I have a pair of Centos 7 hosts on 10gig switch.
> >
> > Using OVS without dpdk enabled I can get 9.4Gb/s with a simple iperf
> test.
> >
> > After switching the receiving host to ovs-dpdk following guidance here:
> > http://docs.openvswitch.org/en/latest/intro/install/dpdk/
> >
> > I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.
> >
> > Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus, 256GB
> > RAM, Intel 92599ES NIC.
> >
> > I have confirmed the doucmented kernel boot options are set and 1GB
> > hugepages are in use.
> >
> > # cat /proc/cmdline
> > BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64
> > root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto
> > rd.lvm.lv <http://rd.lvm.lv>=vg01/swap rd.lvm.lv
> > <http://rd.lvm.lv>=vg01/usr console=ttyS0,115200 iommu=pt intel_iommu=on
> > default_hugepagesz=1G hugepagesz=1G hugepages=8
> >
> > Software is openvswitch 2.10.1 built using the provided rpm spec file. I
> > have used both the Centos provided dpdk 17.11 and a re-compile using
> > latest LTS of 17.11.4.
> >
> > I have tried various performance tweaks, including cpu pinning and
> > isolated cpus.
> >
> > Switching back to regular ovs returns the performance to close to wire
> > speed.
> >
> > Under load the following is logged:
> >
> > 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to [POLLIN] on
> > fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98%
> > CPU usage)
> >
> > Config dump:
> >
> > # ovsdb-client dump Open_vSwitch
> >
> > AutoAttach table
> >
> > _uuid mappings system_description system_name
> >
> > - -- ---
> >
> >
> > Bridge table
> >
> > _uuidauto_attach controller datapath_id
> > datapath_type datapath_version external_ids fail_mode flood_vlans
> > flow_tables ipfix mcast_snooping_enable mirrors name netflow
> > other_config ports
> >
> >
> > --- --
> > -- - -
> > --- --- - - --- -
> > ---
> > -
> >
> > 006f021a-b14d-49fb-a11d-48df2fa2bca1 [] []
> > "001b21a6ddc4" netdev"" {} []
> > [] {} []false [] "br0" []
> > {} [3bd79dba-777c-40d0-b573-bf9e027326f4,
> > 60b0661f-2177-4283-a6cb-a80336
> >
> >
> > Controller table
> >
> > _uuid connection_mode controller_burst_li
[ovs-discuss] poor ovs-dpdk performance vs ovs
I have a pair of Centos 7 hosts on 10gig switch.
Using OVS without dpdk enabled I can get 9.4Gb/s with a simple iperf test.
After switching the receiving host to ovs-dpdk following guidance here:
http://docs.openvswitch.org/en/latest/intro/install/dpdk/
I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.
Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus, 256GB
RAM, Intel 92599ES NIC.
I have confirmed the doucmented kernel boot options are set and 1GB
hugepages are in use.
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64
root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto
rd.lvm.lv=vg01/swap
rd.lvm.lv=vg01/usr console=ttyS0,115200 iommu=pt intel_iommu=on
default_hugepagesz=1G hugepagesz=1G hugepages=8
Software is openvswitch 2.10.1 built using the provided rpm spec file. I
have used both the Centos provided dpdk 17.11 and a re-compile using latest
LTS of 17.11.4.
I have tried various performance tweaks, including cpu pinning and isolated
cpus.
Switching back to regular ovs returns the performance to close to wire
speed.
Under load the following is logged:
2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to [POLLIN] on fd
74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98% CPU
usage)
Config dump:
# ovsdb-client dump Open_vSwitch
AutoAttach table
_uuid mappings system_description system_name
- -- ---
Bridge table
_uuidauto_attach controller datapath_id
datapath_type datapath_version external_ids fail_mode flood_vlans
flow_tables ipfix mcast_snooping_enable mirrors name netflow other_config
ports
--- --
-- - -
--- --- - - --- - ---
-
006f021a-b14d-49fb-a11d-48df2fa2bca1 [] [] "001b21a6ddc4"
netdev"" {} [][] {}
[]false [] "br0" [] {}
[3bd79dba-777c-40d0-b573-bf9e027326f4,
60b0661f-2177-4283-a6cb-a80336
Controller table
_uuid connection_mode controller_burst_limit controller_rate_limit
enable_async_messages external_ids inactivity_probe is_connected
local_gateway local_ip local_netmask max_backoff other_config role status
target
- --- -- -
-
- - --- --
--
Flow_Sample_Collector_Set table
_uuid bridge external_ids id ipfix
- -- -- -
Flow_Table table
_uuid external_ids flow_limit groups name overflow_policy prefixes
- -- -- ---
IPFIX table
_uuid cache_active_timeout cache_max_flows external_ids obs_domain_id
obs_point_id other_config sampling targets
- --- -
---
Interface table
_uuidadmin_state bfd bfd_status cfm_fault
cfm_fault_status cfm_flap_count cfm_health cfm_mpid cfm_remote_mpids
cfm_remote_opstate duplex error external_ids ifindex ingress_policing_burst
ingress_policing_rate lacp_current link_resets link_speed link_stat
--- --- -- -
-- --
-- -- -
-- - ---
--- --
271077eb-97af-4c2d-a5ee-2c63c9367312 up {} {} [][]
[] [] [] [] []
full []{} 13 0 0
[] 11 1000up
4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136 up {} {} [][]
[] [] [] [] []
full []{} 13032858 0 0
[] 0 100 up s
Manager table
_uuid connection_mode external_ids inactivity_probe is_connected
max_backoff other_config status target
- ---
--- -- --
Mirror table
_uuid external_ids name output_port output_vlan select_all select_dst_port
select_src_port select_vlan snaplen statistics
- --- --- -- ---
--- --- --- --
NetFlow table
_uuid active_timeout add_id_to_interface engine_id engine_type external_ids
targets
- -- --- - ---
