Hi Tiago, you are correct, this is a virtualization environment. The tests numbers I am quoting are running directly on the hypervisor, rather than guests.
My ultimate aim is to get good performance for guests talking over VXLAN. Using OvS I get around 5Gb/s between guests talking over VXLAN and near wire speed performance where the guests talk to a bridge without VXLAN. I'd like to improve on the VXLAN performance. I believe VXLAN is offloaded by tx-udp_tnl-segmentation? (...VXLAN is being configured at the interface level by OpenNebula, so handled by the kernel module). Regarding the NIC, I see: [root@v41 ~]# ethtool -k p2p1 | grep tcp-segmentation-offload *tcp-segmentation-offload*: on Are the TSO patches for OvS-DPDK suitable to apply and use, if so anything I should know? eg version to apply to, source to obtain current patches from? Many thanks, Rob On Fri, Nov 16, 2018 at 2:06 AM Lam, Tiago <tiago....@intel.com> wrote: > Hi, > > I have a couple of questions, but at a first glance this seems to be a > result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are > in the works of addressing that. > > The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar > to the ones we have been getting with and without our TSO series [1] > applied. > > On the description below, is this a virtualized environment by any > change? I.e. are you running the iperf tests on top of VMs running on > those Centos hosts, or directly on them? > > Can you also confirm you are using TSO when using OvS with the Kernel by > reporting what you get with $ethtool -k $iface | grep > "tcp-segmentation-offload:". Is it reported as "on"? > > Regards, > Tiago. > > [1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html > > On 15/11/2018 22:17, Robert Brooks wrote: > > I have a pair of Centos 7 hosts on 10gig switch. > > > > Using OVS without dpdk enabled I can get 9.4Gb/s with a simple iperf > test. > > > > After switching the receiving host to ovs-dpdk following guidance here: > > http://docs.openvswitch.org/en/latest/intro/install/dpdk/ > > > > I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU. > > > > Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus, 256GB > > RAM, Intel 92599ES NIC. > > > > I have confirmed the doucmented kernel boot options are set and 1GB > > hugepages are in use. > > > > # cat /proc/cmdline > > BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64 > > root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto > > rd.lvm.lv <http://rd.lvm.lv>=vg01/swap rd.lvm.lv > > <http://rd.lvm.lv>=vg01/usr console=ttyS0,115200 iommu=pt intel_iommu=on > > default_hugepagesz=1G hugepagesz=1G hugepages=8 > > > > Software is openvswitch 2.10.1 built using the provided rpm spec file. I > > have used both the Centos provided dpdk 17.11 and a re-compile using > > latest LTS of 17.11.4. > > > > I have tried various performance tweaks, including cpu pinning and > > isolated cpus. > > > > Switching back to regular ovs returns the performance to close to wire > > speed. > > > > Under load the following is logged: > > > > 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to [POLLIN] on > > fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98% > > CPU usage) > > > > Config dump: > > > > # ovsdb-client dump Open_vSwitch > > > > AutoAttach table > > > > _uuid mappings system_description system_name > > > > ----- -------- ------------------ ----------- > > > > > > Bridge table > > > > _uuid auto_attach controller datapath_id > > datapath_type datapath_version external_ids fail_mode flood_vlans > > flow_tables ipfix mcast_snooping_enable mirrors name netflow > > other_config ports > > > > > > ------------------------------------ ----------- ---------- > > ------------------ ------------- ---------------- ------------ --------- > > ----------- ----------- ----- --------------------- ------- ----- > > ------- ------------ > > --------------------------------------------------------------------- > > > > 006f021a-b14d-49fb-a11d-48df2fa2bca1 [] [] > > "0000001b21a6ddc4" netdev "<built-in>" {} [] > > [] {} [] false [] "br0" [] > > {} [3bd79dba-777c-40d0-b573-bf9e027326f4, > > 60b0661f-2177-4283-a6cb-a80336 > > > > > > Controller table > > > > _uuid connection_mode controller_burst_limit controller_rate_limit > > enable_async_messages external_ids inactivity_probe is_connected > > local_gateway local_ip local_netmask max_backoff other_config role > > status target > > > > ----- --------------- ---------------------- --------------------- > > --------------------- ------------ ---------------- ------------ > > ------------- -------- ------------- ----------- ------------ ---- > > ------ ------ > > > > > > Flow_Sample_Collector_Set table > > > > _uuid bridge external_ids id ipfix > > > > ----- ------ ------------ -- ----- > > > > > > Flow_Table table > > > > _uuid external_ids flow_limit groups name overflow_policy prefixes > > > > ----- ------------ ---------- ------ ---- --------------- -------- > > > > > > IPFIX table > > > > _uuid cache_active_timeout cache_max_flows external_ids obs_domain_id > > obs_point_id other_config sampling targets > > > > ----- -------------------- --------------- ------------ ------------- > > ------------ ------------ -------- ------- > > > > > > Interface table > > > > _uuid admin_state bfd bfd_status > > cfm_fault cfm_fault_status cfm_flap_count cfm_health cfm_mpid > > cfm_remote_mpids cfm_remote_opstate duplex error external_ids ifindex > > ingress_policing_burst ingress_policing_rate lacp_current link_resets > > link_speed link_stat > > > > ------------------------------------ ----------- --- ---------- > > --------- ---------------- -------------- ---------- -------- > > ---------------- ------------------ ------ ----- ------------ -------- > > ---------------------- --------------------- ------------ ----------- > > ----------- ---------- > > > > 271077eb-97af-4c2d-a5ee-2c63c9367312 up {} {} [] > > [] [] [] [] [] > > [] full [] {} 13 0 > > 0 [] 11 10000000 up > > > > 4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136 up {} {} [] > > [] [] [] [] [] > > [] full [] {} 13032858 0 > > 0 [] 0 10000000000 up s > > > > > > Manager table > > > > _uuid connection_mode external_ids inactivity_probe is_connected > > max_backoff other_config status target > > > > ----- --------------- ------------ ---------------- ------------ > > ----------- ------------ ------ ------ > > > > > > Mirror table > > > > _uuid external_ids name output_port output_vlan select_all > > select_dst_port select_src_port select_vlan snaplen statistics > > > > ----- ------------ ---- ----------- ----------- ---------- > > --------------- --------------- ----------- ------- ---------- > > > > > > NetFlow table > > > > _uuid active_timeout add_id_to_interface engine_id engine_type > > external_ids targets > > > > ----- -------------- ------------------- --------- ----------- > > ------------ ------- > > > > > > Open_vSwitch table > > > > _uuid bridges > > cur_cfg datapath_types db_version dpdk_initialized dpdk_version > > external_ids > > iface_types > > > > ------------------------------------ > > -------------------------------------- ------- ---------------- > > ---------- ---------------- -------------- > > > -------------------------------------------------------------------------------------------------------- > > ---------------------------------------- > > > > ea8e7bcc-df11-42bd-96c9-b7eaafdfa863 > > [006f021a-b14d-49fb-a11d-48df2fa2bca1] 45 [netdev, system] "7.16.1" > > true "DPDK 17.11.4" {hostname="v41", > > rundir="/var/run/openvswitch", > > system-id="7d5d1db0-87db-41bc-bb94-661dde831e9f"} [dpdk, dpdkr, > > dpdkvhostuser, dpdkvhostum > > > > > > Port table > > > > _uuid bond_active_slave bond_downdelay > > bond_fake_iface bond_mode bond_updelay cvlans external_ids fake_bridge > > interfaces lacp mac name other_config > > protected qos rstp_statistics rstp_status statistics status tag trunks > > vlan_mode > > > > ------------------------------------ ----------------- -------------- > > --------------- --------- ------------ ------ ------------ ----------- > > -------------------------------------- ---- --- ------ ------------ > > --------- --- --------------- ----------- ---------- ------ --- ------ > > --------- > > > > 60b0661f-2177-4283-a6cb-a80336a8c478 [] 0 > > false [] 0 [] {} false > > [271077eb-97af-4c2d-a5ee-2c63c9367312] [] [] "br0" {} > > false [] {} {} {} {} [] [] [] > > > > 3bd79dba-777c-40d0-b573-bf9e027326f4 [] 0 > > false [] 0 [] {} false > > [4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136] [] [] "p2p1" {} > > false [] {} {} {} {} [] [] [] > > > > > > QoS table > > > > _uuid external_ids other_config queues type > > > > ----- ------------ ------------ ------ ---- > > > > > > Queue table > > > > _uuid dscp external_ids other_config > > > > ----- ---- ------------ ------------ > > > > > > SSL table > > > > _uuid bootstrap_ca_cert ca_cert certificate external_ids private_key > > > > ----- ----------------- ------- ----------- ------------ ----------- > > > > > > sFlow table > > > > _uuid agent external_ids header polling sampling targets > > > > ----- ----- ------------ ------ ------- -------- ------- > > > > > > I see rx_dropped counters. Other than that I'm not sure what's going on, > > as I've tried various tuning suggested by dpdk, Intel and Red Hat, but > > performance remains unimproved. > > > > > > Regards, > > > > > > Rob > > > > > > _______________________________________________ > > discuss mailing list > > disc...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss