On Thu, Jun 28, 2018 at 11:22:49AM +0000, Shahaf Shuler wrote: > Hi Ian, Flavio,
Hi! > Thursday, June 28, 2018 12:14 PM, Ian Stokes: > > Subject: Re: [PATCH v11 0/7] OVS-DPDK flow offload with rte_flow > > > > On 6/25/2018 2:21 PM, Shahaf Shuler wrote: > > > Hi, > > > > > > Here is a joint work from Mellanox and Napatech, to enable the flow hw > > > offload with the DPDK generic flow interface (rte_flow). > > > > > > The basic idea is to associate the flow with a mark id (a unit32_t > > > number). > > > Later, we then get the flow directly from the mark id, which could > > > bypass some heavy CPU operations, including but not limiting to mini > > > flow extract, emc lookup, dpcls lookup, etc. > > > > > > The association is done with CMAP in patch 1. The CPU workload > > > bypassing is done in patch 2. The flow offload is done in patch 3, > > > which mainly does two things: > > > > > > - translate the ovs match to DPDK rte flow patterns > > > - bind those patterns with a RSS + MARK action. > > > > > > Patch 5 makes the offload work happen in another thread, for leaving > > > the datapath as light as possible. > > > > > > A PHY-PHY forwarding with 1000 mega flows (udp,tp_src=1000-1999) and 1 > > > million streams (tp_src=1000-1999, tp_dst=2000-2999) show more than > > > 260% performance boost. > > > > > > Note that it's disabled by default, which can be enabled by: > > > > > > $ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true > > > > > > For Mellanox NICs to be tested with the latest stable 17.11.3. > > > > > > > Thanks for the latest revision Shahaf, Just a quick query, I've proposed > > moving from DPDK 17.11.2 to 17.11.3 in OVS. There are a number of patches > > related to Mellanox cards in that release from what I can see, have you > > validated this series with 17.11.3, I guess I'm just wondering are there any > > patches in 17.11.3 that are required? > > Yes the series was validated and tested against this version. > There are some important fixes for mlx5 PMD to make the OVS-DPDK HWOL work > like: > f75e68a net/mlx5: fix flow validation > > (this one I manually gave Flavio when reported an issue). I had a chance to give a quick try with 17.11.3 and master + v11 and I could reproduce the same issue. However, looks like the performance now is pretty much the same as without HWOF enabled and that's a significant improvement. I'd like to double check everything though. > > Ian > > > v11: > > > - Moved flow mark finding to happen after the next packet prefetch > > > and the metadata initialization. This to support with CT action > > > and to improve the performance. > > Flavio - following your comment on the performance. I am still not > able to reproduce it. Please check the above change which supposed > to help, and the series w/ 17.11.3 > If this still reproduce then I suggest we will have a debug session > to locate the root cause. Ian, I am afraid that the issue might be local and we most probably can fix with a follow up patch later, so if others are happy with the patchset, please go ahead. I can see this only in the mid of next week. Thanks, fbl > > > > - Don't search for flow mark in re-circulation. This is to address > > > the VXLAN issue reported. > > > - Set packet offsets as part of the tcp parsing. This is to address > > > the VXLAN issue reported. > > > - Fixed travis compilation issue. > > > - Fixed VLAN flow item to discard the VLAN_CFI bit. > > > - Fixed brackets location. > > > - rebased on top of master. > > > > > > v10: > > > - rebased on top of master. > > > > > > v9: - introduced IP packet sanity checks in a seperate commit > > > - moved structs and enums definition to the begining of the file > > > - fixed sparse compilation error (error is assumed to be fixed, > > > issues > > > on Travis CI preventing from fully testing it. > > > > > > v8: - enhanced documentation with more details on supported protocols > > > - fixed VLOG to start with capital letter > > > - fixed compilation issues > > > - fixed coding style > > > - addressed the rest of Ian's comments > > > > > > v7: - fixed wrong hash for mark_to_flow that has been refactored in v6 > > > - set the rss_conf for rss action to NULL, to workaround a mlx5 > > > change > > > in DPDK v17.11. Note that it will obey the rss settings OVS-DPDK > > > has > > > set in the beginning. Thus, nothing should be effected. > > > > > > v6: - fixed a sparse warning > > > - added documentation > > > - used hash_int to compute mark to flow hash > > > - added more comments > > > - added lock for pot lookup > > > - rebased on top of the latest code > > > > > > v5: - fixed an issue that it took too long if we do flow add/remove > > > repeatedly. > > > - removed an unused mutex lock > > > - turned most of the log level to DBG > > > - rebased on top of the latest code > > > > > > v4: - use RSS action instead of QUEUE action with MARK > > > - make it work with multiple queue (see patch 1) > > > - rebased on top of latest code > > > > > > v3: - The mark and id association is done with array instead of CMAP. > > > - Added a thread to do hw offload operations > > > - Removed macros completely > > > - dropped the patch to set FDIR_CONF, which is a workround some > > > Intel NICs. > > > - Added a debug patch to show all flow patterns we have created. > > > - Misc fixes > > > > > > v2: - workaround the queue action issue > > > - fixed the tcp_flags being skipped issue, which also fixed the > > > build warnings > > > - fixed l2 patterns for Intel nic > > > - Converted some macros to functions > > > - did not hardcode the max number of flow/action > > > - rebased on top of the latest code > > > > > > Thanks. > > > > > > --- > > > > > > Finn Christensen (1): > > > netdev-dpdk: implement flow offload with rte flow > > > > > > Yuanhan Liu (6): > > > dpif-netdev: associate flow with a mark id > > > flow: Introduce IP packet sanity checks > > > dpif-netdev: retrieve flow directly from the flow mark > > > netdev-dpdk: add debug for rte flow patterns > > > dpif-netdev: do hw flow offload in a thread > > > Documentation: document ovs-dpdk flow offload > > > > > > Documentation/howto/dpdk.rst | 22 ++ > > > NEWS | 2 +- > > > lib/dp-packet.h | 13 + > > > lib/dpif-netdev.c | 499 +++++++++++++++++++++++++- > > > lib/flow.c | 168 +++++++-- > > > lib/flow.h | 1 + > > > lib/netdev-dpdk.c | 734 > > +++++++++++++++++++++++++++++++++++++- > > > lib/netdev.h | 6 + > > > 8 files changed, 1405 insertions(+), 40 deletions(-) > > > > -- Flavio _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev