Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 20 Nov 2019, at 17:43, Ilya Maximets wrote: On 20.11.2019 8:35, Eelco Chaudron wrote: On 19 Nov 2019, at 17:52, Ilya Maximets wrote: On 19.11.2019 17:16, Eelco Chaudron wrote: On 7 Nov 2019, at 12:36, Ilya Maximets wrote: Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not be covered by existing API and we have to use slower generic XDP for such devices. There are few more issues, e.g. TCP is not supported in generic XDP mode for veth interfaces due to kernel limitations, however it is supported in native mode. This change introduces ability to use native XDP without zero-copy along with best-effort configuration option that enabled by default. In best-effort case OVS will sequentially try different modes starting from the fastest one and will choose the first acceptable for current interface. This will guarantee the best possible performance. If user will want to choose specific mode, it's still possible by setting the 'options:xdp-mode'. This change additionally changes the API by renaming the configuration knob from 'xdpmode' to 'xdp-mode' and also renaming the modes themselves to be more user-friendly. The full list of currently supported modes: * native-with-zerocopy - former DRV * native - new one, DRV without zero-copy * generic - former SKB * best-effort - new one, chooses the best available from 3 above modes Since 'best-effort' is a default mode, users will not need to explicitely set 'xdp-mode' in most cases. TCP related tests enabled back in system afxdp testsuite, because 'best-effort' will choose 'native' mode for veth interfaces and this mode has no issues with TCP. Patch in general looks good, two small comments inline. Thanks for review. The only thing that bothers me is the worse performance of the TAP interface with the new default config. Can we somehow keep the old behavior for TAP interfaces? Could you check if TCP works over tap interfaces in generic mode? For me the point is that correctness is better than performance. I also hope that native implementation for tap will be improved over time. So if I understood your email chain with William correctly TCP is not working, so I affray correctness is better than performance. Not exactly. William didn't test the actual TAP interfaces. I tested today with kernel vhost backed virtio-user port and it seems to pass TCP frames in generic mode. The setup is following: tap1 <-- ovs-vswitchd --> tap0 <-- testpmd --> tap2 tap1 -- tap port created by OVS (type=tap) tap0 -- tap port, virtio-user, created by testpmd, opened by OVS with type=afxdp tap2 -- tap port created by testpmd (net_tap) tap1 and tap2 are in their own network namespaces and iperf works on them. TCP works in this scenario regardless of xdp-mode on tap0 interface. However, let's look at the difference before-after this patch: PHY NIC TAP VETH before (skb default) slow by default fast by default broken TCP after (best-effort) fast by default slower by default works For the best speed before you had to configure xdp-mode for physical NICs, after you need to configure xdp-mode for TAPs. No much difference from the configuration side, but with patch applied veths are working. So, I think this change makes sense anyway. We can think about workaround for TAP devices, but we still need to check more cases. For now, I updated the 'Limitations' section with information that generic mode works for TAP devices and might be even faster in some cases. I also reformatted the code for 'unspecified' and 'best-effort' modes as you asked and applied to master. Best regards, Ilya Maximets. Thanks! //Eelco ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 20.11.2019 18:54, William Tu wrote:
> On Wed, Nov 20, 2019 at 05:43:48PM +0100, Ilya Maximets wrote:
>> On 20.11.2019 8:35, Eelco Chaudron wrote:
>>>
>>>
>>> On 19 Nov 2019, at 17:52, Ilya Maximets wrote:
>>>
On 19.11.2019 17:16, Eelco Chaudron wrote:
>
>
> On 7 Nov 2019, at 12:36, Ilya Maximets wrote:
>
>> Until now there was only two options for XDP mode in OVS: SKB or DRV.
>> i.e. 'generic XDP' or 'native XDP with zero-copy enabled'.
>>
>> Devices like 'veth' interfaces in Linux supports native XDP, but
>> doesn't support zero-copy mode. This case can not be covered by
>> existing API and we have to use slower generic XDP for such devices.
>> There are few more issues, e.g. TCP is not supported in generic XDP
>> mode for veth interfaces due to kernel limitations, however it is
>> supported in native mode.
>>
>> This change introduces ability to use native XDP without zero-copy
>> along with best-effort configuration option that enabled by default.
>> In best-effort case OVS will sequentially try different modes starting
>> from the fastest one and will choose the first acceptable for current
>> interface. This will guarantee the best possible performance.
>>
>> If user will want to choose specific mode, it's still possible by
>> setting the 'options:xdp-mode'.
>>
>> This change additionally changes the API by renaming the configuration
>> knob from 'xdpmode' to 'xdp-mode' and also renaming the modes
>> themselves to be more user-friendly.
>>
>> The full list of currently supported modes:
>> * native-with-zerocopy - former DRV
>> * native - new one, DRV without zero-copy
>> * generic - former SKB
>> * best-effort - new one, chooses the best available from
>> 3 above modes
>>
>> Since 'best-effort' is a default mode, users will not need to
>> explicitely set 'xdp-mode' in most cases.
>>
>> TCP related tests enabled back in system afxdp testsuite, because
>> 'best-effort' will choose 'native' mode for veth interfaces
>> and this mode has no issues with TCP.
> Patch in general looks good, two small comments inline.
Thanks for review.
>
> The only thing that bothers me is the worse performance of the TAP
> interface with the new default config. Can we somehow keep the old
> behavior for TAP interfaces?
Could you check if TCP works over tap interfaces in generic mode?
For me the point is that correctness is better than performance.
I also hope that native implementation for tap will be improved
over time.
>>>
>>> So if I understood your email chain with William correctly TCP is not
>>> working, so I affray correctness is better than performance.
>>
>> Not exactly. William didn't test the actual TAP interfaces.
>>
>> I tested today with kernel vhost backed virtio-user port and it seems to pass
>> TCP frames in generic mode.
>>
>> The setup is following:
>>
>> tap1 <-- ovs-vswitchd --> tap0 <-- testpmd --> tap2
>>
>> tap1 -- tap port created by OVS (type=tap)
>> tap0 -- tap port, virtio-user, created by testpmd, opened by OVS with
>> type=afxdp
>> tap2 -- tap port created by testpmd (net_tap)
>>
>> tap1 and tap2 are in their own network namespaces and iperf works on them.
>
> Hi Ilya,
>
> This is an interesting setup.
> Can you share roughly your commands to do this test?
I just copied and modified one of the system-dpdk tests like this:
diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
index a015d52f7..b0d10fcdd 100644
--- a/tests/system-dpdk.at
+++ b/tests/system-dpdk.at
@@ -232,3 +232,83 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch
kernel module is probably
\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
AT_CLEANUP
dnl --
+
+dnl --
+dnl Ping afxdp port
+AT_SETUP([OVS-DPDK - ping afxdp ports])
+AT_KEYWORDS([afxdp tap])
+OVS_DPDK_PRE_CHECK()
+AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
+OVS_DPDK_START()
+
+dnl Find number of sockets
+AT_CHECK([lscpu], [], [stdout])
+AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while (c++<$(3))
{printf "512,"}; print "512"}' > NUMA_NODE])
+
+dnl Add userspace bridge and attach it to OVS
+AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
+dnl Set up namespaces
+ADD_NAMESPACES(ns1, ns2)
+
+dnl Add veth device
+ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
+
+dnl Execute testpmd in background
+on_exit "pkill -f -x -9 'tail -f /dev/null'"
+tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
+ --vdev="net_virtio_user,path=/dev/vhost-net,queue_size=1024" \
+ --vdev="net_tap7,iface=tap7" --file-prefix page0 \
+
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On Wed, Nov 20, 2019 at 05:43:48PM +0100, Ilya Maximets wrote: > On 20.11.2019 8:35, Eelco Chaudron wrote: > > > > > > On 19 Nov 2019, at 17:52, Ilya Maximets wrote: > > > >> On 19.11.2019 17:16, Eelco Chaudron wrote: > >>> > >>> > >>> On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > >>> > Until now there was only two options for XDP mode in OVS: SKB or DRV. > i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. > > Devices like 'veth' interfaces in Linux supports native XDP, but > doesn't support zero-copy mode. This case can not be covered by > existing API and we have to use slower generic XDP for such devices. > There are few more issues, e.g. TCP is not supported in generic XDP > mode for veth interfaces due to kernel limitations, however it is > supported in native mode. > > This change introduces ability to use native XDP without zero-copy > along with best-effort configuration option that enabled by default. > In best-effort case OVS will sequentially try different modes starting > from the fastest one and will choose the first acceptable for current > interface. This will guarantee the best possible performance. > > If user will want to choose specific mode, it's still possible by > setting the 'options:xdp-mode'. > > This change additionally changes the API by renaming the configuration > knob from 'xdpmode' to 'xdp-mode' and also renaming the modes > themselves to be more user-friendly. > > The full list of currently supported modes: > * native-with-zerocopy - former DRV > * native - new one, DRV without zero-copy > * generic - former SKB > * best-effort - new one, chooses the best available from > 3 above modes > > Since 'best-effort' is a default mode, users will not need to > explicitely set 'xdp-mode' in most cases. > > TCP related tests enabled back in system afxdp testsuite, because > 'best-effort' will choose 'native' mode for veth interfaces > and this mode has no issues with TCP. > >>> Patch in general looks good, two small comments inline. > >> > >> Thanks for review. > >> > >>> > >>> The only thing that bothers me is the worse performance of the TAP > >>> interface with the new default config. Can we somehow keep the old > >>> behavior for TAP interfaces? > >> > >> Could you check if TCP works over tap interfaces in generic mode? > >> For me the point is that correctness is better than performance. > >> I also hope that native implementation for tap will be improved > >> over time. > > > > So if I understood your email chain with William correctly TCP is not > > working, so I affray correctness is better than performance. > > Not exactly. William didn't test the actual TAP interfaces. > > I tested today with kernel vhost backed virtio-user port and it seems to pass > TCP frames in generic mode. > > The setup is following: > > tap1 <-- ovs-vswitchd --> tap0 <-- testpmd --> tap2 > > tap1 -- tap port created by OVS (type=tap) > tap0 -- tap port, virtio-user, created by testpmd, opened by OVS with > type=afxdp > tap2 -- tap port created by testpmd (net_tap) > > tap1 and tap2 are in their own network namespaces and iperf works on them. Hi Ilya, This is an interesting setup. Can you share roughly your commands to do this test? btw, add Yiyang in the loop as he is going to try this setup on openstack environment. William > > TCP works in this scenario regardless of xdp-mode on tap0 interface. > > However, let's look at the difference before-after this patch: > > PHY NIC TAP VETH > before (skb default) slow by default fast by default broken TCP > after (best-effort) fast by default slower by default works > > For the best speed before you had to configure xdp-mode for physical NICs, > after you need to configure xdp-mode for TAPs. No much difference from the > configuration side, but with patch applied veths are working. > > So, I think this change makes sense anyway. We can think about workaround > for TAP devices, but we still need to check more cases. > > For now, I updated the 'Limitations' section with information that generic > mode works for TAP devices and might be even faster in some cases. > > I also reformatted the code for 'unspecified' and 'best-effort' modes as > you asked and applied to master. > > Best regards, Ilya Maximets. ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 20.11.2019 8:35, Eelco Chaudron wrote: > > > On 19 Nov 2019, at 17:52, Ilya Maximets wrote: > >> On 19.11.2019 17:16, Eelco Chaudron wrote: >>> >>> >>> On 7 Nov 2019, at 12:36, Ilya Maximets wrote: >>> Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not be covered by existing API and we have to use slower generic XDP for such devices. There are few more issues, e.g. TCP is not supported in generic XDP mode for veth interfaces due to kernel limitations, however it is supported in native mode. This change introduces ability to use native XDP without zero-copy along with best-effort configuration option that enabled by default. In best-effort case OVS will sequentially try different modes starting from the fastest one and will choose the first acceptable for current interface. This will guarantee the best possible performance. If user will want to choose specific mode, it's still possible by setting the 'options:xdp-mode'. This change additionally changes the API by renaming the configuration knob from 'xdpmode' to 'xdp-mode' and also renaming the modes themselves to be more user-friendly. The full list of currently supported modes: * native-with-zerocopy - former DRV * native - new one, DRV without zero-copy * generic - former SKB * best-effort - new one, chooses the best available from 3 above modes Since 'best-effort' is a default mode, users will not need to explicitely set 'xdp-mode' in most cases. TCP related tests enabled back in system afxdp testsuite, because 'best-effort' will choose 'native' mode for veth interfaces and this mode has no issues with TCP. >>> Patch in general looks good, two small comments inline. >> >> Thanks for review. >> >>> >>> The only thing that bothers me is the worse performance of the TAP >>> interface with the new default config. Can we somehow keep the old behavior >>> for TAP interfaces? >> >> Could you check if TCP works over tap interfaces in generic mode? >> For me the point is that correctness is better than performance. >> I also hope that native implementation for tap will be improved >> over time. > > So if I understood your email chain with William correctly TCP is not > working, so I affray correctness is better than performance. Not exactly. William didn't test the actual TAP interfaces. I tested today with kernel vhost backed virtio-user port and it seems to pass TCP frames in generic mode. The setup is following: tap1 <-- ovs-vswitchd --> tap0 <-- testpmd --> tap2 tap1 -- tap port created by OVS (type=tap) tap0 -- tap port, virtio-user, created by testpmd, opened by OVS with type=afxdp tap2 -- tap port created by testpmd (net_tap) tap1 and tap2 are in their own network namespaces and iperf works on them. TCP works in this scenario regardless of xdp-mode on tap0 interface. However, let's look at the difference before-after this patch: PHY NIC TAP VETH before (skb default) slow by default fast by default broken TCP after (best-effort) fast by default slower by default works For the best speed before you had to configure xdp-mode for physical NICs, after you need to configure xdp-mode for TAPs. No much difference from the configuration side, but with patch applied veths are working. So, I think this change makes sense anyway. We can think about workaround for TAP devices, but we still need to check more cases. For now, I updated the 'Limitations' section with information that generic mode works for TAP devices and might be even faster in some cases. I also reformatted the code for 'unspecified' and 'best-effort' modes as you asked and applied to master. Best regards, Ilya Maximets. ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 19 Nov 2019, at 17:52, Ilya Maximets wrote: On 19.11.2019 17:16, Eelco Chaudron wrote: On 7 Nov 2019, at 12:36, Ilya Maximets wrote: Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not be covered by existing API and we have to use slower generic XDP for such devices. There are few more issues, e.g. TCP is not supported in generic XDP mode for veth interfaces due to kernel limitations, however it is supported in native mode. This change introduces ability to use native XDP without zero-copy along with best-effort configuration option that enabled by default. In best-effort case OVS will sequentially try different modes starting from the fastest one and will choose the first acceptable for current interface. This will guarantee the best possible performance. If user will want to choose specific mode, it's still possible by setting the 'options:xdp-mode'. This change additionally changes the API by renaming the configuration knob from 'xdpmode' to 'xdp-mode' and also renaming the modes themselves to be more user-friendly. The full list of currently supported modes: * native-with-zerocopy - former DRV * native - new one, DRV without zero-copy * generic - former SKB * best-effort - new one, chooses the best available from 3 above modes Since 'best-effort' is a default mode, users will not need to explicitely set 'xdp-mode' in most cases. TCP related tests enabled back in system afxdp testsuite, because 'best-effort' will choose 'native' mode for veth interfaces and this mode has no issues with TCP. Patch in general looks good, two small comments inline. Thanks for review. The only thing that bothers me is the worse performance of the TAP interface with the new default config. Can we somehow keep the old behavior for TAP interfaces? Could you check if TCP works over tap interfaces in generic mode? For me the point is that correctness is better than performance. I also hope that native implementation for tap will be improved over time. So if I understood your email chain with William correctly TCP is not working, so I affray correctness is better than performance. Then this patch looks fine to see (see also comments below): Acked-by: Eelco Chaudron Signed-off-by: Ilya Maximets --- With this patch I modified the user-visible API, but I think it's OK since it's still an experimental netdev. Comments are welcome. Version 2: * Rebased on current master. Documentation/intro/install/afxdp.rst | 54 --- NEWS | 12 +- lib/netdev-afxdp.c | 223 -- lib/netdev-afxdp.h | 9 ++ lib/netdev-linux-private.h | 8 +- tests/system-afxdp-macros.at | 7 - vswitchd/vswitch.xml | 38 +++-- 7 files changed, 227 insertions(+), 124 deletions(-) diff --git a/Documentation/intro/install/afxdp.rst b/Documentation/intro/install/afxdp.rst index a136db0c9..937770ad0 100644 --- a/Documentation/intro/install/afxdp.rst +++ b/Documentation/intro/install/afxdp.rst @@ -153,9 +153,8 @@ To kick start end-to-end autotesting:: make check-afxdp TESTSUITEFLAGS='1' .. note:: - Not all test cases pass at this time. Currenly all TCP related - tests, ex: using wget or http, are skipped due to XDP limitations - on veth. cvlan test is also skipped. + Not all test cases pass at this time. Currenly all cvlan tests are skipped + due to kernel issues. If a test case fails, check the log at:: @@ -177,33 +176,35 @@ in :doc:`general`:: ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev Make sure your device driver support AF_XDP, netdev-afxdp supports -the following additional options (see man ovs-vswitchd.conf.db for +the following additional options (see ``man ovs-vswitchd.conf.db`` for more details): - * **xdpmode**: use "drv" for driver mode, or "skb" for skb mode. + * ``xdp-mode``: ``best-effort``, ``native-with-zerocopy``, + ``native`` or ``generic``. Defaults to ``best-effort``, i.e. best of + supported modes, so in most cases you don't need to change it. - * **use-need-wakeup**: default "true" if libbpf supports it, otherwise false. + * ``use-need-wakeup``: default ``true`` if libbpf supports it, + otherwise ``false``. For example, to use 1 PMD (on core 4) on 1 queue (queue 0) device, -configure these options: **pmd-cpu-mask, pmd-rxq-affinity, and n_rxq**. -The **xdpmode** can be "drv" or "skb":: +configure these options: ``pmd-cpu-mask``, ``pmd-rxq-affinity``, and +``n_rxq``:: ethtool -L enp2s0 combined 1 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x10 ovs-vsctl add-port br0 enp2s0 -- set interface enp2s0
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On Tue, Nov 19, 2019 at 09:26:40PM +0100, Ilya Maximets wrote: > On 19.11.2019 21:09, William Tu wrote: > > On Tue, Nov 19, 2019 at 08:51:19PM +0100, Ilya Maximets wrote: > >> On 19.11.2019 20:45, William Tu wrote: > >>> On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: > On 19.11.2019 17:16, Eelco Chaudron wrote: > > > > > > On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > > > >> Until now there was only two options for XDP mode in OVS: SKB or DRV. > >> i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. > >> > >> Devices like 'veth' interfaces in Linux supports native XDP, but > >> doesn't support zero-copy mode. This case can not be covered by > >> existing API and we have to use slower generic XDP for such devices. > >> There are few more issues, e.g. TCP is not supported in generic XDP > >> mode for veth interfaces due to kernel limitations, however it is > >> supported in native mode. > >> > >> This change introduces ability to use native XDP without zero-copy > >> along with best-effort configuration option that enabled by default. > >> In best-effort case OVS will sequentially try different modes starting > >> from the fastest one and will choose the first acceptable for current > >> interface. This will guarantee the best possible performance. > >> > >> If user will want to choose specific mode, it's still possible by > >> setting the 'options:xdp-mode'. > >> > >> This change additionally changes the API by renaming the configuration > >> knob from 'xdpmode' to 'xdp-mode' and also renaming the modes > >> themselves to be more user-friendly. > >> > >> The full list of currently supported modes: > >> * native-with-zerocopy - former DRV > >> * native - new one, DRV without zero-copy > >> * generic - former SKB > >> * best-effort - new one, chooses the best available from > >> 3 above modes > >> > >> Since 'best-effort' is a default mode, users will not need to > >> explicitely set 'xdp-mode' in most cases. > >> > >> TCP related tests enabled back in system afxdp testsuite, because > >> 'best-effort' will choose 'native' mode for veth interfaces > >> and this mode has no issues with TCP. > > Patch in general looks good, two small comments inline. > > Thanks for review. > > > > > The only thing that bothers me is the worse performance of the TAP > > interface with the new default config. Can we somehow keep the old > > behavior for TAP interfaces? > > Could you check if TCP works over tap interfaces in generic mode? > For me the point is that correctness is better than performance. > I also hope that native implementation for tap will be improved > over time. > > >>> > >>> I agree that we should first make sure correctness. > >>> I created a simple TCP test using br0, since br0 is a tap device. > >>> Unfortunately it does not work... > >>> > >>> --- > >>> ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev > >>> > >>> ip netns add at_ns0 > >>> ip link add p0 type veth peer name afxdp-p0 > >>> ip link set p0 netns at_ns0 > >>> ip link set dev afxdp-p0 up > >>> > >>> ovs-vsctl add-port br0 afxdp-p0 > >>> ovs-vsctl -- set interface afxdp-p0 options:n_rxq=1 type="afxdp" > >>> options:xdp-mode=native > >>> > >>> ip netns exec at_ns0 sh << NS_EXEC_HEREDOC > >>> ip addr add "10.1.1.1/24" dev p0 > >>> ip link set dev p0 up > >>> NS_EXEC_HEREDOC > >>> > >>> ovs-vsctl -- set interface br0 options:n_rxq=1 type="afxdp" > >>> options:xdp-mode=native > >> > >> I'm not sure if this is a valid thing to do. > >> After changing the netdev type this device is no linger TAP device. > >> Now it's a permanent tap device that detached from any application > >> and opened by af_xdp from the other side. > >> > >> BTW, changing the type of "internal" port doesn't sound safe. > > > > I see, thanks. > > > > Then if we don't change br0 to afxdp type, the TCP test > > below still fails, which I think should pass because > > we are using native mode for veth now. > > > > I think even native mode doesn't work for TCP? > > Disable Tx offloading on both p0 and br0 and it will work. So is it because when doing tx, af_xdp dev does not support tcp checksum offload, so kernel is dropping tcp packet? William ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On Tue, Nov 19, 2019 at 12:09:02PM -0800, William Tu wrote: > On Tue, Nov 19, 2019 at 08:51:19PM +0100, Ilya Maximets wrote: > > On 19.11.2019 20:45, William Tu wrote: > > > On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: > > >> On 19.11.2019 17:16, Eelco Chaudron wrote: > > >>> > > >>> > > >>> On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > > >>> > > Until now there was only two options for XDP mode in OVS: SKB or DRV. > > i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. > > > > Devices like 'veth' interfaces in Linux supports native XDP, but > > doesn't support zero-copy mode. This case can not be covered by > > existing API and we have to use slower generic XDP for such devices. > > There are few more issues, e.g. TCP is not supported in generic XDP > > mode for veth interfaces due to kernel limitations, however it is > > supported in native mode. > > > > This change introduces ability to use native XDP without zero-copy > > along with best-effort configuration option that enabled by default. > > In best-effort case OVS will sequentially try different modes starting > > from the fastest one and will choose the first acceptable for current > > interface. This will guarantee the best possible performance. > > > > If user will want to choose specific mode, it's still possible by > > setting the 'options:xdp-mode'. > > > > This change additionally changes the API by renaming the configuration > > knob from 'xdpmode' to 'xdp-mode' and also renaming the modes > > themselves to be more user-friendly. > > > > The full list of currently supported modes: > > * native-with-zerocopy - former DRV > > * native - new one, DRV without zero-copy > > * generic - former SKB > > * best-effort - new one, chooses the best available from > > 3 above modes > > > > Since 'best-effort' is a default mode, users will not need to > > explicitely set 'xdp-mode' in most cases. > > > > TCP related tests enabled back in system afxdp testsuite, because > > 'best-effort' will choose 'native' mode for veth interfaces > > and this mode has no issues with TCP. > > >>> Patch in general looks good, two small comments inline. > > >> > > >> Thanks for review. > > >> > > >>> > > >>> The only thing that bothers me is the worse performance of the TAP > > >>> interface with the new default config. Can we somehow keep the old > > >>> behavior for TAP interfaces? > > >> > > >> Could you check if TCP works over tap interfaces in generic mode? > > >> For me the point is that correctness is better than performance. > > >> I also hope that native implementation for tap will be improved > > >> over time. > > >> > > > > > > I agree that we should first make sure correctness. > > > I created a simple TCP test using br0, since br0 is a tap device. > > > Unfortunately it does not work... > > > > > > --- > > > ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev > > > > > > ip netns add at_ns0 > > > ip link add p0 type veth peer name afxdp-p0 > > > ip link set p0 netns at_ns0 > > > ip link set dev afxdp-p0 up > > > > > > ovs-vsctl add-port br0 afxdp-p0 > > > ovs-vsctl -- set interface afxdp-p0 options:n_rxq=1 type="afxdp" > > > options:xdp-mode=native > > > > > > ip netns exec at_ns0 sh << NS_EXEC_HEREDOC > > > ip addr add "10.1.1.1/24" dev p0 > > > ip link set dev p0 up > > > NS_EXEC_HEREDOC > > > > > > ovs-vsctl -- set interface br0 options:n_rxq=1 type="afxdp" > > > options:xdp-mode=native > > > > I'm not sure if this is a valid thing to do. > > After changing the netdev type this device is no linger TAP device. > > Now it's a permanent tap device that detached from any application > > and opened by af_xdp from the other side. > > > > BTW, changing the type of "internal" port doesn't sound safe. > > I see, thanks. > > Then if we don't change br0 to afxdp type, the TCP test > below still fails, which I think should pass because > we are using native mode for veth now. > > I think even native mode doesn't work for TCP? I forgot to turn off tx offload after doing ip netns exec at_ns0 ethtool -K p0 tx off TCP works fine. William ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 19.11.2019 21:09, William Tu wrote: > On Tue, Nov 19, 2019 at 08:51:19PM +0100, Ilya Maximets wrote: >> On 19.11.2019 20:45, William Tu wrote: >>> On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: On 19.11.2019 17:16, Eelco Chaudron wrote: > > > On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > >> Until now there was only two options for XDP mode in OVS: SKB or DRV. >> i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. >> >> Devices like 'veth' interfaces in Linux supports native XDP, but >> doesn't support zero-copy mode. This case can not be covered by >> existing API and we have to use slower generic XDP for such devices. >> There are few more issues, e.g. TCP is not supported in generic XDP >> mode for veth interfaces due to kernel limitations, however it is >> supported in native mode. >> >> This change introduces ability to use native XDP without zero-copy >> along with best-effort configuration option that enabled by default. >> In best-effort case OVS will sequentially try different modes starting >> from the fastest one and will choose the first acceptable for current >> interface. This will guarantee the best possible performance. >> >> If user will want to choose specific mode, it's still possible by >> setting the 'options:xdp-mode'. >> >> This change additionally changes the API by renaming the configuration >> knob from 'xdpmode' to 'xdp-mode' and also renaming the modes >> themselves to be more user-friendly. >> >> The full list of currently supported modes: >> * native-with-zerocopy - former DRV >> * native - new one, DRV without zero-copy >> * generic - former SKB >> * best-effort - new one, chooses the best available from >> 3 above modes >> >> Since 'best-effort' is a default mode, users will not need to >> explicitely set 'xdp-mode' in most cases. >> >> TCP related tests enabled back in system afxdp testsuite, because >> 'best-effort' will choose 'native' mode for veth interfaces >> and this mode has no issues with TCP. > Patch in general looks good, two small comments inline. Thanks for review. > > The only thing that bothers me is the worse performance of the TAP > interface with the new default config. Can we somehow keep the old > behavior for TAP interfaces? Could you check if TCP works over tap interfaces in generic mode? For me the point is that correctness is better than performance. I also hope that native implementation for tap will be improved over time. >>> >>> I agree that we should first make sure correctness. >>> I created a simple TCP test using br0, since br0 is a tap device. >>> Unfortunately it does not work... >>> >>> --- >>> ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev >>> >>> ip netns add at_ns0 >>> ip link add p0 type veth peer name afxdp-p0 >>> ip link set p0 netns at_ns0 >>> ip link set dev afxdp-p0 up >>> >>> ovs-vsctl add-port br0 afxdp-p0 >>> ovs-vsctl -- set interface afxdp-p0 options:n_rxq=1 type="afxdp" >>> options:xdp-mode=native >>> >>> ip netns exec at_ns0 sh << NS_EXEC_HEREDOC >>> ip addr add "10.1.1.1/24" dev p0 >>> ip link set dev p0 up >>> NS_EXEC_HEREDOC >>> >>> ovs-vsctl -- set interface br0 options:n_rxq=1 type="afxdp" >>> options:xdp-mode=native >> >> I'm not sure if this is a valid thing to do. >> After changing the netdev type this device is no linger TAP device. >> Now it's a permanent tap device that detached from any application >> and opened by af_xdp from the other side. >> >> BTW, changing the type of "internal" port doesn't sound safe. > > I see, thanks. > > Then if we don't change br0 to afxdp type, the TCP test > below still fails, which I think should pass because > we are using native mode for veth now. > > I think even native mode doesn't work for TCP? Disable Tx offloading on both p0 and br0 and it will work. > > --William >> >>> ip addr add "10.1.1.2/24" dev br0 >>> ip link set dev br0 up >>> >>> # UDP works ok >>> ip netns exec at_ns0 nc -u -l 1234 >>> nc -u 10.1.1.1 1234 >>> >>> >>> # TCP still fails >>> ip netns exec at_ns0 nc -l 1234 >>> nc 10.1.1.1 1234 >>> >>> ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On Tue, Nov 19, 2019 at 08:51:19PM +0100, Ilya Maximets wrote: > On 19.11.2019 20:45, William Tu wrote: > > On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: > >> On 19.11.2019 17:16, Eelco Chaudron wrote: > >>> > >>> > >>> On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > >>> > Until now there was only two options for XDP mode in OVS: SKB or DRV. > i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. > > Devices like 'veth' interfaces in Linux supports native XDP, but > doesn't support zero-copy mode. This case can not be covered by > existing API and we have to use slower generic XDP for such devices. > There are few more issues, e.g. TCP is not supported in generic XDP > mode for veth interfaces due to kernel limitations, however it is > supported in native mode. > > This change introduces ability to use native XDP without zero-copy > along with best-effort configuration option that enabled by default. > In best-effort case OVS will sequentially try different modes starting > from the fastest one and will choose the first acceptable for current > interface. This will guarantee the best possible performance. > > If user will want to choose specific mode, it's still possible by > setting the 'options:xdp-mode'. > > This change additionally changes the API by renaming the configuration > knob from 'xdpmode' to 'xdp-mode' and also renaming the modes > themselves to be more user-friendly. > > The full list of currently supported modes: > * native-with-zerocopy - former DRV > * native - new one, DRV without zero-copy > * generic - former SKB > * best-effort - new one, chooses the best available from > 3 above modes > > Since 'best-effort' is a default mode, users will not need to > explicitely set 'xdp-mode' in most cases. > > TCP related tests enabled back in system afxdp testsuite, because > 'best-effort' will choose 'native' mode for veth interfaces > and this mode has no issues with TCP. > >>> Patch in general looks good, two small comments inline. > >> > >> Thanks for review. > >> > >>> > >>> The only thing that bothers me is the worse performance of the TAP > >>> interface with the new default config. Can we somehow keep the old > >>> behavior for TAP interfaces? > >> > >> Could you check if TCP works over tap interfaces in generic mode? > >> For me the point is that correctness is better than performance. > >> I also hope that native implementation for tap will be improved > >> over time. > >> > > > > I agree that we should first make sure correctness. > > I created a simple TCP test using br0, since br0 is a tap device. > > Unfortunately it does not work... > > > > --- > > ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev > > > > ip netns add at_ns0 > > ip link add p0 type veth peer name afxdp-p0 > > ip link set p0 netns at_ns0 > > ip link set dev afxdp-p0 up > > > > ovs-vsctl add-port br0 afxdp-p0 > > ovs-vsctl -- set interface afxdp-p0 options:n_rxq=1 type="afxdp" > > options:xdp-mode=native > > > > ip netns exec at_ns0 sh << NS_EXEC_HEREDOC > > ip addr add "10.1.1.1/24" dev p0 > > ip link set dev p0 up > > NS_EXEC_HEREDOC > > > > ovs-vsctl -- set interface br0 options:n_rxq=1 type="afxdp" > > options:xdp-mode=native > > I'm not sure if this is a valid thing to do. > After changing the netdev type this device is no linger TAP device. > Now it's a permanent tap device that detached from any application > and opened by af_xdp from the other side. > > BTW, changing the type of "internal" port doesn't sound safe. I see, thanks. Then if we don't change br0 to afxdp type, the TCP test below still fails, which I think should pass because we are using native mode for veth now. I think even native mode doesn't work for TCP? --William > > > ip addr add "10.1.1.2/24" dev br0 > > ip link set dev br0 up > > > > # UDP works ok > > ip netns exec at_ns0 nc -u -l 1234 > > nc -u 10.1.1.1 1234 > > > > > > # TCP still fails > > ip netns exec at_ns0 nc -l 1234 > > nc 10.1.1.1 1234 > > > > ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 19.11.2019 20:45, William Tu wrote: > On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: >> On 19.11.2019 17:16, Eelco Chaudron wrote: >>> >>> >>> On 7 Nov 2019, at 12:36, Ilya Maximets wrote: >>> Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not be covered by existing API and we have to use slower generic XDP for such devices. There are few more issues, e.g. TCP is not supported in generic XDP mode for veth interfaces due to kernel limitations, however it is supported in native mode. This change introduces ability to use native XDP without zero-copy along with best-effort configuration option that enabled by default. In best-effort case OVS will sequentially try different modes starting from the fastest one and will choose the first acceptable for current interface. This will guarantee the best possible performance. If user will want to choose specific mode, it's still possible by setting the 'options:xdp-mode'. This change additionally changes the API by renaming the configuration knob from 'xdpmode' to 'xdp-mode' and also renaming the modes themselves to be more user-friendly. The full list of currently supported modes: * native-with-zerocopy - former DRV * native - new one, DRV without zero-copy * generic - former SKB * best-effort - new one, chooses the best available from 3 above modes Since 'best-effort' is a default mode, users will not need to explicitely set 'xdp-mode' in most cases. TCP related tests enabled back in system afxdp testsuite, because 'best-effort' will choose 'native' mode for veth interfaces and this mode has no issues with TCP. >>> Patch in general looks good, two small comments inline. >> >> Thanks for review. >> >>> >>> The only thing that bothers me is the worse performance of the TAP >>> interface with the new default config. Can we somehow keep the old behavior >>> for TAP interfaces? >> >> Could you check if TCP works over tap interfaces in generic mode? >> For me the point is that correctness is better than performance. >> I also hope that native implementation for tap will be improved >> over time. >> > > I agree that we should first make sure correctness. > I created a simple TCP test using br0, since br0 is a tap device. > Unfortunately it does not work... > > --- > ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev > > ip netns add at_ns0 > ip link add p0 type veth peer name afxdp-p0 > ip link set p0 netns at_ns0 > ip link set dev afxdp-p0 up > > ovs-vsctl add-port br0 afxdp-p0 > ovs-vsctl -- set interface afxdp-p0 options:n_rxq=1 type="afxdp" > options:xdp-mode=native > > ip netns exec at_ns0 sh << NS_EXEC_HEREDOC > ip addr add "10.1.1.1/24" dev p0 > ip link set dev p0 up > NS_EXEC_HEREDOC > > ovs-vsctl -- set interface br0 options:n_rxq=1 type="afxdp" > options:xdp-mode=native I'm not sure if this is a valid thing to do. After changing the netdev type this device is no linger TAP device. Now it's a permanent tap device that detached from any application and opened by af_xdp from the other side. BTW, changing the type of "internal" port doesn't sound safe. > ip addr add "10.1.1.2/24" dev br0 > ip link set dev br0 up > > # UDP works ok > ip netns exec at_ns0 nc -u -l 1234 > nc -u 10.1.1.1 1234 > > > # TCP still fails > ip netns exec at_ns0 nc -l 1234 > nc 10.1.1.1 1234 > > ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On Tue, Nov 19, 2019 at 05:52:22PM +0100, Ilya Maximets wrote: > On 19.11.2019 17:16, Eelco Chaudron wrote: > > > > > > On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > > > >> Until now there was only two options for XDP mode in OVS: SKB or DRV. > >> i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. > >> > >> Devices like 'veth' interfaces in Linux supports native XDP, but > >> doesn't support zero-copy mode. This case can not be covered by > >> existing API and we have to use slower generic XDP for such devices. > >> There are few more issues, e.g. TCP is not supported in generic XDP > >> mode for veth interfaces due to kernel limitations, however it is > >> supported in native mode. > >> > >> This change introduces ability to use native XDP without zero-copy > >> along with best-effort configuration option that enabled by default. > >> In best-effort case OVS will sequentially try different modes starting > >> from the fastest one and will choose the first acceptable for current > >> interface. This will guarantee the best possible performance. > >> > >> If user will want to choose specific mode, it's still possible by > >> setting the 'options:xdp-mode'. > >> > >> This change additionally changes the API by renaming the configuration > >> knob from 'xdpmode' to 'xdp-mode' and also renaming the modes > >> themselves to be more user-friendly. > >> > >> The full list of currently supported modes: > >> * native-with-zerocopy - former DRV > >> * native - new one, DRV without zero-copy > >> * generic - former SKB > >> * best-effort - new one, chooses the best available from > >> 3 above modes > >> > >> Since 'best-effort' is a default mode, users will not need to > >> explicitely set 'xdp-mode' in most cases. > >> > >> TCP related tests enabled back in system afxdp testsuite, because > >> 'best-effort' will choose 'native' mode for veth interfaces > >> and this mode has no issues with TCP. > > Patch in general looks good, two small comments inline. > > Thanks for review. > > > > > The only thing that bothers me is the worse performance of the TAP > > interface with the new default config. Can we somehow keep the old behavior > > for TAP interfaces? > > Could you check if TCP works over tap interfaces in generic mode? > For me the point is that correctness is better than performance. > I also hope that native implementation for tap will be improved > over time. > I agree that we should first make sure correctness. I created a simple TCP test using br0, since br0 is a tap device. Unfortunately it does not work... --- ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev ip netns add at_ns0 ip link add p0 type veth peer name afxdp-p0 ip link set p0 netns at_ns0 ip link set dev afxdp-p0 up ovs-vsctl add-port br0 afxdp-p0 ovs-vsctl -- set interface afxdp-p0 options:n_rxq=1 type="afxdp" options:xdp-mode=native ip netns exec at_ns0 sh << NS_EXEC_HEREDOC ip addr add "10.1.1.1/24" dev p0 ip link set dev p0 up NS_EXEC_HEREDOC ovs-vsctl -- set interface br0 options:n_rxq=1 type="afxdp" options:xdp-mode=native ip addr add "10.1.1.2/24" dev br0 ip link set dev br0 up # UDP works ok ip netns exec at_ns0 nc -u -l 1234 nc -u 10.1.1.1 1234 # TCP still fails ip netns exec at_ns0 nc -l 1234 nc 10.1.1.1 1234 ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 19.11.2019 17:16, Eelco Chaudron wrote: > > > On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > >> Until now there was only two options for XDP mode in OVS: SKB or DRV. >> i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. >> >> Devices like 'veth' interfaces in Linux supports native XDP, but >> doesn't support zero-copy mode. This case can not be covered by >> existing API and we have to use slower generic XDP for such devices. >> There are few more issues, e.g. TCP is not supported in generic XDP >> mode for veth interfaces due to kernel limitations, however it is >> supported in native mode. >> >> This change introduces ability to use native XDP without zero-copy >> along with best-effort configuration option that enabled by default. >> In best-effort case OVS will sequentially try different modes starting >> from the fastest one and will choose the first acceptable for current >> interface. This will guarantee the best possible performance. >> >> If user will want to choose specific mode, it's still possible by >> setting the 'options:xdp-mode'. >> >> This change additionally changes the API by renaming the configuration >> knob from 'xdpmode' to 'xdp-mode' and also renaming the modes >> themselves to be more user-friendly. >> >> The full list of currently supported modes: >> * native-with-zerocopy - former DRV >> * native - new one, DRV without zero-copy >> * generic - former SKB >> * best-effort - new one, chooses the best available from >> 3 above modes >> >> Since 'best-effort' is a default mode, users will not need to >> explicitely set 'xdp-mode' in most cases. >> >> TCP related tests enabled back in system afxdp testsuite, because >> 'best-effort' will choose 'native' mode for veth interfaces >> and this mode has no issues with TCP. > Patch in general looks good, two small comments inline. Thanks for review. > > The only thing that bothers me is the worse performance of the TAP interface > with the new default config. Can we somehow keep the old behavior for TAP > interfaces? Could you check if TCP works over tap interfaces in generic mode? For me the point is that correctness is better than performance. I also hope that native implementation for tap will be improved over time. > >> Signed-off-by: Ilya Maximets >> --- >> >> With this patch I modified the user-visible API, but I think it's OK >> since it's still an experimental netdev. Comments are welcome. >> >> Version 2: >> * Rebased on current master. >> >> Documentation/intro/install/afxdp.rst | 54 --- >> NEWS | 12 +- >> lib/netdev-afxdp.c | 223 -- >> lib/netdev-afxdp.h | 9 ++ >> lib/netdev-linux-private.h | 8 +- >> tests/system-afxdp-macros.at | 7 - >> vswitchd/vswitch.xml | 38 +++-- >> 7 files changed, 227 insertions(+), 124 deletions(-) >> >> diff --git a/Documentation/intro/install/afxdp.rst >> b/Documentation/intro/install/afxdp.rst >> index a136db0c9..937770ad0 100644 >> --- a/Documentation/intro/install/afxdp.rst >> +++ b/Documentation/intro/install/afxdp.rst >> @@ -153,9 +153,8 @@ To kick start end-to-end autotesting:: >> make check-afxdp TESTSUITEFLAGS='1' >> >> .. note:: >> - Not all test cases pass at this time. Currenly all TCP related >> - tests, ex: using wget or http, are skipped due to XDP limitations >> - on veth. cvlan test is also skipped. >> + Not all test cases pass at this time. Currenly all cvlan tests are >> skipped >> + due to kernel issues. >> >> If a test case fails, check the log at:: >> >> @@ -177,33 +176,35 @@ in :doc:`general`:: >> ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev >> >> Make sure your device driver support AF_XDP, netdev-afxdp supports >> -the following additional options (see man ovs-vswitchd.conf.db for >> +the following additional options (see ``man ovs-vswitchd.conf.db`` for >> more details): >> >> - * **xdpmode**: use "drv" for driver mode, or "skb" for skb mode. >> + * ``xdp-mode``: ``best-effort``, ``native-with-zerocopy``, >> + ``native`` or ``generic``. Defaults to ``best-effort``, i.e. best of >> + supported modes, so in most cases you don't need to change it. >> >> - * **use-need-wakeup**: default "true" if libbpf supports it, otherwise >> false. >> + * ``use-need-wakeup``: default ``true`` if libbpf supports it, >> + otherwise ``false``. >> >> For example, to use 1 PMD (on core 4) on 1 queue (queue 0) device, >> -configure these options: **pmd-cpu-mask, pmd-rxq-affinity, and n_rxq**. >> -The **xdpmode** can be "drv" or "skb":: >> +configure these options: ``pmd-cpu-mask``, ``pmd-rxq-affinity``, and >> +``n_rxq``:: >> >> ethtool -L enp2s0 combined 1 >> ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x10 >> ovs-vsctl add-port br0 enp2s0 -- set interface enp2s0 type="afxdp" \ >>
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 7 Nov 2019, at 12:36, Ilya Maximets wrote: Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not be covered by existing API and we have to use slower generic XDP for such devices. There are few more issues, e.g. TCP is not supported in generic XDP mode for veth interfaces due to kernel limitations, however it is supported in native mode. This change introduces ability to use native XDP without zero-copy along with best-effort configuration option that enabled by default. In best-effort case OVS will sequentially try different modes starting from the fastest one and will choose the first acceptable for current interface. This will guarantee the best possible performance. If user will want to choose specific mode, it's still possible by setting the 'options:xdp-mode'. This change additionally changes the API by renaming the configuration knob from 'xdpmode' to 'xdp-mode' and also renaming the modes themselves to be more user-friendly. The full list of currently supported modes: * native-with-zerocopy - former DRV * native - new one, DRV without zero-copy * generic - former SKB * best-effort - new one, chooses the best available from 3 above modes Since 'best-effort' is a default mode, users will not need to explicitely set 'xdp-mode' in most cases. TCP related tests enabled back in system afxdp testsuite, because 'best-effort' will choose 'native' mode for veth interfaces and this mode has no issues with TCP. Patch in general looks good, two small comments inline. The only thing that bothers me is the worse performance of the TAP interface with the new default config. Can we somehow keep the old behavior for TAP interfaces? Signed-off-by: Ilya Maximets --- With this patch I modified the user-visible API, but I think it's OK since it's still an experimental netdev. Comments are welcome. Version 2: * Rebased on current master. Documentation/intro/install/afxdp.rst | 54 --- NEWS | 12 +- lib/netdev-afxdp.c| 223 -- lib/netdev-afxdp.h| 9 ++ lib/netdev-linux-private.h| 8 +- tests/system-afxdp-macros.at | 7 - vswitchd/vswitch.xml | 38 +++-- 7 files changed, 227 insertions(+), 124 deletions(-) diff --git a/Documentation/intro/install/afxdp.rst b/Documentation/intro/install/afxdp.rst index a136db0c9..937770ad0 100644 --- a/Documentation/intro/install/afxdp.rst +++ b/Documentation/intro/install/afxdp.rst @@ -153,9 +153,8 @@ To kick start end-to-end autotesting:: make check-afxdp TESTSUITEFLAGS='1' .. note:: - Not all test cases pass at this time. Currenly all TCP related - tests, ex: using wget or http, are skipped due to XDP limitations - on veth. cvlan test is also skipped. + Not all test cases pass at this time. Currenly all cvlan tests are skipped + due to kernel issues. If a test case fails, check the log at:: @@ -177,33 +176,35 @@ in :doc:`general`:: ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev Make sure your device driver support AF_XDP, netdev-afxdp supports -the following additional options (see man ovs-vswitchd.conf.db for +the following additional options (see ``man ovs-vswitchd.conf.db`` for more details): - * **xdpmode**: use "drv" for driver mode, or "skb" for skb mode. + * ``xdp-mode``: ``best-effort``, ``native-with-zerocopy``, + ``native`` or ``generic``. Defaults to ``best-effort``, i.e. best of + supported modes, so in most cases you don't need to change it. - * **use-need-wakeup**: default "true" if libbpf supports it, otherwise false. + * ``use-need-wakeup``: default ``true`` if libbpf supports it, + otherwise ``false``. For example, to use 1 PMD (on core 4) on 1 queue (queue 0) device, -configure these options: **pmd-cpu-mask, pmd-rxq-affinity, and n_rxq**. -The **xdpmode** can be "drv" or "skb":: +configure these options: ``pmd-cpu-mask``, ``pmd-rxq-affinity``, and +``n_rxq``:: ethtool -L enp2s0 combined 1 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x10 ovs-vsctl add-port br0 enp2s0 -- set interface enp2s0 type="afxdp" \ -options:n_rxq=1 options:xdpmode=drv \ -other_config:pmd-rxq-affinity="0:4" + other_config:pmd-rxq-affinity="0:4" Or, use 4 pmds/cores and 4 queues by doing:: ethtool -L enp2s0 combined 4 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x36 ovs-vsctl add-port br0 enp2s0 -- set interface enp2s0 type="afxdp" \ -options:n_rxq=4 options:xdpmode=drv \ -other_config:pmd-rxq-affinity="0:1,1:2,2:3,3:4" +options:n_rxq=4 other_config:pmd-rxq-affinity="0:1,1:2,2:3,3:4" .. note:: -
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 13 Nov 2019, at 14:23, Ilya Maximets wrote: On 13.11.2019 1:21, William Tu wrote: On Thu, Nov 7, 2019 at 3:36 AM Ilya Maximets wrote: Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not be covered by existing API and we have to use slower generic XDP for such devices. There are few more issues, e.g. TCP is not supported in generic XDP mode for veth interfaces due to kernel limitations, however it is supported in native mode. This change introduces ability to use native XDP without zero-copy along with best-effort configuration option that enabled by default. In best-effort case OVS will sequentially try different modes starting from the fastest one and will choose the first acceptable for current interface. This will guarantee the best possible performance. If user will want to choose specific mode, it's still possible by setting the 'options:xdp-mode'. This change additionally changes the API by renaming the configuration knob from 'xdpmode' to 'xdp-mode' and also renaming the modes themselves to be more user-friendly. The full list of currently supported modes: * native-with-zerocopy - former DRV * native - new one, DRV without zero-copy * generic - former SKB * best-effort - new one, chooses the best available from 3 above modes Since 'best-effort' is a default mode, users will not need to explicitely set 'xdp-mode' in most cases. TCP related tests enabled back in system afxdp testsuite, because 'best-effort' will choose 'native' mode for veth interfaces and this mode has no issues with TCP. Signed-off-by: Ilya Maximets --- I'm thinking about adding some tests case and patches depends on this API change. If there is no more comment, I will apply to master. I think, it might be good to wait for some comments from Eelco. @Eelco, do you have a plan to review the code? Do you agree with the API update? I’m rather busy this week, will try to review it next week, and I’m ok with the API change. The only thing that bothers me is the worse performance of the TAP interface with the new default config. ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 13.11.2019 1:21, William Tu wrote: > On Thu, Nov 7, 2019 at 3:36 AM Ilya Maximets wrote: >> >> Until now there was only two options for XDP mode in OVS: SKB or DRV. >> i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. >> >> Devices like 'veth' interfaces in Linux supports native XDP, but >> doesn't support zero-copy mode. This case can not be covered by >> existing API and we have to use slower generic XDP for such devices. >> There are few more issues, e.g. TCP is not supported in generic XDP >> mode for veth interfaces due to kernel limitations, however it is >> supported in native mode. >> >> This change introduces ability to use native XDP without zero-copy >> along with best-effort configuration option that enabled by default. >> In best-effort case OVS will sequentially try different modes starting >> from the fastest one and will choose the first acceptable for current >> interface. This will guarantee the best possible performance. >> >> If user will want to choose specific mode, it's still possible by >> setting the 'options:xdp-mode'. >> >> This change additionally changes the API by renaming the configuration >> knob from 'xdpmode' to 'xdp-mode' and also renaming the modes >> themselves to be more user-friendly. >> >> The full list of currently supported modes: >> * native-with-zerocopy - former DRV >> * native - new one, DRV without zero-copy >> * generic - former SKB >> * best-effort - new one, chooses the best available from >>3 above modes >> >> Since 'best-effort' is a default mode, users will not need to >> explicitely set 'xdp-mode' in most cases. >> >> TCP related tests enabled back in system afxdp testsuite, because >> 'best-effort' will choose 'native' mode for veth interfaces >> and this mode has no issues with TCP. >> >> Signed-off-by: Ilya Maximets >> --- > > I'm thinking about adding some tests case and patches depends on > this API change. If there is no more comment, I will apply to master. I think, it might be good to wait for some comments from Eelco. @Eelco, do you have a plan to review the code? Do you agree with the API update? Best regards, Ilya Maximets. ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 08.11.2019 15:22, William Tu wrote: > On Thu, Nov 07, 2019 at 12:36:33PM +0100, Ilya Maximets wrote: >> Until now there was only two options for XDP mode in OVS: SKB or DRV. >> i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. >> >> Devices like 'veth' interfaces in Linux supports native XDP, but >> doesn't support zero-copy mode. This case can not be covered by >> existing API and we have to use slower generic XDP for such devices. >> There are few more issues, e.g. TCP is not supported in generic XDP >> mode for veth interfaces due to kernel limitations, however it is >> supported in native mode. >> >> This change introduces ability to use native XDP without zero-copy >> along with best-effort configuration option that enabled by default. >> In best-effort case OVS will sequentially try different modes starting >> from the fastest one and will choose the first acceptable for current >> interface. This will guarantee the best possible performance. >> >> If user will want to choose specific mode, it's still possible by >> setting the 'options:xdp-mode'. >> >> This change additionally changes the API by renaming the configuration >> knob from 'xdpmode' to 'xdp-mode' and also renaming the modes >> themselves to be more user-friendly. >> >> The full list of currently supported modes: >> * native-with-zerocopy - former DRV >> * native - new one, DRV without zero-copy >> * generic - former SKB >> * best-effort - new one, chooses the best available from >>3 above modes > > Since we are renaming the mode, in doc, should we tell user the mapping > of these mode to kernel AF_XDP's mode? > So let user know generic mode in OVS = generic SKB in kernel, > native mode in OVS = native mode without zc... It might make sense to document that 'generic' uses 'XDP_SKB' and 'native' uses 'XDP_DRV', but it seems that 'generic'/'native' terms are widely used in XDP related community and projects, i.e. should be well known. Best regards, Ilya Maximets. ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On Thu, Nov 7, 2019 at 3:36 AM Ilya Maximets wrote: > > Until now there was only two options for XDP mode in OVS: SKB or DRV. > i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. > > Devices like 'veth' interfaces in Linux supports native XDP, but > doesn't support zero-copy mode. This case can not be covered by > existing API and we have to use slower generic XDP for such devices. > There are few more issues, e.g. TCP is not supported in generic XDP > mode for veth interfaces due to kernel limitations, however it is > supported in native mode. > > This change introduces ability to use native XDP without zero-copy > along with best-effort configuration option that enabled by default. > In best-effort case OVS will sequentially try different modes starting > from the fastest one and will choose the first acceptable for current > interface. This will guarantee the best possible performance. > > If user will want to choose specific mode, it's still possible by > setting the 'options:xdp-mode'. > > This change additionally changes the API by renaming the configuration > knob from 'xdpmode' to 'xdp-mode' and also renaming the modes > themselves to be more user-friendly. > > The full list of currently supported modes: > * native-with-zerocopy - former DRV > * native - new one, DRV without zero-copy > * generic - former SKB > * best-effort - new one, chooses the best available from >3 above modes > > Since 'best-effort' is a default mode, users will not need to > explicitely set 'xdp-mode' in most cases. > > TCP related tests enabled back in system afxdp testsuite, because > 'best-effort' will choose 'native' mode for veth interfaces > and this mode has no issues with TCP. > > Signed-off-by: Ilya Maximets > --- I'm thinking about adding some tests case and patches depends on this API change. If there is no more comment, I will apply to master. Regards, William ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On Thu, Nov 07, 2019 at 01:39:28PM +0100, Eelco Chaudron wrote: > > > On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > > >Until now there was only two options for XDP mode in OVS: SKB or DRV. > >i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. > > > >Devices like 'veth' interfaces in Linux supports native XDP, but > >doesn't support zero-copy mode. This case can not be covered by > >existing API and we have to use slower generic XDP for such devices. > >There are few more issues, e.g. TCP is not supported in generic XDP > >mode for veth interfaces due to kernel limitations, however it is > >supported in native mode. > > > >This change introduces ability to use native XDP without zero-copy > >along with best-effort configuration option that enabled by default. > >In best-effort case OVS will sequentially try different modes starting > >from the fastest one and will choose the first acceptable for current > >interface. This will guarantee the best possible performance. > > > >If user will want to choose specific mode, it's still possible by > >setting the 'options:xdp-mode'. > > > >This change additionally changes the API by renaming the configuration > >knob from 'xdpmode' to 'xdp-mode' and also renaming the modes > >themselves to be more user-friendly. > > > >The full list of currently supported modes: > > * native-with-zerocopy - former DRV > > * native - new one, DRV without zero-copy > > * generic - former SKB > > * best-effort - new one, chooses the best available from > > 3 above modes > > > >Since 'best-effort' is a default mode, users will not need to > >explicitely set 'xdp-mode' in most cases. > > > >TCP related tests enabled back in system afxdp testsuite, because > >'best-effort' will choose 'native' mode for veth interfaces > >and this mode has no issues with TCP. > > > >Signed-off-by: Ilya Maximets > > No review, I was running some performance tests for the OVS conference > presentation so quickly tried this patch (assuming performance would > increase :)… > > So with the native option (default for tap) I see a decrease, :(, in > performance over the skb mode (this is with my usual ovs_perf PVP test > setup). > > With the patch applied, and default configuration > (xdp-mode-in-use=native-with-zerocopy for my tap): > > "Physical to Virtual to Physical test, L3 flows[port redirect]" > , Packet size > Number of flows, 64 >1, 711581 > 100, 673152 > 1000, 617733 > > Here you will even see a performance drop with multiple IP flows (this is a > single queue config). > > With SKB mode (xdp-mode-in-use=generic): > > "Physical to Virtual to Physical test, L3 flows[port redirect]" > , Packet size > Number of flows, 64 >1, 800796 > 100, 800912 > 1000, 800133 > Hi Eelco, I have another question (not directly related to this patch). When doing PVP testing, what's the total CPU consumption? I'm trying to think about the efficiency, ex: Packet forwarding rate / CPU utilization For native-with-zerocopy, is it correct that your system uses ksoftirqd/x 100% (rx port) ksoftirqd/y 100% (tx port) qemu-kvm100% vhost_net 100% ovs-vswitchd 100% (pmd thead) Or with need_wakeup, it's lower than this number? Thanks William ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On Thu, Nov 07, 2019 at 12:36:33PM +0100, Ilya Maximets wrote: > Until now there was only two options for XDP mode in OVS: SKB or DRV. > i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. > > Devices like 'veth' interfaces in Linux supports native XDP, but > doesn't support zero-copy mode. This case can not be covered by > existing API and we have to use slower generic XDP for such devices. > There are few more issues, e.g. TCP is not supported in generic XDP > mode for veth interfaces due to kernel limitations, however it is > supported in native mode. > > This change introduces ability to use native XDP without zero-copy > along with best-effort configuration option that enabled by default. > In best-effort case OVS will sequentially try different modes starting > from the fastest one and will choose the first acceptable for current > interface. This will guarantee the best possible performance. > > If user will want to choose specific mode, it's still possible by > setting the 'options:xdp-mode'. > > This change additionally changes the API by renaming the configuration > knob from 'xdpmode' to 'xdp-mode' and also renaming the modes > themselves to be more user-friendly. > > The full list of currently supported modes: > * native-with-zerocopy - former DRV > * native - new one, DRV without zero-copy > * generic - former SKB > * best-effort - new one, chooses the best available from >3 above modes Since we are renaming the mode, in doc, should we tell user the mapping of these mode to kernel AF_XDP's mode? So let user know generic mode in OVS = generic SKB in kernel, native mode in OVS = native mode without zc... > > Since 'best-effort' is a default mode, users will not need to > explicitely set 'xdp-mode' in most cases. > > TCP related tests enabled back in system afxdp testsuite, because > 'best-effort' will choose 'native' mode for veth interfaces > and this mode has no issues with TCP. > > Signed-off-by: Ilya Maximets > --- LGTM. Acked-by: William Tu I look through the patch and everything looks good. Thanks for fixing the documentation. 'make check-afxdp' also pass the previous failed cases due to TCP. It would also be good if this 'best-effort' mode is supported in libbpf. I think other users of af_xdp also have the same configuration headache. Regards, William > > With this patch I modified the user-visible API, but I think it's OK > since it's still an experimental netdev. Comments are welcome. > > Version 2: > * Rebased on current master. > > Documentation/intro/install/afxdp.rst | 54 --- > NEWS | 12 +- > lib/netdev-afxdp.c| 223 -- > lib/netdev-afxdp.h| 9 ++ > lib/netdev-linux-private.h| 8 +- > tests/system-afxdp-macros.at | 7 - > vswitchd/vswitch.xml | 38 +++-- > 7 files changed, 227 insertions(+), 124 deletions(-) > > diff --git a/Documentation/intro/install/afxdp.rst > b/Documentation/intro/install/afxdp.rst > index a136db0c9..937770ad0 100644 > --- a/Documentation/intro/install/afxdp.rst > +++ b/Documentation/intro/install/afxdp.rst > @@ -153,9 +153,8 @@ To kick start end-to-end autotesting:: >make check-afxdp TESTSUITEFLAGS='1' > > .. note:: > - Not all test cases pass at this time. Currenly all TCP related > - tests, ex: using wget or http, are skipped due to XDP limitations > - on veth. cvlan test is also skipped. > + Not all test cases pass at this time. Currenly all cvlan tests are skipped > + due to kernel issues. > > If a test case fails, check the log at:: > > @@ -177,33 +176,35 @@ in :doc:`general`:: >ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev > > Make sure your device driver support AF_XDP, netdev-afxdp supports > -the following additional options (see man ovs-vswitchd.conf.db for > +the following additional options (see ``man ovs-vswitchd.conf.db`` for > more details): > > - * **xdpmode**: use "drv" for driver mode, or "skb" for skb mode. > + * ``xdp-mode``: ``best-effort``, ``native-with-zerocopy``, > + ``native`` or ``generic``. Defaults to ``best-effort``, i.e. best of > + supported modes, so in most cases you don't need to change it. > > - * **use-need-wakeup**: default "true" if libbpf supports it, otherwise > false. > + * ``use-need-wakeup``: default ``true`` if libbpf supports it, > + otherwise ``false``. > > For example, to use 1 PMD (on core 4) on 1 queue (queue 0) device, > -configure these options: **pmd-cpu-mask, pmd-rxq-affinity, and n_rxq**. > -The **xdpmode** can be "drv" or "skb":: > +configure these options: ``pmd-cpu-mask``, ``pmd-rxq-affinity``, and > +``n_rxq``:: > >ethtool -L enp2s0 combined 1 >ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x10 >ovs-vsctl add-port br0 enp2s0 -- set interface enp2s0 type="afxdp" \ >
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On Thu, Nov 07, 2019 at 02:28:18PM +0100, Eelco Chaudron wrote: > > > On 7 Nov 2019, at 14:21, Ilya Maximets wrote: > > >On 07.11.2019 13:39, Eelco Chaudron wrote: > >> > >> > >>On 7 Nov 2019, at 12:36, Ilya Maximets wrote: > >> > >>>Until now there was only two options for XDP mode in OVS: SKB or DRV. > >>>i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. > >>> > >>>Devices like 'veth' interfaces in Linux supports native XDP, but > >>>doesn't support zero-copy mode. This case can not be covered by > >>>existing API and we have to use slower generic XDP for such devices. > >>>There are few more issues, e.g. TCP is not supported in generic XDP > >>>mode for veth interfaces due to kernel limitations, however it is > >>>supported in native mode. > >>> > >>>This change introduces ability to use native XDP without zero-copy > >>>along with best-effort configuration option that enabled by default. > >>>In best-effort case OVS will sequentially try different modes starting > >>>from the fastest one and will choose the first acceptable for current > >>>interface. This will guarantee the best possible performance. > >>> > >>>If user will want to choose specific mode, it's still possible by > >>>setting the 'options:xdp-mode'. > >>> > >>>This change additionally changes the API by renaming the configuration > >>>knob from 'xdpmode' to 'xdp-mode' and also renaming the modes > >>>themselves to be more user-friendly. > >>> > >>>The full list of currently supported modes: > >>> * native-with-zerocopy - former DRV > >>> * native - new one, DRV without zero-copy > >>> * generic - former SKB > >>> * best-effort - new one, chooses the best available from > >>> 3 above modes > >>> > >>>Since 'best-effort' is a default mode, users will not need to > >>>explicitely set 'xdp-mode' in most cases. > >>> > >>>TCP related tests enabled back in system afxdp testsuite, because > >>>'best-effort' will choose 'native' mode for veth interfaces > >>>and this mode has no issues with TCP. > >>> > >>>Signed-off-by: Ilya Maximets > >> > >>No review, I was running some performance tests for the OVS conference > >>presentation so quickly tried this patch (assuming performance would > >>increase :)… > >> > >>So with the native option (default for tap) I see a decrease, :(, in > >>performance over the skb mode (this is with my usual ovs_perf PVP test > >>setup). > >> > >>With the patch applied, and default configuration > >>(xdp-mode-in-use=native-with-zerocopy for my tap): > > > >Hmm. tap supports zero-copy? Interesting. No, tap doesn't support zero-copy mode. > >Have you tried forcing 'native' mode without zero-copy? > >Maybe it will make sense to enable/disable need-wakeup feature. > > Oops, wrong cut/paste: xdp-mode-in-use=native > > All my the tests where with use-need-wakeup=true > > port 2: tapVM (afxdp: n_rxq=1, use-need-wakeup=true, xdp-mode=best-effort, > xdp-mode-in-use=native) > > > > >> > >>"Physical to Virtual to Physical test, L3 flows[port redirect]" > >>, Packet size > >>Number of flows, 64 > >> 1, 711581 > >> 100, 673152 > >>1000, 617733 > >> > >>Here you will even see a performance drop with multiple IP flows (this > >>is a single queue config). > > > >This is strange.. One explanation might be that For using tap with native mode, OVS is sending XDP frame into the kernel, but since tap device eventually has to create an skb and forward to vhost_net then QEMU, so anyway we are having skb overhead. On the other hand For using tap with skb mode, there might be more optimization exist in kernel, some batch allocation and batch rx/tx (See driver/net/tun.c) --William ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 7 Nov 2019, at 14:21, Ilya Maximets wrote: On 07.11.2019 13:39, Eelco Chaudron wrote: On 7 Nov 2019, at 12:36, Ilya Maximets wrote: Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not be covered by existing API and we have to use slower generic XDP for such devices. There are few more issues, e.g. TCP is not supported in generic XDP mode for veth interfaces due to kernel limitations, however it is supported in native mode. This change introduces ability to use native XDP without zero-copy along with best-effort configuration option that enabled by default. In best-effort case OVS will sequentially try different modes starting from the fastest one and will choose the first acceptable for current interface. This will guarantee the best possible performance. If user will want to choose specific mode, it's still possible by setting the 'options:xdp-mode'. This change additionally changes the API by renaming the configuration knob from 'xdpmode' to 'xdp-mode' and also renaming the modes themselves to be more user-friendly. The full list of currently supported modes: * native-with-zerocopy - former DRV * native - new one, DRV without zero-copy * generic - former SKB * best-effort - new one, chooses the best available from 3 above modes Since 'best-effort' is a default mode, users will not need to explicitely set 'xdp-mode' in most cases. TCP related tests enabled back in system afxdp testsuite, because 'best-effort' will choose 'native' mode for veth interfaces and this mode has no issues with TCP. Signed-off-by: Ilya Maximets No review, I was running some performance tests for the OVS conference presentation so quickly tried this patch (assuming performance would increase :)… So with the native option (default for tap) I see a decrease, :(, in performance over the skb mode (this is with my usual ovs_perf PVP test setup). With the patch applied, and default configuration (xdp-mode-in-use=native-with-zerocopy for my tap): Hmm. tap supports zero-copy? Interesting. Have you tried forcing 'native' mode without zero-copy? Maybe it will make sense to enable/disable need-wakeup feature. Oops, wrong cut/paste: xdp-mode-in-use=native All my the tests where with use-need-wakeup=true port 2: tapVM (afxdp: n_rxq=1, use-need-wakeup=true, xdp-mode=best-effort, xdp-mode-in-use=native) "Physical to Virtual to Physical test, L3 flows[port redirect]" , Packet size Number of flows, 64 1, 711581 100, 673152 1000, 617733 Here you will even see a performance drop with multiple IP flows (this is a single queue config). This is strange.. With SKB mode (xdp-mode-in-use=generic): I'm curious, does TCP work via tap interfaces in generic mode? "Physical to Virtual to Physical test, L3 flows[port redirect]" , Packet size Number of flows, 64 1, 800796 100, 800912 1000, 800133 Cheers, Eelco ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 07.11.2019 13:39, Eelco Chaudron wrote: On 7 Nov 2019, at 12:36, Ilya Maximets wrote: Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not be covered by existing API and we have to use slower generic XDP for such devices. There are few more issues, e.g. TCP is not supported in generic XDP mode for veth interfaces due to kernel limitations, however it is supported in native mode. This change introduces ability to use native XDP without zero-copy along with best-effort configuration option that enabled by default. In best-effort case OVS will sequentially try different modes starting from the fastest one and will choose the first acceptable for current interface. This will guarantee the best possible performance. If user will want to choose specific mode, it's still possible by setting the 'options:xdp-mode'. This change additionally changes the API by renaming the configuration knob from 'xdpmode' to 'xdp-mode' and also renaming the modes themselves to be more user-friendly. The full list of currently supported modes: * native-with-zerocopy - former DRV * native - new one, DRV without zero-copy * generic - former SKB * best-effort - new one, chooses the best available from 3 above modes Since 'best-effort' is a default mode, users will not need to explicitely set 'xdp-mode' in most cases. TCP related tests enabled back in system afxdp testsuite, because 'best-effort' will choose 'native' mode for veth interfaces and this mode has no issues with TCP. Signed-off-by: Ilya Maximets No review, I was running some performance tests for the OVS conference presentation so quickly tried this patch (assuming performance would increase :)… So with the native option (default for tap) I see a decrease, :(, in performance over the skb mode (this is with my usual ovs_perf PVP test setup). With the patch applied, and default configuration (xdp-mode-in-use=native-with-zerocopy for my tap): Hmm. tap supports zero-copy? Interesting. Have you tried forcing 'native' mode without zero-copy? Maybe it will make sense to enable/disable need-wakeup feature. "Physical to Virtual to Physical test, L3 flows[port redirect]" , Packet size Number of flows, 64 1, 711581 100, 673152 1000, 617733 Here you will even see a performance drop with multiple IP flows (this is a single queue config). This is strange.. With SKB mode (xdp-mode-in-use=generic): I'm curious, does TCP work via tap interfaces in generic mode? "Physical to Virtual to Physical test, L3 flows[port redirect]" , Packet size Number of flows, 64 1, 800796 100, 800912 1000, 800133 Cheers, Eelco ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v2] netdev-afxdp: Best-effort configuration of XDP mode.
On 7 Nov 2019, at 12:36, Ilya Maximets wrote: Until now there was only two options for XDP mode in OVS: SKB or DRV. i.e. 'generic XDP' or 'native XDP with zero-copy enabled'. Devices like 'veth' interfaces in Linux supports native XDP, but doesn't support zero-copy mode. This case can not be covered by existing API and we have to use slower generic XDP for such devices. There are few more issues, e.g. TCP is not supported in generic XDP mode for veth interfaces due to kernel limitations, however it is supported in native mode. This change introduces ability to use native XDP without zero-copy along with best-effort configuration option that enabled by default. In best-effort case OVS will sequentially try different modes starting from the fastest one and will choose the first acceptable for current interface. This will guarantee the best possible performance. If user will want to choose specific mode, it's still possible by setting the 'options:xdp-mode'. This change additionally changes the API by renaming the configuration knob from 'xdpmode' to 'xdp-mode' and also renaming the modes themselves to be more user-friendly. The full list of currently supported modes: * native-with-zerocopy - former DRV * native - new one, DRV without zero-copy * generic - former SKB * best-effort - new one, chooses the best available from 3 above modes Since 'best-effort' is a default mode, users will not need to explicitely set 'xdp-mode' in most cases. TCP related tests enabled back in system afxdp testsuite, because 'best-effort' will choose 'native' mode for veth interfaces and this mode has no issues with TCP. Signed-off-by: Ilya Maximets No review, I was running some performance tests for the OVS conference presentation so quickly tried this patch (assuming performance would increase :)… So with the native option (default for tap) I see a decrease, :(, in performance over the skb mode (this is with my usual ovs_perf PVP test setup). With the patch applied, and default configuration (xdp-mode-in-use=native-with-zerocopy for my tap): "Physical to Virtual to Physical test, L3 flows[port redirect]" , Packet size Number of flows, 64 1, 711581 100, 673152 1000, 617733 Here you will even see a performance drop with multiple IP flows (this is a single queue config). With SKB mode (xdp-mode-in-use=generic): "Physical to Virtual to Physical test, L3 flows[port redirect]" , Packet size Number of flows, 64 1, 800796 100, 800912 1000, 800133 Cheers, Eelco ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
