Hi, Flavio Just let you know, your TSO support patch does need higher kernel version, it will be great if you can add document to tell users which kernel version is minimal requirement. I can confirm it can work after I used Ubuntu 18.04 and use kernel 5.3.0-40-generic.
vagrant@ubuntu1804:~$ uname -a Linux ubuntu1804 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux vagrant@ubuntu1804:~$ By the way, TPACKET_V3 also can support TSO without any new change needed, so I think TPACKET_V3 can work normally with userspace-tso-enable=true as long as your TSO patch can work normally for veth-to-veth case. I'll send out my tpacket patch v5 for review. -----邮件原件----- 发件人: Yi Yang (杨燚)-云服务集团 发送时间: 2020年2月23日 11:05 收件人: 'f...@sysclose.org' <f...@sysclose.org> 抄送: 'pkusunyif...@gmail.com' <pkusunyif...@gmail.com>; 'd...@openvswitch.org' <d...@openvswitch.org>; 'i.maxim...@ovn.org' <i.maxim...@ovn.org>; 'txfh2...@aliyun.com' <txfh2...@aliyun.com> 主题: 答复: 答复: [ovs-dev] [PATCH v4 0/3] Add support for TSO with DPDK 重要性: 高 Hi, Flavio After I ran it repeatedly in different servers, I'm very sure it can't work on Ubuntu 16.04 kernel 4.15.0-55-generic and Upstream kernel 4.15.9, so can you tell me your kernel version when you ran run-iperf3.sh I provided ? I doubt this TSO patch for veth needs higher kernel version. -----邮件原件----- 发件人: Flavio Leitner [mailto:f...@sysclose.org] 发送时间: 2020年2月20日 21:41 收件人: Yi Yang (杨燚)-云服务集团 <yangy...@inspur.com> 抄送: pkusunyif...@gmail.com; d...@openvswitch.org; i.maxim...@ovn.org; txfh2...@aliyun.com 主题: Re: 答复: [ovs-dev] [PATCH v4 0/3] Add support for TSO with DPDK On Thu, Feb 20, 2020 at 10:10:36AM +0000, Yi Yang (杨 D)-云服务集团 wrote: > Hi, Flavio > > I find this tso feature doesn't work normally on my Ubuntu 16.04, here > is my result. My kernel version is > > $ uname -a > Linux cmp008 4.15.0-55-generic #60~16.04.2-Ubuntu SMP Thu Jul 4 > 09:03:09 UTC > 2019 x86_64 x86_64 x86_64 GNU/Linux > $ I tested with 4.15.0 upstream and it worked. Can you do the same? > $ ./run-iperf3.sh > Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port > 56466 connected to 10.15.1.3 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-10.00 sec 7.05 MBytes 5.91 Mbits/sec 2212 5.66 KBytes > [ 4] 10.00-20.00 sec 7.67 MBytes 6.44 Mbits/sec 2484 5.66 KBytes > [ 4] 20.00-30.00 sec 7.77 MBytes 6.52 Mbits/sec 2500 5.66 KBytes > [ 4] 30.00-40.00 sec 7.77 MBytes 6.52 Mbits/sec 2490 5.66 KBytes > [ 4] 40.00-50.00 sec 7.76 MBytes 6.51 Mbits/sec 2500 5.66 KBytes > [ 4] 50.00-60.00 sec 7.79 MBytes 6.54 Mbits/sec 2504 5.66 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-60.00 sec 45.8 MBytes 6.40 Mbits/sec 14690 > sender > [ 4] 0.00-60.00 sec 45.7 MBytes 6.40 Mbits/sec > receiver That looks like TSO packets are being dropped and the traffic is basically TCP retransmissions of MTU size. fbl > > Server output: > Accepted connection from 10.15.1.2, port 56464 [ 5] local 10.15.1.3 > port 5201 connected to 10.15.1.2 port 56466 > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-10.00 sec 6.90 MBytes 5.79 Mbits/sec > [ 5] 10.00-20.00 sec 7.71 MBytes 6.47 Mbits/sec [ 5] > 20.00-30.00 sec 7.73 MBytes 6.48 Mbits/sec [ 5] 30.00-40.00 sec > 7.79 MBytes 6.53 Mbits/sec [ 5] 40.00-50.00 sec 7.79 MBytes 6.53 > Mbits/sec [ 5] 50.00-60.00 sec 7.79 MBytes 6.54 Mbits/sec > > > iperf Done. > $ > > But it does work for tap, I'm not sure if it is a kernel issue, which > kernel version are you using? I didn't use tpacket_v3 patch. Here is > my local ovs info. > > $ git log > commit 1223cf123ed141c0a0110ebed17572bdb2e3d0f4 > Author: Ilya Maximets <i.maxim...@ovn.org> > Date: Thu Feb 6 14:24:23 2020 +0100 > > netdev-dpdk: Don't enable offloading on HW device if not requested. > > DPDK drivers has different implementations of transmit functions. > Enabled offloading may cause driver to choose slower variant > significantly affecting performance if userspace TSO wasn't requested. > > Fixes: 29cf9c1b3b9c ("userspace: Add TCP Segmentation Offload support") > Reported-by: David Marchand <david.march...@redhat.com> > Acked-by: David Marchand <david.march...@redhat.com> > Acked-by: Flavio Leitner <f...@sysclose.org> > Acked-by: Kevin Traynor <ktray...@redhat.com> > Signed-off-by: Ilya Maximets <i.maxim...@ovn.org> > > commit 73858f9dbe83daf8cc8d4b604acc23eb62cc3f52 > Author: Flavio Leitner <f...@sysclose.org> > Date: Mon Feb 3 18:45:50 2020 -0300 > > netdev-linux: Prepend the std packet in the TSO packet > > Usually TSO packets are close to 50k, 60k bytes long, so to > to copy less bytes when receiving a packet from the kernel > change the approach. Instead of extending the MTU sized > packet received and append with remaining TSO data from > the TSO buffer, allocate a TSO packet with enough headroom > to prepend the std packet data. > > Fixes: 29cf9c1b3b9c ("userspace: Add TCP Segmentation Offload support") > Suggested-by: Ben Pfaff <b...@ovn.org> > Signed-off-by: Flavio Leitner <f...@sysclose.org> > Signed-off-by: Ben Pfaff <b...@ovn.org> > > commit 2297cbe6cc25b6b1862c499ce8f16f52f75d9e5f > Author: Flavio Leitner <f...@sysclose.org> > Date: Mon Feb 3 11:22:22 2020 -0300 > > netdev-linux-private: fix max length to be 16 bits > > The dp_packet length is limited to 16 bits, so document that > and fix the length value accordingly. > > Fixes: 29cf9c1b3b9c ("userspace: Add TCP Segmentation Offload support") > Signed-off-by: Flavio Leitner <f...@sysclose.org> > Signed-off-by: Ben Pfaff <b...@ovn.org> > > commit 3d6a6f450af5b7eaf4b532983cb14458ae792b72 > Author: David Marchand <david.march...@redhat.com> > Date: Tue Feb 4 22:28:26 2020 +0100 > > netdev-dpdk: Fix port init when lacking Tx offloads for TSO. > > The check on TSO capability did not ensure ip checksum, tcp checksum and > TSO tx offloads were available which resulted in a port init failure > (example below with a ena device): > > *2020-02-04T17:42:52.976Z|00084|dpdk|ERR|Ethdev port_id=0 requested Tx > offloads 0x2a doesn't match Tx offloads capabilities 0xe in > rte_eth_dev_configure()* > > Fixes: 29cf9c1b3b9c ("userspace: Add TCP Segmentation Offload > support") > > Reported-by: Ravi Kerur <rke...@gmail.com> > Signed-off-by: David Marchand <david.march...@redhat.com> > Acked-by: Kevin Traynor <ktray...@redhat.com> > Acked-by: Flavio Leitner <f...@sysclose.org> > Signed-off-by: Ilya Maximets <i.maxim...@ovn.org> > > commit 8e371aa497aa95e3562d53f566c2d634b4b0f589 > Author: Kirill A. Kornilov <kirill@tp> > Date: Mon Jan 13 12:29:10 2020 +0300 > > vswitchd: Add serial number configuration. > > Signed-off-by: Kirill A. Kornilov <korni...@zelax.ru> > Signed-off-by: Ben Pfaff <b...@ovn.org> > > I applied your tap patch. > > $ git diff > diff --git a/lib/netdev-linux.c b/lib/netdev-linux.c index > c6f3d27..74a5728 100644 > --- a/lib/netdev-linux.c > +++ b/lib/netdev-linux.c > @@ -1010,6 +1010,23 @@ netdev_linux_construct_tap(struct netdev *netdev_) > goto error_close; > } > > + if (userspace_tso_enabled()) { > + /* Old kernels don't support TUNSETOFFLOAD. If TUNSETOFFLOAD is > + * available, it will return EINVAL when a flag is unknown. > + * Therefore, try enabling offload with no flags to check > + * if TUNSETOFFLOAD support is available or not. */ > + if (ioctl(netdev->tap_fd, TUNSETOFFLOAD, 0) == 0 || errno != > EINVAL) { > + unsigned long oflags = TUN_F_CSUM | TUN_F_TSO4 | > + TUN_F_TSO6; > + > + if (ioctl(netdev->tap_fd, TUNSETOFFLOAD, oflags) == -1) { > + VLOG_WARN("%s: enabling tap offloading failed: %s", name, > + ovs_strerror(errno)); > + error = errno; > + goto error_close; > + } > + } > + } > + > netdev->present = true; > return 0; > > $ > > Here is performance result for tap. > > $ ./run-iperf3.sh > Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port > 56480 connected to 10.15.1.3 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-10.00 sec 19.4 GBytes 16.7 Gbits/sec 0 3.05 MBytes > [ 4] 10.00-20.00 sec 18.3 GBytes 15.7 Gbits/sec 0 3.05 MBytes > [ 4] 20.00-30.00 sec 17.6 GBytes 15.1 Gbits/sec 0 3.05 MBytes > [ 4] 30.00-40.00 sec 19.3 GBytes 16.6 Gbits/sec 0 3.05 MBytes > [ 4] 40.00-50.00 sec 18.8 GBytes 16.1 Gbits/sec 0 3.05 MBytes > [ 4] 50.00-60.00 sec 17.9 GBytes 15.4 Gbits/sec 0 3.05 MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-60.00 sec 111 GBytes 15.9 Gbits/sec 0 sender > [ 4] 0.00-60.00 sec 111 GBytes 15.9 Gbits/sec > receiver > > Server output: > Accepted connection from 10.15.1.2, port 56478 [ 5] local 10.15.1.3 > port 5201 connected to 10.15.1.2 port 56480 > [ ID] Interval Transfer Bandwidth > [ 5] 0.00-10.00 sec 19.3 GBytes 16.6 Gbits/sec > [ 5] 10.00-20.00 sec 18.3 GBytes 15.7 Gbits/sec [ 5] > 20.00-30.00 sec 17.6 GBytes 15.1 Gbits/sec [ 5] 30.00-40.00 sec > 19.3 GBytes 16.6 Gbits/sec [ 5] 40.00-50.00 sec 18.8 GBytes 16.1 > Gbits/sec [ 5] 50.00-60.00 sec 17.9 GBytes 15.4 Gbits/sec > > > iperf Done. > $ -- fbl _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
