Hi, William As you have known, I sent out a tpacket_v3 kernel side patch to net-next, tpacket maintainer worries it can impact on other use cases, so he hopes we can use TPACKET_V2 for TSO, I tried TPACKET_V2 for TSO, it is indeed a good choice in case that tpacket_v3 in kernel side isn't ready for this.
I have sent out v8 http://patchwork.ozlabs.org/project/openvswitch/patch/[email protected]/ for this. Per my test, its performance is much better than recvmmsg/sendmmsg in case of TSO, here is my test results for your reference. [yangyi@localhost ovs-master]$ uname -a Linux localhost.localdomain 5.5.9+ #40 SMP Mon Mar 30 05:54:05 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux recvmmsg & sendmmsg TSO ======================= [yangyi@localhost ovs-master]$ sudo ../run-iperf3.sh Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port 43354 connected to 10.15.1.3 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 5.64 GBytes 4.84 Gbits/sec 48224 198 KBytes [ 4] 10.00-20.00 sec 5.59 GBytes 4.80 Gbits/sec 46100 182 KBytes [ 4] 20.00-30.00 sec 5.68 GBytes 4.88 Gbits/sec 48959 226 KBytes [ 4] 30.00-40.00 sec 5.58 GBytes 4.80 Gbits/sec 49035 161 KBytes [ 4] 40.00-50.00 sec 5.54 GBytes 4.76 Gbits/sec 49306 256 KBytes [ 4] 50.00-60.00 sec 5.58 GBytes 4.80 Gbits/sec 48558 197 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 33.6 GBytes 4.81 Gbits/sec 290182 sender [ 4] 0.00-60.00 sec 33.6 GBytes 4.81 Gbits/sec receiver Server output: Accepted connection from 10.15.1.2, port 43352 [ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43354 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 5.64 GBytes 4.84 Gbits/sec [ 5] 10.00-20.00 sec 5.59 GBytes 4.80 Gbits/sec [ 5] 20.00-30.00 sec 5.68 GBytes 4.88 Gbits/sec [ 5] 30.00-40.00 sec 5.58 GBytes 4.80 Gbits/sec [ 5] 40.00-50.00 sec 5.54 GBytes 4.76 Gbits/sec [ 5] 50.00-60.00 sec 5.58 GBytes 4.80 Gbits/sec [ 5] 60.00-60.00 sec 255 KBytes 4.15 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-60.00 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-60.00 sec 33.6 GBytes 4.81 Gbits/sec receiver iperf Done. [yangyi@localhost ovs-master]$ TPACKET_V2 TSO ============== [yangyi@localhost ovs-master]$ sudo ../run-iperf3.sh Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port 41600 connected to 10.15.1.3 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 10.6 GBytes 9.11 Gbits/sec 23 3.00 MBytes [ 4] 10.00-20.00 sec 10.6 GBytes 9.13 Gbits/sec 0 3.00 MBytes [ 4] 20.00-30.00 sec 10.7 GBytes 9.15 Gbits/sec 0 3.00 MBytes [ 4] 30.00-40.00 sec 10.6 GBytes 9.13 Gbits/sec 32 3.00 MBytes [ 4] 40.00-50.00 sec 10.7 GBytes 9.17 Gbits/sec 0 3.00 MBytes [ 4] 50.00-60.00 sec 10.5 GBytes 9.06 Gbits/sec 0 3.00 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 63.7 GBytes 9.12 Gbits/sec 55 sender [ 4] 0.00-60.00 sec 63.7 GBytes 9.12 Gbits/sec receiver Server output: Accepted connection from 10.15.1.2, port 41598 [ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 41600 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 10.6 GBytes 9.11 Gbits/sec [ 5] 10.00-20.00 sec 10.6 GBytes 9.13 Gbits/sec [ 5] 20.00-30.00 sec 10.7 GBytes 9.15 Gbits/sec [ 5] 30.00-40.00 sec 10.6 GBytes 9.13 Gbits/sec [ 5] 40.00-50.00 sec 10.7 GBytes 9.17 Gbits/sec [ 5] 50.00-60.00 sec 10.5 GBytes 9.06 Gbits/sec [ 5] 60.00-60.00 sec 2.72 MBytes 15.8 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-60.00 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-60.00 sec 63.7 GBytes 9.12 Gbits/sec receiver iperf Done. [yangyi@localhost ovs-master]$ I also tested it by using Ubuntu 4.15 kernel, it have better performance. [yangyi@localhost ovs-master]$ uname -a Linux localhost.localdomain 4.15.18 #10 SMP Wed Mar 25 06:02:27 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux [yangyi@localhost ovs-master]$ recvmmsg & sendmmsg TSO ======================= [yangyi@localhost ovs-master]$ sudo ../run-iperf3.sh Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port 59550 connected to 10.15.1.3 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 6.28 GBytes 5.39 Gbits/sec 56673 171 KBytes [ 4] 10.00-20.00 sec 6.41 GBytes 5.50 Gbits/sec 56704 184 KBytes [ 4] 20.00-30.00 sec 6.64 GBytes 5.71 Gbits/sec 55720 189 KBytes [ 4] 30.00-40.00 sec 6.52 GBytes 5.60 Gbits/sec 53433 178 KBytes [ 4] 40.00-50.00 sec 6.41 GBytes 5.51 Gbits/sec 52541 185 KBytes [ 4] 50.00-60.00 sec 6.52 GBytes 5.60 Gbits/sec 56081 141 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 38.8 GBytes 5.55 Gbits/sec 331152 sender [ 4] 0.00-60.00 sec 38.8 GBytes 5.55 Gbits/sec receiver Server output: Accepted connection from 10.15.1.2, port 59548 [ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 59550 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 6.25 GBytes 5.37 Gbits/sec [ 5] 10.00-20.00 sec 6.41 GBytes 5.51 Gbits/sec [ 5] 20.00-30.00 sec 6.64 GBytes 5.71 Gbits/sec [ 5] 30.00-40.00 sec 6.52 GBytes 5.60 Gbits/sec [ 5] 40.00-50.00 sec 6.41 GBytes 5.51 Gbits/sec [ 5] 50.00-60.00 sec 6.51 GBytes 5.60 Gbits/sec [ 5] 60.00-60.04 sec 22.5 MBytes 4.71 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-60.04 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-60.04 sec 38.8 GBytes 5.55 Gbits/sec receiver iperf Done. [yangyi@localhost ovs-master]$ TPACKET_V2 TSO =================== [yangyi@localhost ovs-master]$ sudo ../run-iperf3.sh Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port 32884 connected to 10.15.1.3 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 10.7 GBytes 9.21 Gbits/sec 7 3.13 MBytes [ 4] 10.00-20.00 sec 10.8 GBytes 9.25 Gbits/sec 0 3.13 MBytes [ 4] 20.00-30.00 sec 10.8 GBytes 9.25 Gbits/sec 0 3.13 MBytes [ 4] 30.00-40.00 sec 10.8 GBytes 9.29 Gbits/sec 0 3.13 MBytes [ 4] 40.00-50.00 sec 10.8 GBytes 9.30 Gbits/sec 0 3.13 MBytes [ 4] 50.00-60.00 sec 10.7 GBytes 9.20 Gbits/sec 0 3.13 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 64.6 GBytes 9.25 Gbits/sec 7 sender [ 4] 0.00-60.00 sec 64.6 GBytes 9.25 Gbits/sec receiver Server output: Accepted connection from 10.15.1.2, port 32882 [ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 32884 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 10.7 GBytes 9.17 Gbits/sec [ 5] 10.00-20.00 sec 10.8 GBytes 9.25 Gbits/sec [ 5] 20.00-30.00 sec 10.8 GBytes 9.25 Gbits/sec [ 5] 30.00-40.00 sec 10.8 GBytes 9.29 Gbits/sec [ 5] 40.00-50.00 sec 10.8 GBytes 9.30 Gbits/sec [ 5] 50.00-60.00 sec 10.7 GBytes 9.20 Gbits/sec [ 5] 60.00-60.04 sec 46.0 MBytes 9.29 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-60.04 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-60.04 sec 64.6 GBytes 9.24 Gbits/sec receiver iperf Done. [yangyi@localhost ovs-master]$ -----邮件原件----- 发件人: William Tu [mailto:[email protected]] 发送时间: 2020年3月19日 22:53 收件人: Yi Yang (杨燚)-云服务集团 <[email protected]> 抄送: [email protected]; [email protected]; [email protected] 主题: Re: [ovs-dev] 答复: [PATCH v7] Use TPACKET_V3 to accelerate veth for userspace datapath On Wed, Mar 18, 2020 at 8:12 PM Yi Yang (杨燚)-云服务集团 <[email protected]> wrote: > > Hi, folks > > As I said, TPACKET_V3 does have kernel implementation issue, I tried to fix > it in Linux kernel 5.5.9, here is my test data with tpacket_v3 and tso > enabled. On my low end server, my goal is to reach 16Gbps at least, I still > have another idea to improve it. > Can you share your kernel fix? Or have you sent patch somewhere? William _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
