Hi, William Finally, my highend server is available and so I can do performance comparison again, tpacket_v3 obviously has big performance improvement, here is my data. By the way, in order to get stable performance data, please use taskset to pin ovs-vswitchd to a physical core (you shouldn't schedule other task to its logical sibling core for stable performance data), iperf3 client an iperf3 use different cores, for my case, ovs-vswitchd is pinned to core 1, iperf3 server is pinned to core 4, iperf3 client is pinned to core 5.
According to my test, tpacket_v3 can get about 55% improvement (from 1.34 to 2.08, (2.08-1.34)/1.34 = 0.55) , with my further optimization (use zero copy for receive side), it can have more improvement (from 1.34 to 2.21, (2.21-1.34)/1.34 = 0.65), so I still think performance improvement is big, please reconsider it again. William, I can help you do performance check on your servers if you'd like, from these data and previous data, we can draw a conclusion performance data is very platform sensitive. You can schedule a meeting for further discussion if needed. No zero copy and no tpacket_v3 (recvmmsg, sendmmsg) =================================================== eipadmin@eip01:~$ sudo ./run-iperf3.sh Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port 43194 connected to 10.15.1.3 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 1.58 GBytes 1.35 Gbits/sec 13851 103 KBytes [ 4] 10.00-20.00 sec 1.56 GBytes 1.34 Gbits/sec 14018 94.7 KBytes [ 4] 20.00-30.00 sec 1.56 GBytes 1.34 Gbits/sec 13942 94.7 KBytes [ 4] 30.00-40.00 sec 1.56 GBytes 1.34 Gbits/sec 13565 106 KBytes [ 4] 40.00-50.00 sec 1.54 GBytes 1.32 Gbits/sec 14567 106 KBytes [ 4] 50.00-60.00 sec 1.56 GBytes 1.34 Gbits/sec 13738 84.8 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 9.35 GBytes 1.34 Gbits/sec 83681 sender [ 4] 0.00-60.00 sec 9.35 GBytes 1.34 Gbits/sec receiver Server output: Accepted connection from 10.15.1.2, port 43192 [ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43194 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 1.57 GBytes 1.35 Gbits/sec [ 5] 10.00-20.00 sec 1.56 GBytes 1.34 Gbits/sec [ 5] 20.00-30.00 sec 1.56 GBytes 1.34 Gbits/sec [ 5] 30.00-40.00 sec 1.56 GBytes 1.34 Gbits/sec [ 5] 40.00-50.00 sec 1.54 GBytes 1.32 Gbits/sec [ 5] 50.00-60.00 sec 1.56 GBytes 1.34 Gbits/sec iperf Done. eipadmin@eip01:~$ No zero copy but with tpacket_v3 ================================ eipadmin@eip01:~$ sudo ./run-iperf3.sh Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port 43174 connected to 10.15.1.3 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 2.36 GBytes 2.02 Gbits/sec 0 3.04 MBytes [ 4] 10.00-20.00 sec 2.43 GBytes 2.09 Gbits/sec 0 3.04 MBytes [ 4] 20.00-30.00 sec 2.44 GBytes 2.09 Gbits/sec 0 3.04 MBytes [ 4] 30.00-40.00 sec 2.43 GBytes 2.09 Gbits/sec 0 3.04 MBytes [ 4] 40.00-50.00 sec 2.43 GBytes 2.09 Gbits/sec 0 3.04 MBytes [ 4] 50.00-60.00 sec 2.44 GBytes 2.10 Gbits/sec 0 3.04 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 14.5 GBytes 2.08 Gbits/sec 0 sender [ 4] 0.00-60.00 sec 14.5 GBytes 2.08 Gbits/sec receiver Server output: Accepted connection from 10.15.1.2, port 43172 [ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43174 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 2.35 GBytes 2.02 Gbits/sec [ 5] 10.00-20.00 sec 2.43 GBytes 2.09 Gbits/sec [ 5] 20.00-30.00 sec 2.44 GBytes 2.09 Gbits/sec [ 5] 30.00-40.00 sec 2.43 GBytes 2.09 Gbits/sec [ 5] 40.00-50.00 sec 2.43 GBytes 2.09 Gbits/sec [ 5] 50.00-60.00 sec 2.44 GBytes 2.10 Gbits/sec iperf Done. eipadmin@eip01:~$ Have zero copy patch and tpacket_v3 =================================== eipadmin@eip01:~$ sudo ./run-iperf3.sh Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port 43182 connected to 10.15.1.3 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 2.54 GBytes 2.18 Gbits/sec 0 3.03 MBytes [ 4] 10.00-20.00 sec 2.58 GBytes 2.22 Gbits/sec 0 3.03 MBytes [ 4] 20.00-30.00 sec 2.58 GBytes 2.22 Gbits/sec 0 3.03 MBytes [ 4] 30.00-40.00 sec 2.59 GBytes 2.22 Gbits/sec 0 3.03 MBytes [ 4] 40.00-50.00 sec 2.57 GBytes 2.21 Gbits/sec 0 3.03 MBytes [ 4] 50.00-60.00 sec 2.57 GBytes 2.21 Gbits/sec 0 3.03 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 15.4 GBytes 2.21 Gbits/sec 0 sender [ 4] 0.00-60.00 sec 15.4 GBytes 2.21 Gbits/sec receiver Server output: Accepted connection from 10.15.1.2, port 43180 [ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43182 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 2.53 GBytes 2.17 Gbits/sec [ 5] 10.00-20.00 sec 2.58 GBytes 2.22 Gbits/sec [ 5] 20.00-30.00 sec 2.58 GBytes 2.22 Gbits/sec [ 5] 30.00-40.00 sec 2.59 GBytes 2.22 Gbits/sec [ 5] 40.00-50.00 sec 2.57 GBytes 2.21 Gbits/sec [ 5] 50.00-60.00 sec 2.57 GBytes 2.21 Gbits/sec iperf Done. eipadmin@eip01:~$ -----邮件原件----- 发件人: William Tu [mailto:[email protected]] 发送时间: 2020年3月14日 22:18 收件人: Yi Yang (杨燚)-云服务集团 <[email protected]> 抄送: [email protected]; [email protected]; [email protected]; [email protected] 主题: Re: [ovs-dev] 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath On Fri, Mar 13, 2020 at 9:45 PM Yi Yang (杨燚)-云服务集团 <[email protected]> wrote: > > Io_uring is a feature brought in by Linux kernel 5.1, so it can't be > used on Linux system with kernel version < 5.1. tpacket_v3 is only one > way to avoid system call on almost all the Linux kernel versions, it > is unique from this perspective. Maybe you will miss it if someone > fixes kernel side issue :-) > > In addition, according to what Flavio said, TSO can't support VXLAN > currently, but in most cloud scenarios, VXLAN is only one choice, so for such > cases, TSO can be ignored. > > My point is we can provide one option for such use cases, once kernel side > issue is fixed, all the Linux distributions can apply this fix, users can get > immediate benefits without change. So maybe adding a switch > userspace-use-tpacket-v3 in other-config (set to False by default) is an > acceptable way to handle this. > The tpacket_v3 patch now shows very little performance improvement. So there is little incentive to merge and maintain this code. Do you know if kernel side is fixed, will tpacket_v3 have better performance improvement? Or another way is to study io_uring and compare its performance with tpacket_v3. William _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
