[ovs-dev] 答复: 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath
Ok, I will send out v7 with these changes. -邮件原件- 发件人: William Tu [mailto:u9012...@gmail.com] 发送时间: 2020年3月18日 11:56 收件人: Yi Yang (杨燚)-云服务集团 抄送: i.maxim...@ovn.org; b...@ovn.org; yang_y...@163.com; ovs-dev@openvswitch.org 主题: Re: [ovs-dev] 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath On Tue, Mar 17, 2020 at 7:00 PM Yi Yang (杨燚)-云服务集团 wrote: > > By the way, with tpacket_v3, zero copy optimization and is_pmd=true, the > performance is much better, 3.77Gbps, (3.77-1.34)/1.34 = 1.81 , i.e. 181% > improvement, here is the performance data. > Can you send out the tpacket_v3 patch together with these optimizations to the mailing list? Thanks William ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] 答复: 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath
By the way, with tpacket_v3, zero copy optimization and is_pmd=true, the performance is much better, 3.77Gbps, (3.77-1.34)/1.34 = 1.81 , i.e. 181% improvement, here is the performance data. is_pmd = true = eipadmin@eip01:~$ sudo ./run-iperf3.sh Connecting to host 10.15.1.3, port 5201 [ 4] local 10.15.1.2 port 43210 connected to 10.15.1.3 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 4.34 GBytes 3.73 Gbits/sec0 3.03 MBytes [ 4] 10.00-20.00 sec 4.40 GBytes 3.78 Gbits/sec0 3.03 MBytes [ 4] 20.00-30.00 sec 4.40 GBytes 3.78 Gbits/sec0 3.03 MBytes [ 4] 30.00-40.00 sec 4.40 GBytes 3.78 Gbits/sec0 3.03 MBytes [ 4] 40.00-50.00 sec 4.40 GBytes 3.78 Gbits/sec0 3.03 MBytes [ 4] 50.00-60.00 sec 4.40 GBytes 3.78 Gbits/sec0 3.03 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 26.3 GBytes 3.77 Gbits/sec0 sender [ 4] 0.00-60.00 sec 26.3 GBytes 3.77 Gbits/sec receiver Server output: Accepted connection from 10.15.1.2, port 43208 [ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43210 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 4.32 GBytes 3.71 Gbits/sec [ 5] 10.00-20.00 sec 4.40 GBytes 3.78 Gbits/sec [ 5] 20.00-30.00 sec 4.40 GBytes 3.78 Gbits/sec [ 5] 30.00-40.00 sec 4.40 GBytes 3.78 Gbits/sec [ 5] 40.00-50.00 sec 4.40 GBytes 3.78 Gbits/sec [ 5] 50.00-60.00 sec 4.40 GBytes 3.78 Gbits/sec iperf Done. eipadmin@eip01:~$ -邮件原件- 发件人: William Tu [mailto:u9012...@gmail.com] 发送时间: 2020年3月17日 22:58 收件人: Yi Yang (杨燚)-云服务集团 抄送: i.maxim...@ovn.org; b...@ovn.org; yang_y...@163.com; ovs-dev@openvswitch.org 主题: Re: [ovs-dev] 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath On Tue, Mar 17, 2020 at 2:08 AM Yi Yang (杨燚)-云服务集团 wrote: > > Hi, William > > Finally, my highend server is available and so I can do performance > comparison again, tpacket_v3 obviously has big performance improvement, here > is my data. By the way, in order to get stable performance data, please use > taskset to pin ovs-vswitchd to a physical core (you shouldn't schedule other > task to its logical sibling core for stable performance data), iperf3 client > an iperf3 use different cores, for my case, ovs-vswitchd is pinned to core 1, > iperf3 server is pinned to core 4, iperf3 client is pinned to core 5. > > According to my test, tpacket_v3 can get about 55% improvement (from 1.34 to > 2.08, (2.08-1.34)/1.34 = 0.55) , with my further optimization (use zero copy > for receive side), it can have more improvement (from 1.34 to 2.21, > (2.21-1.34)/1.34 = 0.65), so I still think performance improvement is big, > please reconsider it again. > That's great improvement. What is your optimization "zero copy for receive side"? Does it include in the patch? Regards William ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] 答复: 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath
William, are you trying my patch for zero copy? I can send you for a try on your platform. Per your af_xdp change, I find dp_packet can use pre-allocated buffer, so I used that way, because tpacket_v3 has setup rx ring there, so dp_packet can directly use those rx ring buffer. -邮件原件- 发件人: William Tu [mailto:u9012...@gmail.com] 发送时间: 2020年3月17日 22:58 收件人: Yi Yang (杨燚)-云服务集团 抄送: i.maxim...@ovn.org; b...@ovn.org; yang_y...@163.com; ovs-dev@openvswitch.org 主题: Re: [ovs-dev] 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath On Tue, Mar 17, 2020 at 2:08 AM Yi Yang (杨燚)-云服务集团 wrote: > > Hi, William > > Finally, my highend server is available and so I can do performance > comparison again, tpacket_v3 obviously has big performance improvement, here > is my data. By the way, in order to get stable performance data, please use > taskset to pin ovs-vswitchd to a physical core (you shouldn't schedule other > task to its logical sibling core for stable performance data), iperf3 client > an iperf3 use different cores, for my case, ovs-vswitchd is pinned to core 1, > iperf3 server is pinned to core 4, iperf3 client is pinned to core 5. > > According to my test, tpacket_v3 can get about 55% improvement (from 1.34 to > 2.08, (2.08-1.34)/1.34 = 0.55) , with my further optimization (use zero copy > for receive side), it can have more improvement (from 1.34 to 2.21, > (2.21-1.34)/1.34 = 0.65), so I still think performance improvement is big, > please reconsider it again. > That's great improvement. What is your optimization "zero copy for receive side"? Does it include in the patch? Regards William ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] 答复: 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath
[ 4] 0.00-60.00 sec 15.4 GBytes 2.21 Gbits/sec0 sender [ 4] 0.00-60.00 sec 15.4 GBytes 2.21 Gbits/sec receiver Server output: Accepted connection from 10.15.1.2, port 43180 [ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43182 [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 2.53 GBytes 2.17 Gbits/sec [ 5] 10.00-20.00 sec 2.58 GBytes 2.22 Gbits/sec [ 5] 20.00-30.00 sec 2.58 GBytes 2.22 Gbits/sec [ 5] 30.00-40.00 sec 2.59 GBytes 2.22 Gbits/sec [ 5] 40.00-50.00 sec 2.57 GBytes 2.21 Gbits/sec [ 5] 50.00-60.00 sec 2.57 GBytes 2.21 Gbits/sec iperf Done. eipadmin@eip01:~$ -邮件原件- 发件人: William Tu [mailto:u9012...@gmail.com] 发送时间: 2020年3月14日 22:18 收件人: Yi Yang (杨燚)-云服务集团 抄送: i.maxim...@ovn.org; b...@ovn.org; yang_y...@163.com; ovs-dev@openvswitch.org 主题: Re: [ovs-dev] 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath On Fri, Mar 13, 2020 at 9:45 PM Yi Yang (杨燚)-云服务集团 wrote: > > Io_uring is a feature brought in by Linux kernel 5.1, so it can't be > used on Linux system with kernel version < 5.1. tpacket_v3 is only one > way to avoid system call on almost all the Linux kernel versions, it > is unique from this perspective. Maybe you will miss it if someone > fixes kernel side issue :-) > > In addition, according to what Flavio said, TSO can't support VXLAN > currently, but in most cloud scenarios, VXLAN is only one choice, so for such > cases, TSO can be ignored. > > My point is we can provide one option for such use cases, once kernel side > issue is fixed, all the Linux distributions can apply this fix, users can get > immediate benefits without change. So maybe adding a switch > userspace-use-tpacket-v3 in other-config (set to False by default) is an > acceptable way to handle this. > The tpacket_v3 patch now shows very little performance improvement. So there is little incentive to merge and maintain this code. Do you know if kernel side is fixed, will tpacket_v3 have better performance improvement? Or another way is to study io_uring and compare its performance with tpacket_v3. William ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] 答复: 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath
There might still be a misunderstanding. There can be a difference between the kernel that OVS runs on (version A) and the kernel headers against which it is built (version B). Often, the latter are supplied by the distribution and they are not usually kept as up to date, so B < A is common. I don't know whether this is likely to be a problem in this particular case. On Sat, Mar 14, 2020 at 03:35:46AM +, Yi Yang (杨燚)-云服务集团 wrote: > Got it, then we can safely remove inclue/linux/if_packet.h in ovs > because the minimal Linux version OVS supports has supported > tpacket_v3. Thanks Ben for clarification. > > -邮件原件- > 发件人: Ben Pfaff [mailto:b...@ovn.org] > 发送时间: 2020年3月13日 23:57 > 收件人: Yi Yang (杨燚)-云服务集团 > 抄送: u9012...@gmail.com; yang_y...@163.com; ovs-dev@openvswitch.org > 主题: Re: 答复: [ovs-dev] [PATCH v6] Use TPACKET_V3 to accelerate veth for > userspace datapath > > On Fri, Mar 13, 2020 at 01:04:07AM +, Yi Yang (杨燚)-云服务集团 wrote: > > Per my understanding, Ben meant a build system (which isn't Linux > > probably, it doesn't have include/linux/if_packet.h) should be able to > > build tpacket_v3 code in order that built-out binary can work on Linux > > system with tpacket_v3 feature, this is Ben's point, that is why he > > wanted me to add include/linux/if_packet.h in ovs repo. > > > > Ben, can you help double confirm if include/linux/if_packet.h in ovs > > is necessary? > > I think my meaning was misunderstood. Linux always has if_packet.h. > Only recent enough Linux has TPACKET_V3 in if_packet.h. If the system is > Linux but the TPACKET_V3 types and constants are not defined in if_packet.h, > then the build system should define them. ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] 答复: 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath
Io_uring is a feature brought in by Linux kernel 5.1, so it can't be used on Linux system with kernel version < 5.1. tpacket_v3 is only one way to avoid system call on almost all the Linux kernel versions, it is unique from this perspective. Maybe you will miss it if someone fixes kernel side issue :-) In addition, according to what Flavio said, TSO can't support VXLAN currently, but in most cloud scenarios, VXLAN is only one choice, so for such cases, TSO can be ignored. My point is we can provide one option for such use cases, once kernel side issue is fixed, all the Linux distributions can apply this fix, users can get immediate benefits without change. So maybe adding a switch userspace-use-tpacket-v3 in other-config (set to False by default) is an acceptable way to handle this. -邮件原件- 发件人: dev [mailto:ovs-dev-boun...@openvswitch.org] 代表 Ilya Maximets 发送时间: 2020年3月14日 0:48 收件人: William Tu ; Ben Pfaff 抄送: yang_y...@163.com; ovs-dev@openvswitch.org; i.maxim...@ovn.org 主题: Re: [ovs-dev] 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath On 3/13/20 5:22 PM, William Tu wrote: > On Fri, Mar 13, 2020 at 8:57 AM Ben Pfaff wrote: >> >> On Fri, Mar 13, 2020 at 01:04:07AM +, Yi Yang (杨燚)-云服务集团 wrote: >>> Per my understanding, Ben meant a build system (which isn't Linux >>> probably, it doesn't have include/linux/if_packet.h) should be able >>> to build tpacket_v3 code in order that built-out binary can work on >>> Linux system with tpacket_v3 feature, this is Ben's point, that is >>> why he wanted me to add include/linux/if_packet.h in ovs repo. >>> >>> Ben, can you help double confirm if include/linux/if_packet.h in ovs >>> is necessary? >> >> I think my meaning was misunderstood. Linux always has if_packet.h. >> Only recent enough Linux has TPACKET_V3 in if_packet.h. If the >> system is Linux but the TPACKET_V3 types and constants are not >> defined in if_packet.h, then the build system should define them. > > Thanks! > > My suggestion is that if the system is Linux but the TPACKET_V3 types > and constants are not defined in if_packet.h, then just skip using > TPACKET_V3 and > use the current recvmmsg approach. Because when we start TPACKET_V3 > patch, the af_packet on veth performance is about 200Mbps, so > tpacket_v3 has huge performance benefits. > > With YiYang's patch > "Use batch process recv for tap and raw socket in netdev datapath" > the af_packet on veth improves to 1.47Gbps. And tpacket_v3 shows > similar or 7% better performance. So there isn't a huge benefits now. With such a small performance benefit does it make sense to have these 700 lines of code that is so hard to read and maintain? Another point is that hopefully segmentation offloading in userspace datapath will evolve so we could enable it by default and all this code will become almost useless. If you're looking for poll mode/async -like solutions we could try and check io_uring way for calling same recvmsg/sendmsg. That might have more benefits and it will support all the functionality supported by these calls. Even better, we could also make io_uring support as an internal library and reuse it for other OVS subsystems like making async poll/timers/logging/etc in the future. Best regards, Ilya Maximets. ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
[ovs-dev] 答复: 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath
Got it, then we can safely remove inclue/linux/if_packet.h in ovs because the minimal Linux version OVS supports has supported tpacket_v3. Thanks Ben for clarification. -邮件原件- 发件人: Ben Pfaff [mailto:b...@ovn.org] 发送时间: 2020年3月13日 23:57 收件人: Yi Yang (杨燚)-云服务集团 抄送: u9012...@gmail.com; yang_y...@163.com; ovs-dev@openvswitch.org 主题: Re: 答复: [ovs-dev] [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath On Fri, Mar 13, 2020 at 01:04:07AM +, Yi Yang (杨燚)-云服务集团 wrote: > Per my understanding, Ben meant a build system (which isn't Linux > probably, it doesn't have include/linux/if_packet.h) should be able to > build tpacket_v3 code in order that built-out binary can work on Linux > system with tpacket_v3 feature, this is Ben's point, that is why he > wanted me to add include/linux/if_packet.h in ovs repo. > > Ben, can you help double confirm if include/linux/if_packet.h in ovs > is necessary? I think my meaning was misunderstood. Linux always has if_packet.h. Only recent enough Linux has TPACKET_V3 in if_packet.h. If the system is Linux but the TPACKET_V3 types and constants are not defined in if_packet.h, then the build system should define them. ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
