On Thu, Feb 13, 2020 at 03:07:33PM +0100, Ilya Maximets wrote: > On 2/13/20 2:52 PM, Yi Yang (杨燚)-云服务集团 wrote: > > Thanks Ilya, iperf3 udp should be single direction, source IP address and > > destination IP address are two VMs' IP, udp bandwidth will be 0 if they are > > wrong, but obviously UDP loss rate is 0, so it isn't the case you're > > saying, do we have way to disable MAC learning or MAC broadcast? > > NORMAL action acts like an L2 learning switch. If you don't > want to use MAC learning, remove flow with NORMAL action and > add direct forwarding flow like output:<desired port>. But > I don't think that you want to do that in OpenStack setup.
Also iperf3 establishes the control connection which uses TCP in both directions. So, in theory, the FDB should be updated. > > Is NORMAL action or MAC learning slow path process? If so, ovs-vswitchd > > daemon should have high cpu utilization. > > It's not a slow path, so there will be no cpu usage by ovs-vswitchd > userspace process. To confirm that you're flooding packets, you > may dump installed datapath flows with the following command: > > ovs-appctl dpctl/dump-flows > > In case of flood, you will see datapath flow with big number of > output ports like this: > > <...> actions:<port #1>,<port #2>,<port #3>... I'd suggest to look at the fdb: ovs-appctl fdb/show <br> and port stats to see if there is traffic moving as well. Maybe it's not your UDP test packet, but another unrelated traffic in the network. HTH, fbl > > > > > -----邮件原件----- > > 发件人: Ilya Maximets [mailto:i.maxim...@ovn.org] > > 发送时间: 2020年2月13日 21:23 > > 收件人: Flavio Leitner <f...@sysclose.org>; Yi Yang (杨燚)-云服务集团 > > <yangy...@inspur.com> > > 抄送: ovs-discuss@openvswitch.org; ovs-...@openvswitch.org; Ilya Maximets > > <i.maxim...@ovn.org> > > 主题: Re: [ovs-dev] OVS performance issue: why small udp packet pps > > performance between VMs is highly related with number of ovs ports and > > number of VMs? > > > > On 2/13/20 12:48 PM, Flavio Leitner wrote: > >> On Thu, Feb 13, 2020 at 09:18:38AM +0000, Yi Yang (杨燚)-云服务集团 wrote: > >>> Hi, all > >>> > >>> We find ovs has serious performance issue, we only launch one VM in > >>> one compute, and do iperf small udp pps performance test between > >>> these two VMs, we can see about 180000 pps (packets per second, -l > >>> 16), but > >>> > >>> 1) if we add 100 veth ports in br-int bridge, respectively, then the pps > >>> performance will be about 50000 pps. > >>> 2) If we launch one more VM in every compute node, but don’t run any > >>> workload, the pps performance will be about 90000 pps. (note, no > >>> above veth ports in this test) > >>> 3) If we launch two more VMs in every compute node (totally 3 VMs > >>> every compute nodes), but don’t run any workload , the pps > >>> performance will be about 50000 pps (note, no above veth ports in > >>> this test) > >>> > >>> Anybody can help explain why it is so? Is there any known way to > >>> optimized this? I really think ovs performance is bad (we can draw > >>> such conclusion from our test result at least), I don’t want to > >>> defame ovs ☺ > >>> > >>> BTW, we used ovs kernel datapath and vhost, we can see every port has a > >>> vhost kernel thread, it is running with 100% cpu utilization if we run > >>> iperf in VM, bu for those idle VMs, the corresponding vhost still has > >>> about 30% cpu utilization, I don’t understand why. > >>> > >>> In addition, we find udp performance is also very bad for small UDP > >>> packet for physical NIC. But it can reach 260000 pps for –l 80 which > >>> enough covers vxlan header (8 bytes) + inner eth header (14) + ipudp > >>> header (28) + 16 = 66, if we consider performance overhead ovs bridge > >>> introduces, pps performance between VMs should be able to reach 200000 > >>> pps at least, other VMs and ports shouldn’t have so big hurt against it > >>> because they are idle, no any workload there. > >> > >> What do you have in the flow table? It sounds like the traffic is > >> being broadcast to all ports. Check the FDB to see if OvS is learning > >> the mac addresses. > >> > >> It's been a while since I don't run performance tests with kernel > >> datapath, but it should be no different than Linux bridge with just > >> action NORMAL in the flow table. > >> > > > > I agree that if your performance heavily depends on the number of ports > > than you're most likely just flooding all the packets to all the ports. > > Since you're using UDP traffic, please, be sure that you're sending some > > packets in backward direction, so OVS and all other switches (if any) will > > learn/not forget to which port packets should be sent. Also, check if your > > IP addresses are correct. If for some reason it's not possible for OVS to > > learn MAC addresses correctly, avoid using action:NORMAL. > > > > Best regards, Ilya Maximets. > > > -- fbl _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
