Hi Wentian, I ran a perf test with a similar topo to your setup and got the same result. The only difference is that the iperf server is running on the host rather than in another VM. So the throughput is close to 20Gbps with Linux kernel data plane while only 1.5Gbps with VPP dataplane. I think we might have run into the same issue as https://lists.fd.io/g/vpp-dev/message/9571.
Before that, I tried TRex and Pktgen-DPDK, and the results were different. Usually the throughput would be a bit higher with VPP dataplane than Linux kernel data plane, but not much. When I was checking the CPU usage with VPP dataplane (I change the rx-mode to interrupt from polling), it is pretty low (< 10%), so it sounds to me the bottleneck is not the CPU. I also ran a similar test with vhost-user driver, following https://www.redhat.com/en/blog/hands-vhost-user-warm-welcome-dpdk, I got much better throughput (close to 20Gbps with Pktgen-DPDK), thanks to the capability to access share memory brought by vhost-user. There are much less memory copies than the above user case where vhost-net is used. I don't have a chance to run any test in the cloud environment but still believe it is an important use case for VPP to run in KVM setup. If anyone in the VPP community can shed some light on this issue, I believe both Wentian and I would appreciate it very much. Xiaodong On Sun, Oct 2, 2022 at 8:10 AM Bu Wentian <buwent...@outlook.com> wrote: > Hi Xiaodong, > > Thank you for your reply! > > I'm exactly using the VPP 22.06 installed through apt from FD.io repo. The > linux-cp and linux-nl plugins also come with the VPP from the repo. > > The virtual NICs on my VMs use the virtio(assigned by "model=virtio" when > installing with virt-install). The VMs are connected through libvirtd > networks (auto-create bridges). In my experiments, I can ping M2 from M1, > and the neighbor table and routing table in VPP seem to be correct. > > I'm not sure which driver the VPP is using (maybe vfio-pci?). The packet > counter looked like this: > vpp# show int GE1 > Name Idx State MTU (L3/IP4/IP6/MPLS) > Counter Count > GE1 1 up 9000/0/0/0 rx > packets 3719027 > rx > bytes 5630430079 > tx > packets 1107500 > tx > bytes 73176221 > drops > 76 > ip4 > 3718961 > ip6 > 61 > > tx-error 1 > vpp# show int GE2 > Name Idx State MTU (L3/IP4/IP6/MPLS) > Counter Count > GE2 2 up 9000/0/0/0 rx > packets 1107520 > rx > bytes 73177597 > tx > packets 3718998 > tx > bytes 5630427889 > drops > 63 > ip4 > 1107455 > ip6 > 62 > > tx-error 1162 > > Could you give me more information about how can I get details of the > error packets? And the main problem is that the VPP forwarding performance > is much worse than Linux kernel (2Gbps vs 26Gbps), is there any method to > improve it, or what can I do to find the reason? > > Sincerely, > Wentian > > > > > > > ------------------------------ > *发件人:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> 代表 Xiaodong Xu < > stid.s...@gmail.com> > *发送时间:* 2022年10月2日 1:05 > *收件人:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> > *主题:* Re: [vpp-dev] Throughput of VPP on KVM is Significantly Worse than > Linux Kernel > > Which vpp version are you using in your testing? As of VPP 22.06, linux-cp > and linux-nl plugins have been supported and the binary builds are > available at FD.io repository ( > https://s3-docs.fd.io/vpp/22.10/gettingstarted/installing/ubuntu.html > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs3-docs.fd.io%2Fvpp%2F22.10%2Fgettingstarted%2Finstalling%2Fubuntu.html&data=05%7C01%7C%7Cbadce86b78394ff18b8408daa3cf2dc3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638002407476279512%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=y5dLTkQx%2B3E%2Bg2VEpwMos6YhNZSD0mSlOdfGR6t77tY%3D&reserved=0>). > > > Can you install vpp from the FD.io repo and try again? (BTW, you might > want to disable the ping plugin if linux-cp is used) I would also suggest > you add static routes to rule out any issue with FRR (in which case you > don't actually need linux-cp plugin). > > In the meanwhile, I wonder what uio driver you are using for your VPP > machine (igb_uio, uio_pci_generic, or vfio-pci). Assuming you are running > virtio-net driver on the guest machine, and you are connecting the M1 and > R1, R1 and M2 with linux kernel bridge. > > If you still run into any issue, you may want to check the neighbor table > and routing table in the VPP system first, and maybe the interface counter > as well. > > Regards, > Xiaodong > > On Sat, Oct 1, 2022 at 3:55 AM Bu Wentian <buwent...@outlook.com> wrote: > > Hi everyone, > I am a beginner of VPP, and I'm trying to use VPP+FRR on KVM VMs as > routers. I have installed VPP and FRR on Ubuntu 20.04.5 VMs, and made them > running in a seperated network namespace. I use VPP Linux-cp plugin to > synchronize the route from kernel stack to VPP. The VPP and FRR seems to > work, but when I use iperf3 to test the throughput, I find the performance > of VPP is not good. > > I created a very simple topology to test the throughput: > M1 ----- R1(with VPP) ----- M2 > M1, M2 are also Ubuntu VMs(without VPP), in different subnets. I ran > iperf3 server on M1 and client on M2, but only got about 2.1Gbps > throughput, which is significantly worse than using Linux kernel as a > router(about 26.1Gbps). > > I made another experiment on the topology: > M1 ------ R1(with VPP) ---- R2(with VPP) ------ M2 > The iperf3 result is even worse (only 1.6Gbps). > > I also noticed that many retransmissions happend during the iperf3 test. > If I use Linux kernel as router rather than VPP, no retransmission will > happen. > Part of iperf3 output: > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 166 MBytes 1.39 Gbits/sec 23 344 KBytes > [ 5] 1.00-2.00 sec 179 MBytes 1.50 Gbits/sec 49 328 KBytes > [ 5] 2.00-3.00 sec 203 MBytes 1.70 Gbits/sec 47 352 KBytes > [ 5] 3.00-4.00 sec 203 MBytes 1.70 Gbits/sec 54 339 KBytes > [ 5] 4.00-5.00 sec 211 MBytes 1.77 Gbits/sec 59 325 KBytes > > > Another phenomenon I found is that when I ran iperf3 directly on the R1 > and R2, I got 0 throughput at all. The output of iperf3 is like this: > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 324 KBytes 2.65 Mbits/sec 4 8.74 KBytes > [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes > [ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes > [ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes > [ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes > [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes > [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes > [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes > [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes > [ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 324 KBytes 266 Kbits/sec 7 > sender > [ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec > receiver > > > All my VMs use 4vcpus and 8G RAM. The host machine has 16Cores(32 Threads) > and 32GB RAM. > The VMs are connected by libvirtd networks. > I installed the VPP +FRR following this tutorial: > https://ipng.ch/s/articles/2021/12/23/vpp-playground.html > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fipng.ch%2Fs%2Farticles%2F2021%2F12%2F23%2Fvpp-playground.html&data=05%7C01%7C%7Cbadce86b78394ff18b8408daa3cf2dc3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638002407476279512%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XHwcWCBs%2F1Fk6mjLlul6S5d390cru%2BzjoTXFclbmg5g%3D&reserved=0> > The VPP startup.conf is in the attachment. > > I want to know why the VPP throughput is worse than Linux kernel, and what > can I do to improve it (I hope it better than Linux kernel forwarding). I > have searched on google for the solution but got nothing helpful. It will > be appreciated if anyone could give me a help. Please contact me if more > information or logs are needed. > > > Sincerely, > Wentian Bu > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21958): https://lists.fd.io/g/vpp-dev/message/21958 Mute This Topic: https://lists.fd.io/mt/94080585/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
