I tried the native virtio driver with gso enabled, the throughput (without any tuning) is close to 16Gbps. It's a big improvement, though there is still some gap between vpp and Linux kernel. Does it suggest that the virtio driver in dpdk has some issues?
Regards, Xiaodong On Wed, Oct 12, 2022 at 3:40 AM Mohsin Kazmi via lists.fd.io <sykazmi= cisco....@lists.fd.io> wrote: > Hi, > > > > You can use VPP native virtio driver in VM. > > > > ./dpdk-devbind.py -b vfio-pci 00:03.0 00:04.0 > > cd /home/vpp > > > > set loggin class pci level debug > > set loggin class virtio level debug > > create int virtio 0000:00:03.0 gso-enabled > > create int virtio 0000:00:04.0 gso-enabled > > > > https://s3-docs.fd.io/vpp/22.10/cli-reference/clis/clicmd_src_vnet_gso.html > is the command to use software segmentation of packets when interface > doesn’t support offload. While you need to enable GSO on the interfaces > instead of chunking in the software. > > > > Please pin the VM threads and Iperf threads to get the performance results. > > > > -Best Regards, > > Mohsin Kazmi > > > > *From: *vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> on behalf of Xiaodong > Xu <stid.s...@gmail.com> > *Date: *Thursday, October 6, 2022 at 10:24 PM > *To: *vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> > *Subject: *Re: [vpp-dev] Throughput of VPP on KVM is Significantly Worse > than Linux Kernel > > Hi Wentian, > > > > Please take a look at > https://s3-docs.fd.io/vpp/22.10/cli-reference/clis/clicmd_src_vnet_gso.html > for the GSO feature in VPP. > > I tried to turn on the TSO option too, however VPP doesn't seem to be able > to forward traffic once TSO is turned on. > > > > The test case for vhost-user driver I had run is similar to the topo shown > at > https://wiki.fd.io/view/VPP/Use_VPP_to_Chain_VMs_Using_Vhost-User_Interface#Two_Chained_QEMU_Instances_with_VPP_Vhost-User_Interfaces. > The difference is that I don't have the 'VPP l2 xc1' / 'VPP l2 xc3' > instances in the diagram, also the 'testpmd' application is replaced with > VPP in my testing. You do need a VPP instance running on the host machine > though, in order to bridge the two VMs together. > > > > Xiaodong > > On Thu, Oct 6, 2022 at 7:40 AM Bu Wentian <buwent...@outlook.com> wrote: > > Hi Xiaodong, > > > > Could you please tell me how to enable GSO in VPP? I read the startup.conf > and searched the VPP documents but didn't find options about GSO. I found a > TSO option in the default startup.conf. According to the comment, the "tso > on" option must be enabled with "enable-tcp-udp-checksum" set. However, I > tried to do so, but the VPP failed to start after enabling the two options. > > > > I also read the document you mentioned ( > https://www.redhat.com/en/blog/hands-vhost-user-warm-welcome-dpdk), and I > found a similar example in VPP document: > https://s3-docs.fd.io/vpp/22.06/usecases/vhost/index.html . But the two > examples both seem to using VPP as a bridge in the host, and run test in > VMs. In my application scene, I need to run VPP in the guest rather than > host. Can vhost-user work in this scene? > > > > > > Sincerely, > > Wentian > > > > > ------------------------------ > > *发**件人:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> 代表 Xiaodong Xu < > stid.s...@gmail.com> > *发**送**时间:* 2022年10月6日 0:40 > *收件人:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> > *主**题:* Re: [vpp-dev] Throughput of VPP on KVM is Significantly Worse > than Linux Kernel > > > > I actually tried to enable GSO on both input and output interfaces in VPP, > and got a little bit of difference for the result. In most cases the > throughput would be the same, but in some cases I see a burst of 4.5Gbps > (3x of the original rate). > > > > As Benoit said, even without GSO the throughput is way too low for VPP. I > also see that KVM has the options 'gso', 'tso4' for either host or guest ( > https://libvirt.org/formatdomain.html#setting-nic-driver-specific-options > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flibvirt.org%2Fformatdomain.html%23setting-nic-driver-specific-options&data=05%7C01%7C%7Cf6c220e7703a4c204c8808daa6f15b71%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638005852829199513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VkaJIs2LlwhiSgaZpFUu3JKc2wzvqaxESQX%2B0Va5%2Bu4%3D&reserved=0>) > but they are all turned on by default. > > > > In the meanwhile, I wonder if there is a GRO option (or maybe jumbo > frame?) for VPP? As I suppose the GSO option applies to sending rather than > receiving. > > > > Regards, > > Xiaodong > > > > On Wed, Oct 5, 2022 at 8:55 AM Bu Wentian <buwent...@outlook.com> wrote: > > Hi Benoit, > > I checked the packet sizes by tcpdump, and the result is similar to what > you said. I ran iperf server on M2 and iperf client on M1, and tcpdump on > both. I found that on M1(sender), most of the packet length in tcpdump were > 65160, irrespective of which dataplane the router used. However, on > M2(receiver), when using Linux kernel as router, most of the packet length > were 65160; when using VPP as router, most of the packet length varied from > 1448 to 5792. I have pasted some output of tcpdump below. > > > > I have tried some tuning tricks on VPP VM such as CPU pinning, but didn't > get significant improve on throughput. Xiaodong also said that the > bottleneck is not CPU. I am not sure whether it is GSO that caused the > performance difference between Linux routing and VPP routing. I still have > a problem: since the Linux routing and VPP routing both work on Layer3, > which means that they don't care about what the payload of IP packet is, > and just forward them. And the IP packets are from M1, so there should be > no difference between the two experiments. So why could the router GSO can > cause the performance difference? And what can I do to improve VPP > performance if it is GSO reason? > > > > Sincerely, > > Wentian > > > > > > Part of the tcpdump output on M1: > > 15:28:24.563697 IP nt1m1.33998 > 10.250.1.100.5201: Flags [P.], seq > 2657974901:2657985037, ack 1, win 32, options [nop,nop,TS val 3264786600 > ecr 615633982], length 10136 > > 15:28:24.564599 IP nt1m1.33998 > 10.250.1.100.5201: Flags [P.], seq > 2657985037:2658040061, ack 1, win 32, options [nop,nop,TS val 3264786601 > ecr 615633982], length 55024 > > 15:28:24.564709 IP nt1m1.33998 > 10.250.1.100.5201: Flags [P.], seq > 2658040061:2658105221, ack 1, win 32, options [nop,nop,TS val 3264786601 > ecr 615633982], length 65160 > > 15:28:24.564778 IP nt1m1.33998 > 10.250.1.100.5201: Flags [P.], seq > 2658105221:2658170381, ack 1, win 32, options [nop,nop,TS val 3264786601 > ecr 615633983], length 65160 > > 15:28:24.564833 IP nt1m1.33998 > 10.250.1.100.5201: Flags [P.], seq > 2658170381:2658235541, ack 1, win 32, options [nop,nop,TS val 3264786601 > ecr 615633983], length 65160 > > 15:28:24.564900 IP nt1m1.33998 > 10.250.1.100.5201: Flags [P.], seq > 2658235541:2658300701, ack 1, win 32, options [nop,nop,TS val 3264786601 > ecr 615633983], length 65160 > > 15:28:24.564956 IP nt1m1.33998 > 10.250.1.100.5201: Flags [P.], seq > 2658300701:2658329661, ack 1, win 32, options [nop,nop,TS val 3264786602 > ecr 615633983], length 28960 > > > > > > Part of the tcpdump output on M2 (when using VPP routing): > > 15:33:41.573257 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699471685:2699476029, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 4344 > > 15:33:41.573262 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699476029:2699477477, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 1448 > > 15:33:41.573268 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699477477:2699478925, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 1448 > > 15:33:41.573271 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699478925:2699480373, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 1448 > > 15:33:41.573279 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699480373:2699484717, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 4344 > > 15:33:41.573287 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699484717:2699487613, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 2896 > > 15:33:41.573289 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699487613:2699489061, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 1448 > > 15:33:41.573296 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699489061:2699491957, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 2896 > > 15:33:41.573303 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699491957:2699496301, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 4344 > > 15:33:41.573311 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699496301:2699499197, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 2896 > > 15:33:41.573312 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699499197:2699500645, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 1448 > > 15:33:41.573323 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699500645:2699503541, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 2896 > > 15:33:41.573328 IP 10.250.0.100.43298 > nt1m2.5201: Flags [P.], seq > 2699503541:2699504989, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 1448 > > 15:33:41.573328 IP 10.250.0.100.43298 > nt1m2.5201: Flags [.], seq > 2699504989:2699507885, ack 1, win 32, options [nop,nop,TS val 3265103413 > ecr 615950794], length 2896 > > > > Part of the tcpdump output on M2 (when using Linux routing): > > 15:54:06.222861 IP 10.250.0.100.47000 > nt1m2.5201: Flags [P.], seq > 93174477:93239637, ack 1, win 32, options [nop,nop,TS val 3266328062 ecr > 617175444], length 65160 > > 15:54:06.222861 IP 10.250.0.100.47000 > nt1m2.5201: Flags [P.], seq > 93239637:93304797, ack 1, win 32, options [nop,nop,TS val 3266328062 ecr > 617175444], length 65160 > > 15:54:06.222861 IP 10.250.0.100.47000 > nt1m2.5201: Flags [P.], seq > 93304797:93369957, ack 1, win 32, options [nop,nop,TS val 3266328062 ecr > 617175444], length 65160 > > 15:54:06.222900 IP 10.250.0.100.47000 > nt1m2.5201: Flags [P.], seq > 93369957:93435117, ack 1, win 32, options [nop,nop,TS val 3266328062 ecr > 617175444], length 65160 > > 15:54:06.222924 IP 10.250.0.100.47000 > nt1m2.5201: Flags [P.], seq > 93435117:93500277, ack 1, win 32, options [nop,nop,TS val 3266328062 ecr > 617175444], length 65160 > > 15:54:06.223035 IP 10.250.0.100.47000 > nt1m2.5201: Flags [P.], seq > 93500277:93565437, ack 1, win 32, options [nop,nop,TS val 3266328062 ecr > 617175444], length 65160 > > 15:54:06.223035 IP 10.250.0.100.47000 > nt1m2.5201: Flags [P.], seq > 93565437:93630597, ack 1, win 32, options [nop,nop,TS val 3266328062 ecr > 617175444], length 65160 > > > > > > > ------------------------------ > > *发**件人:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> 代表 Benoit Ganne > (bganne) via lists.fd.io > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.fd.io%2F&data=05%7C01%7C%7Cf6c220e7703a4c204c8808daa6f15b71%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638005852829199513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YzImQhc%2Ftc5MUQCmH6LrieMjDhro0zQA4puZQdFcpoc%3D&reserved=0> > <bganne=cisco....@lists.fd.io> > *发**送**时间:* 2022年10月3日 16:25 > *收件人:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> > *主**题:* Re: [vpp-dev] Throughput of VPP on KVM is Significantly Worse > than Linux Kernel > > > > Hi Wentian, Xiaodong, > > When testing VM-to-VM iperf (ie TCP) throughput like Wentian does, the > most important factor is whether GSO is turned on: when using Linux as a > router, it is by default whereas when using VPP as a router it is not. > GSO means Linux is going to use 64K bytes TCP packets whereas no GSO means > VPP is going to use MTU-sized packets (probably 1500 bytes). > You can check by using tcpdump to look at packet on both sides of the > iperf and see packet sizes: when using Linux routing, I'd expect to see GSO > packets (10's of Kbytes) and non-GSO packets (1500 bytes) with VPP. > > With that said, the performances you have with VPP are still really slow, > even without GSO. Getting high-performance in VMs environment is tricky, > you must takes special care about qemu configuration (virtio queue depth, > IO mode etc), cpu-pinning (vCPU threads, vhost threads, vpp workers, etc). > If your usecase is to use this setup as a traditional router connected > through physical interfaces and forwarding packets over a network, and this > is a test setup, I'd recommend to change the setup to match reality, this > is going to be much simpler to setup & optimize correctly. > > Best > ben > > > -----Original Message----- > > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Xiaodong Xu > > Sent: Monday, October 3, 2022 1:18 > > To: vpp-dev@lists.fd.io > > Subject: Re: [vpp-dev] Throughput of VPP on KVM is Significantly Worse > > than Linux Kernel > > > > Hi Wentian, > > > > I ran a perf test with a similar topo to your setup and got the same > > result. The only difference is that the iperf server is running on the > > host rather than in another VM. So the throughput is close to 20Gbps with > > Linux kernel data plane while only 1.5Gbps with VPP dataplane. > > I think we might have run into the same issue as > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.fd.io%2Fg%2Fvpp-dev%2Fmessage%2F9571&data=05%7C01%7C%7Cb45008c6d2934c1d3b3a08daa518cdd3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638003823231871290%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cGXsKO%2FlgwzRnwjQIJdH%2BM5IbASu4sT6d1J12o8BB%2F8%3D&reserved=0 > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.fd.io%2Fg%2Fvpp-dev%2Fmessage%2F9571&data=05%7C01%7C%7Cf6c220e7703a4c204c8808daa6f15b71%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638005852829199513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=boTX%2FhXcPsrEnjIV%2FVf5c%2B7xh96NTEHj77P9ugQX344%3D&reserved=0> > . > > > > Before that, I tried TRex and Pktgen-DPDK, and the results were > different. > > Usually the throughput would be a bit higher with VPP dataplane than > Linux > > kernel data plane, but not much. When I was checking the CPU usage with > > VPP dataplane (I change the rx-mode to interrupt from polling), it is > > pretty low (< 10%), so it sounds to me the bottleneck is not the CPU. > > > > I also ran a similar test with vhost-user driver, following > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fen%2Fblog%2Fhands-vhost-user-warm-welcome-dpdk&data=05%7C01%7C%7Cb45008c6d2934c1d3b3a08daa518cdd3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638003823231871290%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bShUqDr9kokLFDAaMTRvtbcvG4pvuRstEAmS2PpW9K8%3D&reserved=0 > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2Fen%2Fblog%2Fhands-vhost-user-warm-welcome-dpdk&data=05%7C01%7C%7Cf6c220e7703a4c204c8808daa6f15b71%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638005852829199513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8XSatrasv0eBeEBqgCi06l3zMoOryqcV%2FHCTn9jUekk%3D&reserved=0>, > I got > > much better throughput (close to 20Gbps with Pktgen-DPDK), thanks to the > > capability to access share memory brought by vhost-user. There are much > > less memory copies than the above user case where vhost-net is used. > > > > I don't have a chance to run any test in the cloud environment but still > > believe it is an important use case for VPP to run in KVM setup. If > anyone > > in the VPP community can shed some light on this issue, I believe both > > Wentian and I would appreciate it very much. > > > > Xiaodong > > > > On Sun, Oct 2, 2022 at 8:10 AM Bu Wentian <buwent...@outlook.com > > <mailto:buwent...@outlook.com <buwent...@outlook.com>> > wrote: > > > > > > Hi Xiaodong, > > > > Thank you for your reply! > > > > I'm exactly using the VPP 22.06 installed through apt from FD.io > > repo. The linux-cp and linux-nl plugins also come with the VPP from the > > repo. > > > > The virtual NICs on my VMs use the virtio(assigned by > > "model=virtio" when installing with virt-install). The VMs are connected > > through libvirtd networks (auto-create bridges). In my experiments, I can > > ping M2 from M1, and the neighbor table and routing table in VPP seem to > > be correct. > > > > I'm not sure which driver the VPP is using (maybe vfio-pci?). The > > packet counter looked like this: > > vpp# show int GE1 > > Name Idx State MTU > > (L3/IP4/IP6/MPLS) Counter Count > > GE1 1 up 9000/0/0/0 > > rx packets 3719027 > > > > rx bytes 5630430079 > > > > tx packets 1107500 > > > > tx bytes 73176221 > > > > drops 76 > > > > ip4 3718961 > > > > ip6 61 > > > > tx-error 1 > > vpp# show int GE2 > > Name Idx State MTU > > (L3/IP4/IP6/MPLS) Counter Count > > GE2 2 up 9000/0/0/0 > > rx packets 1107520 > > > > rx bytes 73177597 > > > > tx packets 3718998 > > > > tx bytes 5630427889 > > > > drops 63 > > > > ip4 1107455 > > > > ip6 62 > > > > tx-error 1162 > > > > Could you give me more information about how can I get details of > > the error packets? And the main problem is that the VPP forwarding > > performance is much worse than Linux kernel (2Gbps vs 26Gbps), is there > > any method to improve it, or what can I do to find the reason? > > > > Sincerely, > > Wentian > > > > > > > > > > > > > > ________________________________ > > > > 发件人: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io > <vpp-dev@lists.fd.io>> <vpp- > > d...@lists.fd.io <mailto:vpp-dev@lists.fd.io <vpp-dev@lists.fd.io>> > 代表 > Xiaodong Xu > > <stid.s...@gmail.com <mailto:stid.s...@gmail.com <stid.s...@gmail.com>> > > > > 发送时间: 2022年10月2日 1:05 > > 收件人: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io > <vpp-dev@lists.fd.io>> <vpp- > > d...@lists.fd.io <mailto:vpp-dev@lists.fd.io <vpp-dev@lists.fd.io>> > > > 主题: Re: [vpp-dev] Throughput of VPP on KVM is Significantly Worse > > than Linux Kernel > > > > Which vpp version are you using in your testing? As of VPP 22.06, > > linux-cp and linux-nl plugins have been supported and the binary builds > > are available at FD.io repository (https://s3- > > docs.fd.io/vpp/22.10/gettingstarted/installing/ubuntu.html > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.fd.io%2Fvpp%2F22.10%2Fgettingstarted%2Finstalling%2Fubuntu.html&data=05%7C01%7C%7Cf6c220e7703a4c204c8808daa6f15b71%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638005852829199513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Dvj543%2F9D01KoaNAUEF8UbFJQCkj%2F%2FDRN5klYNfK7LQ%3D&reserved=0> > > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs3- > > docs.fd.io > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.fd.io%2F&data=05%7C01%7C%7Cf6c220e7703a4c204c8808daa6f15b71%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638005852829199513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dKeYfyxlnYquppL7tEZYhV9VYH4cwmkiyGO8HOIaNMY%3D&reserved=0> > %2Fvpp%2F22.10%2Fgettingstarted%2Finstalling%2Fubuntu.html&data= > > > 05%7C01%7C%7Cbadce86b78394ff18b8408daa3cf2dc3%7C84df9e7fe9f640afb435aaaaaa > > > aaaaaa%7C1%7C0%7C638002407476279512%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj > > > AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata > > =y5dLTkQx%2B3E%2Bg2VEpwMos6YhNZSD0mSlOdfGR6t77tY%3D&reserved=0> ). > > > > Can you install vpp from the FD.io repo and try again? (BTW, you > > might want to disable the ping plugin if linux-cp is used) I would also > > suggest you add static routes to rule out any issue with FRR (in which > > case you don't actually need linux-cp plugin). > > > > In the meanwhile, I wonder what uio driver you are using for your > > VPP machine (igb_uio, uio_pci_generic, or vfio-pci). Assuming you are > > running virtio-net driver on the guest machine, and you are connecting > the > > M1 and R1, R1 and M2 with linux kernel bridge. > > > > If you still run into any issue, you may want to check the > neighbor > > table and routing table in the VPP system first, and maybe the interface > > counter as well. > > > > Regards, > > Xiaodong > > > > On Sat, Oct 1, 2022 at 3:55 AM Bu Wentian <buwent...@outlook.com > > <mailto:buwent...@outlook.com <buwent...@outlook.com>> > wrote: > > > > > > Hi everyone, > > I am a beginner of VPP, and I'm trying to use VPP+FRR on > KVM > > VMs as routers. I have installed VPP and FRR on Ubuntu 20.04.5 VMs, and > > made them running in a seperated network namespace. I use VPP Linux-cp > > plugin to synchronize the route from kernel stack to VPP. The VPP and FRR > > seems to work, but when I use iperf3 to test the throughput, I find the > > performance of VPP is not good. > > > > I created a very simple topology to test the throughput: > > M1 ----- R1(with VPP) ----- M2 > > M1, M2 are also Ubuntu VMs(without VPP), in different > > subnets. I ran iperf3 server on M1 and client on M2, but only got about > > 2.1Gbps throughput, which is significantly worse than using Linux kernel > > as a router(about 26.1Gbps). > > > > I made another experiment on the topology: > > M1 ------ R1(with VPP) ---- R2(with VPP) ------ M2 > > The iperf3 result is even worse (only 1.6Gbps). > > > > I also noticed that many retransmissions happend during > the > > iperf3 test. If I use Linux kernel as router rather than VPP, no > > retransmission will happen. > > Part of iperf3 output: > > [ ID] Interval Transfer Bitrate Retr > > Cwnd > > > > [ 5] 0.00-1.00 sec 166 MBytes 1.39 Gbits/sec 23 > > 344 KBytes > > [ 5] 1.00-2.00 sec 179 MBytes 1.50 Gbits/sec 49 > > 328 KBytes > > [ 5] 2.00-3.00 sec 203 MBytes 1.70 Gbits/sec 47 > > 352 KBytes > > [ 5] 3.00-4.00 sec 203 MBytes 1.70 Gbits/sec 54 > > 339 KBytes > > [ 5] 4.00-5.00 sec 211 MBytes 1.77 Gbits/sec 59 > > 325 KBytes > > > > > > Another phenomenon I found is that when I ran iperf3 > > directly on the R1 and R2, I got 0 throughput at all. The output of > iperf3 > > is like this: > > [ ID] Interval Transfer Bitrate Retr > > Cwnd > > [ 5] 0.00-1.00 sec 324 KBytes 2.65 Mbits/sec 4 > > 8.74 KBytes > > [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 > > 8.74 KBytes > > [ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 > > 8.74 KBytes > > [ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 > > 8.74 KBytes > > [ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 > > 8.74 KBytes > > [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 > > 8.74 KBytes > > [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 1 > > 8.74 KBytes > > [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 > > 8.74 KBytes > > [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 > > 8.74 KBytes > > [ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 > > 8.74 KBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-10.00 sec 324 KBytes 266 Kbits/sec 7 > > sender > > [ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec > > receiver > > > > > > All my VMs use 4vcpus and 8G RAM. The host machine has > > 16Cores(32 Threads) and 32GB RAM. > > The VMs are connected by libvirtd networks. > > I installed the VPP +FRR following this tutorial: > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fipng.ch%2Fs%2Farticles%2F2021%2F12%2F23%2Fvpp-playground.html&data=05%7C01%7C%7Cb45008c6d2934c1d3b3a08daa518cdd3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638003823231871290%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=c66dSPbOEQY3%2Ftm84G40e%2Fu8%2BtP18pMYLWKRWcu9hzg%3D&reserved=0 > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fipng.ch%2Fs%2Farticles%2F2021%2F12%2F23%2Fvpp-playground.html&data=05%7C01%7C%7Cf6c220e7703a4c204c8808daa6f15b71%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638005852829199513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j4%2BZxcyq3zVxgQmprKTBkmziYnrB%2FoyPR2vnXLkIWmA%3D&reserved=0> > > < > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fipng.ch%2F&data=05%7C01%7C%7Cb45008c6d2934c1d3b3a08daa518cdd3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638003823231871290%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zSmgAvE8hB4YbbT9DO%2FsFAMbRUDlsV1fcdgUyduxx0Q%3D&reserved=0 > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fipng.ch%2F&data=05%7C01%7C%7Cf6c220e7703a4c204c8808daa6f15b71%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638005852829199513%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2B7bG94seosm0vU5MdUyV5%2Bk7rWyebVtVps7qx3cOaW4%3D&reserved=0> > > %2Fs%2Farticles%2F2021%2F12%2F23%2Fvpp- > > > playground.html&data=05%7C01%7C%7Cbadce86b78394ff18b8408daa3cf2dc3%7C84df9 > > > e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638002407476279512%7CUnknown%7CTWFpb > > > GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D% > > > 7C3000%7C%7C%7C&sdata=XHwcWCBs%2F1Fk6mjLlul6S5d390cru%2BzjoTXFclbmg5g%3D&r > > eserved=0> > > The VPP startup.conf is in the attachment. > > > > I want to know why the VPP throughput is worse than Linux > > kernel, and what can I do to improve it (I hope it better than Linux > > kernel forwarding). I have searched on google for the solution but got > > nothing helpful. It will be appreciated if anyone could give me a help. > > Please contact me if more information or logs are needed. > > > > > > Sincerely, > > Wentian Bu > > > > > > > > > > > > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22052): https://lists.fd.io/g/vpp-dev/message/22052 Mute This Topic: https://lists.fd.io/mt/94080585/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
