Hi All, I am trying out L2FWD and L3FWD samples on KVM guests.
This is my setup: Host: # uname -a Linux scale01 4.13.0-32-generic #35-Ubuntu SMP Thu Jan 25 09:13:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 17.10 Release: 17.10 Codename: artful VM deployment (u for ubuntu): u01-- u02 -- u03 u02 is the L2/L3 forwarder. Here is the VM instantiation command line for u02: qemu-system-x86_64 -enable-kvm -name guest=u02,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-26-u02/master-key.aes -machine pc-i440fx-artful,accel=kvm,usb=off,dump-guest-core=off -cpu host -m 16384 -realtime mlock=off -smp 8,sockets=1,cores=8,threads=1 -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/26-u02,share=yes,size=17179869184,host-nodes=1,policy=bind -numa node,nodeid=0,cpus=0-7,memdev=ram-node0 -uuid de14a442-a595-4e96-b150-f284a90fb84a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-26-u02/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -drive file=/data/ssd0/u02.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fds=25:29:30:31,id=hostnet0,vhost=on,vhostfds=32:33:34:35 -device virtio-net-pci,mrg_rxbuf=on,mq=on,vectors=10,netdev=hostnet0,id=net0,mac=52:54:00:7e:b7:d6,bus=pci.0,addr=0x7 -netdev tap,fds=36:37:38:39,id=hostnet1,vhost=on,vhostfds=40:41:42:43 -device virtio-net-pci,mrg_rxbuf=on,mq=on,vectors=10,netdev=hostnet1,id=net1,mac=52:54:00:7e:b7:d7,bus=pci.0,addr=0x8 -netdev tap,fds=44:45:46:47,id=hostnet2,vhost=on,vhostfds=48:49:50:51 -device virtio-net-pci,mrg_rxbuf=on,mq=on,vectors=10,netdev=hostnet2,id=net2,mac=52:54:00:7e:b7:d5,bus=pci.0,addr=0x9 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on Linux bridge is used for the networks. Here is the output of brctl on the host: # brctl show bridge name bridge id STP enabled interfaces br-cli 8000.52540072392f no br-cli-nic vnet0 vnet4 br-srv 8000.5254006d0037 no br-srv-nic vnet1 vnet6 virbr0 8000.525400192db5 yes virbr0-nic vnet2 vnet3 vnet5 When I use the Linux bridge / routing on u02 (L2 or L3 forwarder inside u02), I get about 20 Gbps unidirectional iperf throughput in both L2 and L3 modes. I get about 3 Gbps with DPDK. To get DPDK to work, I had to turn off checksum offloading. I have the following questions: 1. How may I use IPv4 checksum offload with igb_uio? Since the original guest Linux driver - virtio_net can do this, it seems I have to only make changes on the guest. 2. I am unable to use more than one CPU with igb_uio. Do I need to make some configuration change? Please note that I do not see these issues with ESXi / VMXNet3 virtual NICs. The problem is seen only with virtio on KVM. It is not possible for me to use PCI pass-through / SRIOV at this stage of our development. I need to make it work on emulated NICs. Regards, Deepak Mohanty