Hello, This is a repost from the mailing list: [email protected]
I have built openvswitch with dpdk and user vhost support and tried to run some simple performance tests to make sure the environment was healthy. The issue that I am facing is that under a UDP workload, I am getting good performance but once i try to run any type of TCP workload (such as a SCP), the connection will stall and/or will be very slow. For example, the same SCP command to another VM or a bare metal system will take a coupld of seconds to complete whereas on my user vhost VM it can take up to 4 minutes. But sometimes the SCP wil be fast. I also noticed that when the SCP is stalled, a flood ping triggered to the uservhost VM from another window would unblock / unstall the SCP command... and the SCP command would complete in a couple of seconds. Here's my setup: the host is Fedora 20, the hardware is a on a NUMA server: 2 nodes / sockets with 8 cores each, each core has 2 threads. I am using a 82599 Intel NIC and my network is 10 Gbps. The fedora server has 32 huge pages enabled, I added the following line in /etc/grub2.conf: default_hugepagesz=1G hugepagesz=1G hugepages=32 Made sure the huge pages are here and free at reboot: /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/free_hugepages 16 /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/free_hugepages 16 Here is how I built my openvswitch dpdk environment on the Fedora server: 1. download latest ovs-master.zip from https://github.com/openvswitch/ovs and unzip it in /home/ovs 2. download dpdk-1.7.1 from dpdk website and deploy the tarball in /home/dpdk 3. apply the patch: http://patchwork.openvswitch.org/patch/6280/ in the /home/ovs directory 4. build dpdk as per INSTALL.DPDK instructions (with DPDK_BUILD=/home/dpdk/x86_64-native-linuxapp-gcc/) 5. build ovs with dpdk support as per INSTALL.DPDK instructions 6. build eventfd_link module in the ovs directory (make module) 7. create the huge page FS and mount it: mkdir -p /mnt/huge 8. cleaned up any existing OVS DB configuration as per INSTALL.DPDK instructions 9. Copied the DPDK source to a FS in case my VM will need to use it later (cp -aL /home/dpdk/* /tmp/qemu_share/DPDK) 10. Load several modules: tun openvswitch cuse fuse /home/ovs/utilities/eventfd_link/eventfd_link.ko /home/dpdk/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko 11. assigned 1 NIC to dpdk control: dpdk_nic_bind.py --bind=igb_uio 0000:b0:00.1 12. init OVS DB: ./ovsdb/ovsdb-tool create /usr/local/etc/openvswitch/conf.db ./vswitchd/vswitch.ovsschema ./ovsdb/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach ./utilities/ovs-vsctl --no-wait init 13. Start vswitchd process with 8GB on each numa node (if reserve memory on just 1 numa node, creating dpdk port will fail: cannot allocate memory) ./vswitchd/ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 8192,8192 -- unix:/usr/local/var/run/openvswitch/db.sock --pidfile --detach 14. Create bridge and its dpdk and dpdkvhost ports: ./utilities/ovs-vsctl add-br br0 ./utilities/ovs-vsctl set bridge br0 datapath_type=netdev ./utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk ./utilities/ovs-vsctl add-port br0 dpdkvhost0 -- set Interface dpdkvhost0 type=dpdkvhost 15. Start a VM which has Oracle Linux 6.5 installed on it: qemu-system-x86_64 -smp 2 -hda /var/lib/libvirt/images/appdev81-1v2.img -m 4096 -cpu host --enable-kvm -name 'appdev81-1v2' -vnc :2 -pidfile /tmp/appdev81-1v2.pid \ -drive file=fat:rw:/tmp/qemu_share -mem-path /dev/hugepages -mem-prealloc \ -netdev tap,id=net1,script=no,downscript=no,ifname=dpdkvhost0,vhost=on \ -device virtio-net-pci,netdev=net1,mac=52:54:00:00:01:02,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off \ -daemonize At that point the VM will boot fine and get an IP Adddress from the DHCP server, but any 'heavy' TCP workload will stall. I've tried several tweaks that did not help: - deactivate offload features on the guest (ethtool --offload eth0 gso off ; ethtool --offload eth0 tso off ; ethtool --offload eth0 gro off ; ethtool --offload eth0 rx off tx off) - tune the network buffers in the guest (sysctl -w net.core.rmem_max=134217728 ; sysctl -w net.core.wmem_max=134217728 ; sysctl -w net.ipv4.tcp_rmem='4096 87380 67108864' sysctl -w net.ipv4.tcp_wmem='4096 65536 67108864' ;sysctl -w net.ipv4.tcp_mem='67108864 67108864 67108864' sysctl -w net.ipv4.route.flush=1) - try to reinstall the guest from Oracle Linux 6.5 CD - try to install the guest from Fedora 20 CD More info on my environment: Host: Fedora 20 with Kernel 3.16.3-200.fc20.x86_64 And KVM qemu-kvm-tools-1.6.2-8.fc20.x86_64 and qemu-kvm-1.6.2-9.fc20.x86_64 Let me know if I need to provide more information. Any help is welcome. -- jb
_______________________________________________ discuss mailing list [email protected] http://openvswitch.org/mailman/listinfo/discuss
