I see the about the same numbers. Ethtool -k in the non-DPDK VMs show a lot of offloads that are not available in the DPDK VMs. One of these is tcp segmentation offload. When I turned this off, the rate dropped from about 17 to 2 Gbps.
On Wed, Jan 4, 2017 at 9:59 AM, Rajalakshmi Prabhakar <krajalaks...@tataelxsi.co.in> wrote: > Hello, > > FYI. > > Highlighted my observation. Kindly comment on the same. > > Best Regards > Rajalakshmi Prabhakar > > Specialist - Communication BU | Wireless Division > TATA ELXSI > IITM Research park , Kanagam road , Taramani , Chennai 600 113 India > Tel +91 44 66775031 Cell +91 9789832957 > www.tataelxsi.com<http://www.tataelxsi.com/> > > > ________________________________ > From: Bodireddy, Bhanuprakash <bhanuprakash.bodire...@intel.com> > Sent: Wednesday, December 28, 2016 6:46 PM > To: Rajalakshmi Prabhakar; d...@dpdk.org; us...@dpdk.org > Subject: RE: DPDK: Inter VM communication of iperf3 TCP throughput is very > low on same host compare to non DPDK throughput > >>-----Original Message----- >>From: Rajalakshmi Prabhakar [mailto:krajalaks...@tataelxsi.co.in] >>Sent: Tuesday, December 27, 2016 9:52 AM >>To: d...@dpdk.org; us...@dpdk.org >>Cc: Bodireddy, Bhanuprakash <bhanuprakash.bodire...@intel.com> >>Subject: DPDK: Inter VM communication of iperf3 TCP throughput is very low >>on same host compare to non DPDK throughput >> >>Hello, >>Kindly Support me to get high throughput in inter VM communication of iperf3 >>TCP in OpenStack DPDK host. I am not sure that I am mailing to the right ID >>sorry for the inconvenience. > OVS mailing list should be the appropriate one for the problem you reported > here. > Use ovs-disc...@openvswitch.org (or) d...@openvswitch.org. > >> >>Host - ubuntu16.04 >>devstack - stable/newton >>which install DPDK 16.07 and OVS 2.6 versions >>with DPDK plugin and following DPDK configurations >>Grub changes >>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash default_hugepagesz=1G >>hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on" >>local.conf - changes for DPDK >>enable_plugin networking-ovs-dpdk >>https://git.openstack.org/openstack/networking-ovs-dpdk master >>OVS_DPDK_MODE=controller_ovs_dpdk >>OVS_NUM_HUGEPAGES=8 >>OVS_CORE_MASK=2 >>OVS_PMD_CORE_MASK=4 > Only one PMD core is used in your case, scaling the PMD threads can be one > option for higher throughputs. > >>OVS_DPDK_BIND_PORT=False >>OVS_SOCKET_MEM=2048 >>OVS_DPDK_VHOST_USER_DEBUG=n >>OVS_ALLOCATE_HUGEPAGES=True >>OVS_HUGEPAGE_MOUNT_PAGESIZE=1G >>MULTI_HOST=1 >>OVS_DATAPATH_TYPE=netdev >>before VM creation >>#nova flavor-key m1.small set hw:mem_page_size=1048576 >>Able to create two ubuntu instance in flavor m1.small > How many cores are assigned for the VM and have you tried CPU pinning options > instead of allowing the threads to float across the cores? > 1vcpu and its pinned > >>Achieved iperf3 tcp throughput of ~7.5Gbps > Are you seeing high drops at the vHost ports and retransmissions? Do you see > the same throughput difference with UDP traffic? > However I can't explain now the throughput gap you are observing here. Couple > of things worth checking > I am seeing drops in vHostport and same differnece is observed fro UDP also > - For thread starvation (htop to see thread activity on the cores) > - I see that you have single socket setup and no QPI involved. As you have > HT enabled, check if appropriate thread siblings are used. > - Check pmd thread/port statistics for anomaly. > > BTW, the responses can be slow at this point due to yearend vacation. > > Regards, > Bhanuprakash. > >>Ensured the vhostport is created and HugePage is consumed at the end of >>2VM created each of 2GB ie 4GB for VMs and 2GB for socket totally 6GB >>$ sudo cat /proc/meminfo |grep Huge >>AnonHugePages: 0 kB >>HugePages_Total: 8 >>HugePages_Free: 2 >>HugePages_Rsvd: 0 >>HugePages_Surp: 0 >>Hugepagesize: 1048576 kB >>The same scenario carried for without DPDK case of openstack and achieved >>higher throughput of ~19Gbps, which is contradictory to the expected results. >>Kindly suggest me what additional DPDK configuration to be done for high >>throughput. Also tried cpu pinning and multi queue for OpenStack DPDK but >>no improvement in the result. >>Test PC is single NUMA only.I am not doing NIC binding as only trying to >>validate inter-VM communication in same host. PFB my PC configurations. >>$ lscpu >>Architecture: x86_64 >>CPU op-mode(s): 32-bit, 64-bit >>Byte Order: Little Endian >>CPU(s): 12 >>On-line CPU(s) list: 0-11 >>Thread(s) per core: 2 >>Core(s) per socket: 6 >>Socket(s): 1 >>NUMA node(s): 1 >>Vendor ID: GenuineIntel >>CPU family: 6 >>Model: 63 >>Model name: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz >>Stepping: 2 >>CPU MHz: 1212.000 >>CPU max MHz: 2400.0000 >>CPU min MHz: 1200.0000 >>BogoMIPS: 4794.08 >>Virtualization: VT-x >>L1d cache: 32K >>L1i cache: 32K >>L2 cache: 256K >>L3 cache: 15360K >>NUMA node0 CPU(s): 0-11 >>Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat >>pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1g b >>rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology >>nonstop_t sc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx >>smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic >>movbe popcnt tsc_deadline _timer aes xsave avx f16c rdrand lahf_lm abm >>epb tpr_shadow vnmi flexpriority ep t vpid fsgsbase tsc_adjust bmi1 avx2 >>smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat >>pln pts >>I am following INSTALL.DPDK.ADVANCED.md but no clue on low throughput. >> >> >>Best Regards >>Rajalakshmi Prabhakar >> >>Specialist - Communication BU | Wireless Division >>TATA ELXSI >>IITM Research park , Kanagam road , Taramani , Chennai 600 113 India >>Tel +91 44 66775031 Cell +91 9789832957 >>www.tataelxsi.com<http://www.tataelxsi.com> > _______________________________________________ > dev mailing list > d...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev