Hi! Maybe you're facing this problem:
https://bugs.launchpad.net/neutron/+bug/1252900 I'm seeing that you tried to turn GRO off but, is your Network Node a Virtual Machine, or a physical server? If it is a VM (KVM), you'll need to turn GRO off at the hypervisor too. Happy new year! Cheers! Thiago On 31 December 2013 13:55, sylecn <[email protected]> wrote: > Hi all, > > I have come across a network problem when doing network performance > testing. From vm, wget a file on external network is much slower than > expected. > > The network is setup like this [1]. > > To download a file, packets travel from target server public ip to public > ip on neutron node, via iptables nat rule to 10.21.144.5 on neutron node, > via neutron virtual router (l3-agent) to 10.20.144.100, then to virtual > subnet's gateway at 10.21.145.1 and finally to vm's fixed ip. This is a > long chain. > > Here are some numbers: > wget http://mytestserver/big-file.deb > > I use /etc/hosts to set mytestserver to the IP I want to test. > > Download from 10.20.165.2: 47.2 MB/s (395Mbps) slow > but acceptable. > Download from 10.20.165.2's public IP: 6.75 MB/s (56Mbps) way > too slow. > > I also tried the reverse: run web server on vm, and wget on 10.20.165.2: > > Download from vm's floatingip (10.21.144.106) from outside: 105MB/s > (880Mbps) > > I have yet to test upload speed to outside network. Upload to > 10.20.165.0/24 and 10.21.144.0/24 are both at 40+MB/s. Upload to public > ip not tested yet. > > Here are some raw iperf performance result: > > with "GRO off" on br-ex and eth1: > | scenario | outgoing (from vm) | incoming (to vm) | > |----------------------------+--------------------+------------------| > | vm to vm on same host | 3.16 Gbps | - | > | vm to vm on different host | 613 Mbps | - | > | vm to 10.21.144.0/24 | 887 Mbps | 860 Mbps | > | vm to 10.20.165.0/24 | 888 Mbps | 818 Mbps | > | vm to <public ip> | 919 Mbps | 817 Mbps | > > I'm using neutron with ovs plugin and GRE tunnels. There are no errors in > neutron's log files, except > > ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver > > and similar which has always existed since I deploy openstack. > > Things I have tried and did not resolve the problem: > > - turn off GRO on neutron node br-ex/eth1 interface (kept off for all > wget/scp tests) > - allow all ports in security groups > - drop iptables rules on neutron node > - lower MTU to 1400 (This helped a little, all wget numbers above have > MTU set to 1400 in vm) > > I also suspected this nat rule is too slow: > > iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE > > But test had proved me wrong. I have tested using 10.21.144.5 as the > default gateway in a physical server and wget the same file from is > lightning fast. > > So is this download speed typical? If not, how can I isolate the problem > and figure out what cause the slowness? > > Thanks, > Yuanle > > [1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : [email protected] > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
