Hi all, I have come across a network problem when doing network performance testing. From vm, wget a file on external network is much slower than expected.
The network is setup like this [1]. To download a file, packets travel from target server public ip to public ip on neutron node, via iptables nat rule to 10.21.144.5 on neutron node, via neutron virtual router (l3-agent) to 10.20.144.100, then to virtual subnet's gateway at 10.21.145.1 and finally to vm's fixed ip. This is a long chain. Here are some numbers: wget http://mytestserver/big-file.deb I use /etc/hosts to set mytestserver to the IP I want to test. Download from 10.20.165.2: 47.2 MB/s (395Mbps) slow but acceptable. Download from 10.20.165.2's public IP: 6.75 MB/s (56Mbps) way too slow. I also tried the reverse: run web server on vm, and wget on 10.20.165.2: Download from vm's floatingip (10.21.144.106) from outside: 105MB/s (880Mbps) I have yet to test upload speed to outside network. Upload to 10.20.165.0/24and 10.21.144.0/24 are both at 40+MB/s. Upload to public ip not tested yet. Here are some raw iperf performance result: with "GRO off" on br-ex and eth1: | scenario | outgoing (from vm) | incoming (to vm) | |----------------------------+--------------------+------------------| | vm to vm on same host | 3.16 Gbps | - | | vm to vm on different host | 613 Mbps | - | | vm to 10.21.144.0/24 | 887 Mbps | 860 Mbps | | vm to 10.20.165.0/24 | 888 Mbps | 818 Mbps | | vm to <public ip> | 919 Mbps | 817 Mbps | I'm using neutron with ovs plugin and GRE tunnels. There are no errors in neutron's log files, except ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver and similar which has always existed since I deploy openstack. Things I have tried and did not resolve the problem: - turn off GRO on neutron node br-ex/eth1 interface (kept off for all wget/scp tests) - allow all ports in security groups - drop iptables rules on neutron node - lower MTU to 1400 (This helped a little, all wget numbers above have MTU set to 1400 in vm) I also suspected this nat rule is too slow: iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE But test had proved me wrong. I have tested using 10.21.144.5 as the default gateway in a physical server and wget the same file from is lightning fast. So is this download speed typical? If not, how can I isolate the problem and figure out what cause the slowness? Thanks, Yuanle [1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
