Hi Stefan, Ok I think I've figured out why you were unable to reproduce the slowness, as I mentioned earlier we use the m2 instance type that runs on underlying Xen 3.4 where the t1.micro is probably running on newer infrastructure so I decided to give m3 a try and xen is actually newer (4.2) following is the result of your iperf script on m2 and m3 instances and as you can see the old instance is really affected:
m3 client / m3 server / gro on Average bandwidth = 958.334 Mbits/sec StdDev = 21.3581 Mbits/sec Server Output Sample: [SUM] 0.0-50.5 sec 5.87 GBytes 1000 Mbits/sec [ 5] local ... port 5001 connected with ... port 36291 [ 5] 0.0-10.1 sec 1.16 GBytes 984 Mbits/sec [ 4] local ... port 5001 connected with ... port 36292 [ 4] 0.0-10.1 sec 1.19 GBytes 1.01 Gbits/sec [ 5] local ... port 5001 connected with ... port 36293 [ 5] 0.0-10.1 sec 1.17 GBytes 997 Mbits/sec [ 6] local ... port 5001 connected with ... port 36294 [ 6] 0.0-20.2 sec 1.17 GBytes 498 Mbits/sec [SUM] 0.0-20.2 sec 2.34 GBytes 995 Mbits/sec m2 client / m3 server / gro on Average bandwidth = 643.833 Mbits/sec StdDev = 31.689 Mbits/sec m2 client / m3 server / gro off Average bandwidth = 643.744 Mbits/sec StdDev = 29.4622 Mbits/sec Server Output Sample: [ 6] 0.0-181.1 sec 803 MBytes 37.2 Mbits/sec [ 5] local ... port 5001 connected with ... port 32660 [ 4] 0.0-191.1 sec 802 MBytes 35.2 Mbits/sec [ 6] local ... port 5001 connected with ... port 32661 [ 5] 0.0-201.2 sec 803 MBytes 33.5 Mbits/sec [ 4] local ... port 5001 connected with ... port 32662 [ 6] 0.0-211.2 sec 782 MBytes 31.1 Mbits/sec [ 5] local ... port 5001 connected with ... port 32663 [ 4] 0.0-221.3 sec 798 MBytes 30.2 Mbits/sec [ 6] local ... port 5001 connected with ... port 32664 [ 5] 0.0-231.4 sec 795 MBytes 28.8 Mbits/sec [ 4] local ... port 5001 connected with ... port 32665 [ 6] 0.0-241.4 sec 796 MBytes 27.7 Mbits/sec [ 4] 0.0-251.5 sec 801 MBytes 26.7 Mbits/sec So looks like the problem happens with Xen prior to 4.2 and I still think it is a regression since the kernel 3.11 handles inbound network traffic better with or without gro. Thanks, Rodrigo. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1391339 Title: Trusty kernel inbound network performance regression when GRO is enabled Status in “linux” package in Ubuntu: Confirmed Bug description: After upgrading our EC2 instances from Lucid to Trusty we noticed an increase on download times, Lucid instances were able to download twice as fast as Trusty. After some investigation and testing older kernels (precise, raring and saucy) we confirmed that this only happens on trusty kernel or newer since utopic kernel shows the same result and disabling gro with `ethtool -K eth0 gro off` seems to fix the problem making download speed the same as the Lucid instances again. The problem is easily reproducible using Apache Bench a couple times on files bigger than 100MB on 1Gb network (EC2) using HTTP or HTTPS. Following is an example of download throughput with and without gro: [email protected] ~# ethtool -K eth0 gro off [email protected] ~# for i in {1..10}; do ab -n 10 $URL | grep "Transfer rate"; done Transfer rate: 85183.40 [Kbytes/sec] received Transfer rate: 86375.80 [Kbytes/sec] received Transfer rate: 94720.24 [Kbytes/sec] received Transfer rate: 84783.82 [Kbytes/sec] received Transfer rate: 84933.09 [Kbytes/sec] received Transfer rate: 84714.04 [Kbytes/sec] received Transfer rate: 84795.58 [Kbytes/sec] received Transfer rate: 84636.54 [Kbytes/sec] received Transfer rate: 84924.26 [Kbytes/sec] received Transfer rate: 84994.10 [Kbytes/sec] received [email protected] ~# ethtool -K eth0 gro on [email protected] ~# for i in {1..10}; do ab -n 10 $URL | grep "Transfer rate"; done Transfer rate: 74193.53 [Kbytes/sec] received Transfer rate: 56808.91 [Kbytes/sec] received Transfer rate: 56011.58 [Kbytes/sec] received Transfer rate: 82227.74 [Kbytes/sec] received Transfer rate: 70806.54 [Kbytes/sec] received Transfer rate: 72848.10 [Kbytes/sec] received Transfer rate: 58451.94 [Kbytes/sec] received Transfer rate: 61221.33 [Kbytes/sec] received Transfer rate: 58620.21 [Kbytes/sec] received Transfer rate: 69950.03 [Kbytes/sec] received [email protected] ~# Similar results can be observed using iperf and netperf as well. Tested kernels: Not affected: 3.8.0-44-generic (precise/raring), 3.11.0-26-generic (saucy) Affected: 3.13.0-39-generic (trusty), 3.16.0-24-generic (utopic) Let me know if I can provide any other information that might be helpful like perf traces and reports. Rodrigo. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1391339/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
