Hi Stefan, Interesting finding indeed, looks like the author has only tested on Xen-4.4 and the EC2 instance type that we use report Xen-3.4 as the logline below shows:
Xen version: 3.4.3.amazon (preserve-AD) I'm using a common S3 url in my test cases and the results are the same either using curl or ab to measure the download speed. I've used iperf and netperf between two of our instances and got similar results, the difference is that using the S3 url is noticeable. What is also interesting is that despite the fact that Lucid (with kernel 3.8.11) instance shows GRO as enabled it may not actually use GRO at all since the GRO API was only introduced on 3.13 and that could mean the processing impact will remain the same in both instances, I will try to measure this and will report but in the meanwhile let me know if is there anything I can mesure that may help to identify the issue. Thanks, Rodrigo. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1391339 Title: Trusty kernel inbound network performance regression when GRO is enabled Status in “linux” package in Ubuntu: Confirmed Bug description: After upgrading our EC2 instances from Lucid to Trusty we noticed an increase on download times, Lucid instances were able to download twice as fast as Trusty. After some investigation and testing older kernels (precise, raring and saucy) we confirmed that this only happens on trusty kernel or newer since utopic kernel shows the same result and disabling gro with `ethtool -K eth0 gro off` seems to fix the problem making download speed the same as the Lucid instances again. The problem is easily reproducible using Apache Bench a couple times on files bigger than 100MB on 1Gb network (EC2) using HTTP or HTTPS. Following is an example of download throughput with and without gro: root@runtime-common.22 ~# ethtool -K eth0 gro off root@runtime-common.22 ~# for i in {1..10}; do ab -n 10 $URL | grep "Transfer rate"; done Transfer rate: 85183.40 [Kbytes/sec] received Transfer rate: 86375.80 [Kbytes/sec] received Transfer rate: 94720.24 [Kbytes/sec] received Transfer rate: 84783.82 [Kbytes/sec] received Transfer rate: 84933.09 [Kbytes/sec] received Transfer rate: 84714.04 [Kbytes/sec] received Transfer rate: 84795.58 [Kbytes/sec] received Transfer rate: 84636.54 [Kbytes/sec] received Transfer rate: 84924.26 [Kbytes/sec] received Transfer rate: 84994.10 [Kbytes/sec] received root@runtime-common.22 ~# ethtool -K eth0 gro on root@runtime-common.22 ~# for i in {1..10}; do ab -n 10 $URL | grep "Transfer rate"; done Transfer rate: 74193.53 [Kbytes/sec] received Transfer rate: 56808.91 [Kbytes/sec] received Transfer rate: 56011.58 [Kbytes/sec] received Transfer rate: 82227.74 [Kbytes/sec] received Transfer rate: 70806.54 [Kbytes/sec] received Transfer rate: 72848.10 [Kbytes/sec] received Transfer rate: 58451.94 [Kbytes/sec] received Transfer rate: 61221.33 [Kbytes/sec] received Transfer rate: 58620.21 [Kbytes/sec] received Transfer rate: 69950.03 [Kbytes/sec] received root@runtime-common.22 ~# Similar results can be observed using iperf and netperf as well. Tested kernels: Not affected: 3.8.0-44-generic (precise/raring), 3.11.0-26-generic (saucy) Affected: 3.13.0-39-generic (trusty), 3.16.0-24-generic (utopic) Let me know if I can provide any other information that might be helpful like perf traces and reports. Rodrigo. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1391339/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp