[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
I believe my test case is flawed, so I cannot verify with certainty if the issue is fixed or not. This is the same test case as I used before, for which I posted code in a gist: https://gist.github.com/stephank/764e3414d57bc3bcb6b3 Here's what I tried: - I started two new c3.large machines from ami-69e76c1e (eu-west-1 HVM 64-bit trusty with instance store) - I downloaded io.js 1.2.0 on machine A, together with the pub.js and sub.js scripts from my gist. - I installed redis-server on machine B and reconfigured redis to bind on to the internal IP (in 10.x.x.x) - The machines were initially running linux-virtual 3.13.0.45.52. I reproduced the issue in this setup by running sub.js twice, then pub.js once on machine A, connecting them to redis on machine B. The 'rides the rocket' message showed up in the logs, and the subs lost their connection. - I enabled trusty proposed on both machines with a pin, and selectively upgraded linux-virtual on both machines. Then rebooted on both. The kernel on both machines is now linux-virtual 3.13.0.46.53. - I ran the same test again, sub.js twice, pub.js once on machine A, connecting to machine B. There were no 'rides the rocket' messages, but the subs still lose their connections. I sporadically get 'net_ratelimit: x callbacks suppressed', but not on every test run. - I disabled scather/gather on both machines, which also dropped their MTU to 1500, and ran the test again several times. There were no more 'net_ratelimit' messages, but the subs still lose their connections. - I installed redis-server on machine A the same way, listening on the internal IP, and ran the same test on machine A, but this time connecting to itself on the internal IP. The test now runs indefinitely. (But this probably doesn't touch the driver.) So I'm not sure what to take away from this. I suppose I could continue by trying to fix my test case to run properly without scather/gather, before again enabling it. Or find a way to trigger it using a different test, such as with redis-benchmark. Stefan, is it sufficient verification if your own testing now shows it fixed? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audi
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
As Stefan Bader mentions in #22, the current workaround is: sudo ethtool -K eth0 sg off -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Thanks for the continued help fixing this! I couldn't reproduce it using redis-benchmark on EC2, but that's okay. Scatter/gather IO is solely a performance flag in the driver? As in, it won't affect applications? The only effect I noticed after disabling it is that it's apparently required for jumbo framing: vif vif-0 eth0: Reducing MTU because no SG offload And it dropped to 1500. But I can live with that. Also, do you have a link to the upstream discussion? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscri
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
So I have a smaller test case. Basically, install Redis (from apt) on one machine, and Node.js (binaries from nodejs.org) with the below scripts on the other. Run pub.js once, and sub.js twice, this quickly triggers the error. The first arg to each script is the address of the redis machine; I use the internal 10.0.0.0/8 address. https://gist.github.com/stephank/764e3414d57bc3bcb6b3 I initially tried to do this using openbsd-inetd echo and several netcat processes, but that doesn't seem to trigger it. Maybe it's something specific about the way Redis distributes pubsub messages to its subscribers? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
If you'd like me to run this on EC2, I can give it a try. A custom kernel would simply be a replacement package? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
I can't comment on the driver implementation details, but I can give some further details about our experience. The app in question was a second screen app for the dutch public broadcasting network for the Eurovision Song Contest. The app was live for two semi-finals on tuesday the 6th and thursday the 8th, as well as the finals saturday the 10th. Load was lowest on the thursday, when the Netherlands did not perform, and highest saturday during the finals. We ran c3.large instances for all shows. During the first run on tuesday was when we first noticed the issue. Shortly before the second run on thursday we noticed the high MTU setting as a possible cause, and changed it to 1500 on half of our machines in the redundant setup. There was a clear difference in connection stability between these machines. For the third run on saturday, we had all machines on the normal MTU of 1500, as we adjusted our startup scripts to force the setting. We had zero connection issues that night, and clean kernel logs, even though this night saw the highest network load of all three. We have several m1.small instances running 24/7 as well, and these have clean kernel logs, but their network load is quite low. The MTU on these has always been untouched, and is a normal 1500, apparently by default. In the instance type list, EC2 shows Compute Optimized instances as having Enhanced Networking. Even though we don't qualify for it, perhaps the networking setup is different for these instances. https://aws.amazon.com/ec2/instance-types/ About a custom kernel, we'd have to look into deploying it, or reproducing the issue on a smaller test setup. I'd prefer looking into the latter, because maybe we can reproduce it between just two instances with stress tools. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-ups
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
For what it's worth, the MTU appears to differ per instance type. At least c3.large has an MTU of 9000 by default, while m1.small has a normal MTU of 1500. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] UdevDb.txt
apport information ** Attachment added: "UdevDb.txt" https://bugs.launchpad.net/bugs/1317811/+attachment/4108537/+files/UdevDb.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] ProcModules.txt
apport information ** Attachment added: "ProcModules.txt" https://bugs.launchpad.net/bugs/1317811/+attachment/4108536/+files/ProcModules.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] UdevLog.txt
apport information ** Attachment added: "UdevLog.txt" https://bugs.launchpad.net/bugs/1317811/+attachment/4108538/+files/UdevLog.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] WifiSyslog.txt
apport information ** Attachment added: "WifiSyslog.txt" https://bugs.launchpad.net/bugs/1317811/+attachment/4108539/+files/WifiSyslog.txt ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] ProcInterrupts.txt
apport information ** Attachment added: "ProcInterrupts.txt" https://bugs.launchpad.net/bugs/1317811/+attachment/4108535/+files/ProcInterrupts.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] ProcEnviron.txt
apport information ** Attachment added: "ProcEnviron.txt" https://bugs.launchpad.net/bugs/1317811/+attachment/4108534/+files/ProcEnviron.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
The machine is no longer running, but I can run apport-collect from a similar machine. The only difference being that we've since added a line to our startup script to reduce the MTU to 1500. ** Tags added: apport-collected ** Description changed: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 May 9 09:54 seq + crw-rw 1 root audio 116, 33 May 9 09:54 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + CRDA: Error: [Errno 2] No such file or directory + CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning + DistroRelease: Ubuntu 14.04 + Ec2AMI: ami-896c96fe + Ec2AMIManifest: (unknown) + Ec2AvailabilityZone: eu-west-1c + Ec2InstanceType: c3.large + Ec2Kernel: aki-52a34525 + Ec2Ramdisk: unavailable + IwConfig: Error: [Errno 2] No such file or directory + Lspci: + + Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 + Package: linux (not installed) + PciMultimedia: + + ProcFB: + + ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 + ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 + RelatedPackageVersions: + linux-restricted-modules-3.13.0-24-generic N/A + linux-backports-modules-3.13.0-24-generic N/A + linux-firmware N/A + RfKill: Error: [Errno 2] No such file or directory + Tags: trusty ec2-images + Uname: Linux 3.13.0-24-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video + _MarkForUpload: True -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent
[Kernel-packages] [Bug 1317811] ProcCpuinfo.txt
apport information ** Attachment added: "ProcCpuinfo.txt" https://bugs.launchpad.net/bugs/1317811/+attachment/4108533/+files/ProcCpuinfo.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] BootDmesg.txt
apport information ** Attachment added: "BootDmesg.txt" https://bugs.launchpad.net/bugs/1317811/+attachment/4108532/+files/BootDmesg.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] [NEW] Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Public bug reported: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) ** Affects: linux (Ubuntu) Importance: Undecided Status: Incomplete ** Tags: amd64 apport-bug ec2-images trusty -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Incomplete Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error:
