[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
ifconfig eth0: flags=4163 mtu 1500 inet 10.50.3.37 netmask 255.255.255.192 broadcast 10.50.3.63 inet6 fe80::c36:5fff:fea1:893b prefixlen 64 scopeid 0x20 ether 0e:36:5f:a1:89:3b txqueuelen 1000 (Ethernet) RX packets 10654784591 bytes 10992813714262 (9.9 TiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 5092521006 bytes 8917322665845 (8.1 TiB) TX errors 0 dropped 190075 overruns 0 carrier 0 collisions 0 next 1 min dropped 190075 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Released Status in linux source package in Utopic: Fix Released Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb856
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
xen_netfront: xennet: skb rides the rocket: 19 slots when i did dmesg from linux mechine i got this output, please sugest to what i have to do im using kernal 3.10.0-123.8.1.el7.x86_64 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Released Status in linux source package in Utopic: Fix Released Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
ifconfig eth0: flags=4163 mtu 1500 inet 10.50.3.37 netmask 255.255.255.192 broadcast 10.50.3.63 inet6 fe80::c36:5fff:fea1:893b prefixlen 64 scopeid 0x20 ether 0e:36:5f:a1:89:3b txqueuelen 1000 (Ethernet) RX packets 10653395098 bytes 10991518671473 (9.9 TiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 5091742032 bytes 8916018268199 (8.1 TiB) TX errors 0 dropped 190064 overruns 0 carrier 0 collisions 0 tx dropped 190064 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Released Status in linux source package in Utopic: Fix Released Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 T
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
** Changed in: linux (Ubuntu Utopic) Status: Fix Committed => Fix Released ** Tags removed: kernel-bug-break-fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Released Status in linux source package in Utopic: Fix Released Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
This bug was fixed in the package linux - 3.13.0-46.75 --- linux (3.13.0-46.75) trusty; urgency=low [ Seth Forshee ] * Release Tracking Bug - LP: #1419963 [ Andy Whitcroft ] * [Debian] arm64 -- build ubuntu drivers - LP: #1411284 * hyper-v -- fix comment handing in /etc/network/interfaces - LP: #1413020 [ Kamal Mostafa ] * [Packaging] force "dpkg-source -I -i" behavior [ Upstream Kernel Changes ] * Revert "[SCSI] mpt2sas: Remove phys on topology change." - LP: #1419838 * Revert "[SCSI] mpt3sas: Remove phys on topology change" - LP: #1419838 * Btrfs: fix transaction abortion when remounting btrfs from RW to RO - LP: #1411320 * Btrfs: fix a crash of clone with inline extents's split - LP: #1413129 * net/mlx4_en: Add VXLAN ndo calls to the PF net device ops too - LP: #1407760 * KVM: x86: SYSENTER emulation is broken - LP: #1414651 - CVE-2015-0239 * powerpc/xmon: Fix another endiannes issue in RTAS call from xmon - LP: #1415919 * ipv6: fix swapped ipv4/ipv6 mtu_reduced callbacks - LP: #1404558, #1419837 * usb: gadget: at91_udc: move prepare clk into process context - LP: #1419837 * KVM: x86: Fix far-jump to non-canonical check - LP: #1419837 * x86/tls: Validate TLS entries to protect espfix - LP: #1419837 * userns: Check euid no fsuid when establishing an unprivileged uid mapping - LP: #1419837 * userns: Document what the invariant required for safe unprivileged mappings. - LP: #1419837 * userns: Only allow the creator of the userns unprivileged mappings - LP: #1419837 * x86_64, switch_to(): Load TLS descriptors before switching DS and ES - LP: #1419837 * isofs: Fix infinite looping over CE entries - LP: #1419837 * batman-adv: Calculate extra tail size based on queued fragments - LP: #1419837 * KEYS: close race between key lookup and freeing - LP: #1419837 * isofs: Fix unchecked printing of ER records - LP: #1419837 * x86_64, vdso: Fix the vdso address randomization algorithm - LP: #1419837 * groups: Consolidate the setgroups permission checks - LP: #1419837 * userns: Don't allow setgroups until a gid mapping has been setablished - LP: #1419837 * userns: Don't allow unprivileged creation of gid mappings - LP: #1419837 * move d_rcu from overlapping d_child to overlapping d_alias - LP: #1419837 * deal with deadlock in d_walk() - LP: #1419837 * Linux 3.13.11-ckt14 - LP: #1419837 * gre: fix the inner mac header in nbma tunnel xmit path - LP: #1419838 * netlink: Always copy on mmap TX. - LP: #1419838 * netlink: Don't reorder loads/stores before marking mmap netlink frame as available - LP: #1419838 * in6: fix conflict with glibc - LP: #1419838 * tg3: tg3_disable_ints using uninitialized mailbox value to disable interrupts - LP: #1419838 * batman-adv: Unify fragment size calculation - LP: #1419838 * batman-adv: avoid NULL dereferences and fix if check - LP: #1419838 * net: Fix stacked vlan offload features computation - LP: #1419838 * net: Reset secmark when scrubbing packet - LP: #1419838 * tcp: Do not apply TSO segment limit to non-TSO packets - LP: #1419838 * alx: fix alx_poll() - LP: #1419838 * team: avoid possible underflow of count_pending value for notify_peers and mcast_rejoin - LP: #1419838 * enic: fix rx skb checksum - LP: #1419838 * net/core: Handle csum for CHECKSUM_COMPLETE VXLAN forwarding - LP: #1419838 * macvlan: unregister net device when netdev_upper_dev_link() fails - LP: #1419838 * netfilter: conntrack: disable generic tracking for known protocols - LP: #1419838 * xen-netfront: Fix handling packets on compound pages with skb_linearize - LP: #1317811, #1419838 * xen-netfront: use correct linear area after linearizing an skb - LP: #1317811, #1419838 * eCryptfs: Force RO mount when encrypted view is enabled - LP: #1419838 * smiapp: Take mutex during PLL update in sensor initialisation - LP: #1419838 * smiapp-pll: Correct clock debug prints - LP: #1419838 * sound: simplify au0828 quirk table - LP: #1419838 * sound: Update au0828 quirks table - LP: #1419838 * af9005: fix kernel panic on init if compiled without IR - LP: #1419838 * writeback: fix a subtle race condition in I_DIRTY clearing - LP: #1419838 * usb: renesas_usbhs: gadget: fix NULL pointer dereference in ep_disable() - LP: #1419838 * KVM: s390: flush CPU on load control - LP: #1419838 * UBI: Fix double free after do_sync_erase() - LP: #1419838 * UBI: Fix invalid vfree() - LP: #1419838 * Drivers: hv: vmbus: Fix a race condition when unregistering a device - LP: #1419838 * driver core: Fix unbalanced device reference in drivers_probe - LP: #1419838 * PCI: Restore detection of read-only BARs - LP: #1419838 * scsi: correct ret
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
This bug was fixed in the package linux - 3.13.0-46.75 --- linux (3.13.0-46.75) trusty; urgency=low [ Seth Forshee ] * Release Tracking Bug - LP: #1419963 [ Andy Whitcroft ] * [Debian] arm64 -- build ubuntu drivers - LP: #1411284 * hyper-v -- fix comment handing in /etc/network/interfaces - LP: #1413020 [ Kamal Mostafa ] * [Packaging] force "dpkg-source -I -i" behavior [ Upstream Kernel Changes ] * Revert "[SCSI] mpt2sas: Remove phys on topology change." - LP: #1419838 * Revert "[SCSI] mpt3sas: Remove phys on topology change" - LP: #1419838 * Btrfs: fix transaction abortion when remounting btrfs from RW to RO - LP: #1411320 * Btrfs: fix a crash of clone with inline extents's split - LP: #1413129 * net/mlx4_en: Add VXLAN ndo calls to the PF net device ops too - LP: #1407760 * KVM: x86: SYSENTER emulation is broken - LP: #1414651 - CVE-2015-0239 * powerpc/xmon: Fix another endiannes issue in RTAS call from xmon - LP: #1415919 * ipv6: fix swapped ipv4/ipv6 mtu_reduced callbacks - LP: #1404558, #1419837 * usb: gadget: at91_udc: move prepare clk into process context - LP: #1419837 * KVM: x86: Fix far-jump to non-canonical check - LP: #1419837 * x86/tls: Validate TLS entries to protect espfix - LP: #1419837 * userns: Check euid no fsuid when establishing an unprivileged uid mapping - LP: #1419837 * userns: Document what the invariant required for safe unprivileged mappings. - LP: #1419837 * userns: Only allow the creator of the userns unprivileged mappings - LP: #1419837 * x86_64, switch_to(): Load TLS descriptors before switching DS and ES - LP: #1419837 * isofs: Fix infinite looping over CE entries - LP: #1419837 * batman-adv: Calculate extra tail size based on queued fragments - LP: #1419837 * KEYS: close race between key lookup and freeing - LP: #1419837 * isofs: Fix unchecked printing of ER records - LP: #1419837 * x86_64, vdso: Fix the vdso address randomization algorithm - LP: #1419837 * groups: Consolidate the setgroups permission checks - LP: #1419837 * userns: Don't allow setgroups until a gid mapping has been setablished - LP: #1419837 * userns: Don't allow unprivileged creation of gid mappings - LP: #1419837 * move d_rcu from overlapping d_child to overlapping d_alias - LP: #1419837 * deal with deadlock in d_walk() - LP: #1419837 * Linux 3.13.11-ckt14 - LP: #1419837 * gre: fix the inner mac header in nbma tunnel xmit path - LP: #1419838 * netlink: Always copy on mmap TX. - LP: #1419838 * netlink: Don't reorder loads/stores before marking mmap netlink frame as available - LP: #1419838 * in6: fix conflict with glibc - LP: #1419838 * tg3: tg3_disable_ints using uninitialized mailbox value to disable interrupts - LP: #1419838 * batman-adv: Unify fragment size calculation - LP: #1419838 * batman-adv: avoid NULL dereferences and fix if check - LP: #1419838 * net: Fix stacked vlan offload features computation - LP: #1419838 * net: Reset secmark when scrubbing packet - LP: #1419838 * tcp: Do not apply TSO segment limit to non-TSO packets - LP: #1419838 * alx: fix alx_poll() - LP: #1419838 * team: avoid possible underflow of count_pending value for notify_peers and mcast_rejoin - LP: #1419838 * enic: fix rx skb checksum - LP: #1419838 * net/core: Handle csum for CHECKSUM_COMPLETE VXLAN forwarding - LP: #1419838 * macvlan: unregister net device when netdev_upper_dev_link() fails - LP: #1419838 * netfilter: conntrack: disable generic tracking for known protocols - LP: #1419838 * xen-netfront: Fix handling packets on compound pages with skb_linearize - LP: #1317811, #1419838 * xen-netfront: use correct linear area after linearizing an skb - LP: #1317811, #1419838 * eCryptfs: Force RO mount when encrypted view is enabled - LP: #1419838 * smiapp: Take mutex during PLL update in sensor initialisation - LP: #1419838 * smiapp-pll: Correct clock debug prints - LP: #1419838 * sound: simplify au0828 quirk table - LP: #1419838 * sound: Update au0828 quirks table - LP: #1419838 * af9005: fix kernel panic on init if compiled without IR - LP: #1419838 * writeback: fix a subtle race condition in I_DIRTY clearing - LP: #1419838 * usb: renesas_usbhs: gadget: fix NULL pointer dereference in ep_disable() - LP: #1419838 * KVM: s390: flush CPU on load control - LP: #1419838 * UBI: Fix double free after do_sync_erase() - LP: #1419838 * UBI: Fix invalid vfree() - LP: #1419838 * Drivers: hv: vmbus: Fix a race condition when unregistering a device - LP: #1419838 * driver core: Fix unbalanced device reference in drivers_probe - LP: #1419838 * PCI: Restore detection of read-only BARs - LP: #1419838 * scsi: correct ret
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
On the other hand, turning off scatter gather caused a rather constant 0.1% packet loss (transmit drops) in my setup (trusty on Xen). VPN gateways, with NAT and firewalling, relatively high throughput at times. Enabling scatter gather removes the constant small packet loss but then I do get the "rides the rocket" events. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages P
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Turning off scatter gather disables the use of fragments in send buffers. So for the xen-netfront driver there is no chance that those fragments end up requiring more than the 19 pages that it can handle in one transaction. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packag
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
I think that is at least enough data to claim this issue verified. @Jon, you could check whether you have the same issue by looking at dmesg and ifconfig output on the sender side. If you see the "rides the rocket" messages together with an incrementing tx drop count, that is this bug. And it should be going away even without disabling sg when using the proposed kernel for the sender. And though it probably should not cause that bad effects, but on the other hand higher layer applications can expect the tcp layer to correct dropped packets. Which maybe is done but possibly ending up (by trying to avoid copying data in memory) with the same (or similar) fragmented send buffer. And in the end all re-tries are dropped. ** Tags removed: verification-needed-trusty ** Tags added: verification-done-trusty ** Tags added: verification-done-utopic ** Changed in: linux (Ubuntu) Assignee: Stefan Bader (smb) => (unassigned) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Crazy idea, could there be situations when sender increases mtu over receiver side? Fundamental question, I tried to understan what ethtool -K eth0 sg off dose in protocoll level can anyone explain, it look's medicine at moment. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kerne
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
I can confirm that ethtool -K eth0 sg off did correct bacula backup problem: -bacula-sd - bacula-fd communication error that stops backup process saying " Error: bsock.c:427 Write error sending reset by peer" -so far no IPV6 trafic jam's whit aiccu, but single missing packet's should not stop aiccu? At Ubuntu 14.04 3.13.0-39-generic LTS, twin server configuration ( suricata, logstash, NFQUEUE, shoreline, keepalived, haproxy, mariadb- galera-cluster, aiccu IPCV6,...), one of twin servers has 4G (Hawei E398) tongle, wifi (hostapd) tongle and dose firewall, routing,... 4 cores so it's not busy ever,... So could these missing packet's realy stop lenghty prosess like backup where 0.1T to 0.7T transfered and other mechanisms can not correct, hide the problem ? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeSt
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Since yesterday morning I've had 3.13.0-46.75 running on 6 VMs. Those VMs haven't had any "xen_netfront: xennet: skb rides the rocket" messages. Meanwhile, 8 other VMs with 3.13.0-44 did have these errors. So, looks good to me. Not absolute proof, but looks good. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscri
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
hi, I have had two mysteriouse problem's, bacula stop's and says as reason connecyion lost due too big packets's ( tried to cange mtu no success) second have been aiccu that dose not recover time to time after connection get up againg, need's service stop & start,... I just gave sudo ethtool -K eth0 sg off and bacula seem's yo work now,... tomorrow morning if all backup's done this might been the corrective action. Backup causes at giganet around 17-40mb/s continouse trafic + internet usage at server/router/4Gtongle connection point. Any ideas how to test? Could this explain my problem? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
I usually only do a redis-benchmark (with -d1000) which triggers the fragmentation on the server side. The benchmark itself never complained. Though I could verify that with the old kernel the tx.dropped count in ifconfig went up. And with the proposed kernel (I picked Utopic/3.16 since you and Heikki look at 3.13) there were no dropped packages. I realize now that the way the message was changed from a normal ratelimited one to a debug ratelimited one is a bit useless as this results in a lot of "callback suppressed" without showing a single line of what that might be. But basically there will be one of those messages whenever the fragments do not fit into the 19 pages of ring buffer. Just before the change this always meant the package was dropped and now it depends on whether the skb could be serialized (which might happen under memory pressure). I would say we wait for the feedback from Heikki and call it good if his VMs survive. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Doh! What I meant was the serialization of the skb may fail under memory pressure. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Installed 3.13.0-46.75 on 6 VMs which exhibited this problem daily. I'll confirm tomorrow evening, if it's gone away. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
I believe my test case is flawed, so I cannot verify with certainty if the issue is fixed or not. This is the same test case as I used before, for which I posted code in a gist: https://gist.github.com/stephank/764e3414d57bc3bcb6b3 Here's what I tried: - I started two new c3.large machines from ami-69e76c1e (eu-west-1 HVM 64-bit trusty with instance store) - I downloaded io.js 1.2.0 on machine A, together with the pub.js and sub.js scripts from my gist. - I installed redis-server on machine B and reconfigured redis to bind on to the internal IP (in 10.x.x.x) - The machines were initially running linux-virtual 3.13.0.45.52. I reproduced the issue in this setup by running sub.js twice, then pub.js once on machine A, connecting them to redis on machine B. The 'rides the rocket' message showed up in the logs, and the subs lost their connection. - I enabled trusty proposed on both machines with a pin, and selectively upgraded linux-virtual on both machines. Then rebooted on both. The kernel on both machines is now linux-virtual 3.13.0.46.53. - I ran the same test again, sub.js twice, pub.js once on machine A, connecting to machine B. There were no 'rides the rocket' messages, but the subs still lose their connections. I sporadically get 'net_ratelimit: x callbacks suppressed', but not on every test run. - I disabled scather/gather on both machines, which also dropped their MTU to 1500, and ran the test again several times. There were no more 'net_ratelimit' messages, but the subs still lose their connections. - I installed redis-server on machine A the same way, listening on the internal IP, and ran the same test on machine A, but this time connecting to itself on the internal IP. The test now runs indefinitely. (But this probably doesn't touch the driver.) So I'm not sure what to take away from this. I suppose I could continue by trying to fix my test case to run properly without scather/gather, before again enabling it. Or find a way to trigger it using a different test, such as with redis-benchmark. Stefan, is it sufficient verification if your own testing now shows it fixed? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audi
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- trusty' to 'verification-done-trusty'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-trusty -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
** Changed in: linux (Ubuntu Trusty) Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Fix Committed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
** Changed in: linux (Ubuntu Utopic) Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Fix Committed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Unfortunately I cannot speed up the process. The fixes have been picked into our stable trees and got a chance to move over to the distro trees next week (which would get them into the next update). Meanwhile you could be working around it by disabling scatter gather (see comment #22). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
also waiting on this -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Any updates on when will the fix come out ? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
This will be fixed in Utopic and Trusty. This was only delayed because the upstream fix was found to cause another regression just about when it would have been picked up. I just re-submitted that and the fix for the regression for getting picked up by our stable trees. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Just chiming in with Jon, I'm using 14.04.1 LTS on EC2 hs1.8xlarge with kernel 3.16.0-29 and I can still reliably produce this error.I thought that this was fixed in 3.14+ but no such luck. Under a particular load the server becomes unresponsive to network requests. I've tried turning off tso and sg on eth0 but this did not resolve the issue. I'm not sure if there is another feature that is causing this in my configuration but I'd be willing to test for it if someone could point me at documentation. Also if there are any logs I can provide to help diagnose this issue please let me know, I'm really eager to see this bug resolved. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _M
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
So will this be fixed in 14.04 at all? I just upgraded to kernel 3.13.0-44 and I'm seeing more of these messages then before. This is on a system that does NAT and DNS. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
** Description changed: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d + break-fix: - 11d3d2a16cc1f05c6ece69a4392e99efb85666a6 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connecti
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Not surprising as we held back for Trusty and Utopic after being told that there was a regression. And as the task status show this only became fixed in current development (Vivid). But Now that both parts are there it is time to get back to stable. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpa
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
I'm still seeing this issue in Ubuntu 14.04 on Ec2, despite using the latest kernel release: 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Here are some of the logs I found: kern.log:1634:Jan 15 00:22:59 staging-cool-load-balancer-20150114-10-36-6-94 kernel: [10070.280631] xen_netfront: xennet: skb rides the rocket: 22 slots kern.log:3523:Jan 15 20:01:23 staging-cool-load-balancer-20150114-10-36-6-94 kernel: [80773.747470] xen_netfront: xennet: skb rides the rocket: 19 slots kern.log:3524:Jan 15 20:01:23 staging-cool-load-balancer-20150114-10-36-6-94 kernel: [80773.791014] xen_netfront: xennet: skb rides the rocket: 19 slots kern.log:3525:Jan 15 20:02:14 staging-cool-load-balancer-20150114-10-36-6-94 kernel: [80824.734485] xen_netfront: xennet: skb rides the rocket: 19 slots kern.log:3526:Jan 15 20:02:22 staging-cool-load-balancer-20150114-10-36-6-94 kernel: [80833.403077] xen_netfront: xennet: skb rides the rocket: 19 slots kern.log:3871:Jan 15 23:39:20 staging-cool-load-balancer-20150114-10-36-6-94 kernel: [93850.874250] xen_netfront: xennet: skb rides the rocket: 20 slots kern.log:3872:Jan 15 23:39:20 staging-cool-load-balancer-20150114-10-36-6-94 kernel: [93851.452640] xen_netfront: xennet: skb rides the rocket: 19 slots kern.log:3873:Jan 15 23:39:20 staging-cool-load-balancer-20150114-10-36-6-94 kernel: [93851.453131] xen_netfront: xennet: skb rides the rocket: 20 slots kern.log:3874:Jan 15 23:39:21 staging-cool-load-balancer-20150114-10-36-6-94 kernel: [93851.695471] xen_netfront: xennet: skb rides the rocket: 19 slots -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRele
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
** Changed in: linux (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Released Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
** Changed in: linux (Ubuntu Trusty) Status: Triaged => Confirmed ** Changed in: linux (Ubuntu Utopic) Status: Triaged => Confirmed ** Changed in: linux (Ubuntu) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: Fix Committed Status in linux source package in Trusty: Confirmed Status in linux source package in Utopic: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True break-fix: - 97a6d1bb2b658ac85ed88205ccd1ab809899884d To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
HVM instance would have the same issue when using PV network drivers (which usually they do for performance). However one also needs to cause fragmented skbs which contain multiple compound page fragments. And that depends on many factors which may not always be easy to meet. By now, there actually seems to be a work-around that has been applied to upstream v3.17. Looks like we have to pick the following (or actually get it into the stable process): commit 97a6d1bb2b658ac85ed88205ccd1ab809899884d Author: Zoltan Kiss Date: Mon Aug 11 18:32:23 2014 +0100 xen-netfront: Fix handling packets on compound pages with skb_linearize ** Also affects: linux (Ubuntu Utopic) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Trusty) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Trusty) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Trusty) Status: New => Triaged ** Changed in: linux (Ubuntu Utopic) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Utopic) Status: New => Triaged ** Description changed: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: - xen_netfront: xennet: skb rides the rocket: 19 slots + xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: - total 0 - crw-rw 1 root audio 116, 1 May 9 09:01 seq - crw-rw 1 root audio 116, 33 May 9 09:01 timer + total 0 + crw-rw 1 root audio 116, 1 May 9 09:01 seq + crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: - + Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: - + ProcFB: - + ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: - linux-restricted-modules-3.13.0-24-generic N/A - linux-backports-modules-3.13.0-24-generic N/A - linux-firmware N/A + linux-restricted-modules-3.13.0-24-generic N/A + linux-backports-modules-3.13.0-24-generic N/A + linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) - --- + --- AlsaDevices: - total 0 - crw-rw 1 root audio 116, 1 May 9 09:54 seq - crw-rw 1 root audio 116, 33 May 9 09:54 timer + total 0 + crw-rw 1 root audio 116, 1 May 9 09:54 seq + crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: - + Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: - + ProcFB: - + ProcKernelCmdLine: root=LA
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
HVM instances does not seem to have this issue, only PV/paravirtual instances. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in linux package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Right, unfortunately a real fix without the need to disable scatter gather will unlikely happen soon. None of the approaches discussed until now seem to find the agreement of everybody as they all would not be perfect. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
As Stefan Bader mentions in #22, the current workaround is: sudo ethtool -K eth0 sg off -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
I am getting this error on Ubuntu 14.04 with latest kernel 3.13.0-30-generic running on c3.2xlarge instances on EC2 PV. Changing MTU to 1500 didn't help. Any other suggestions? We are getting 10-20 dropped packets a day (out of millions) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Oh, ok. It does work quite well on my local guests that come up with 1500 MTU. Maybe the EC2 guests would need a bigger data size value than 1000. But yeah, as long as I have some way to verify whatever comes up to fix this, it is ok. Yes, the loss of jumbo frames was expected. As long as high throughput is not critical it is at least good enough as a work-around. About a upstream discussion: http://www.spinics.net/lists/netdev/msg282340.html Basically it looks like the problem was kind of known but probably did not happens often enough. Or actually complicated to fix. It appears that other drivers will not have that issue as long as the limit is in the actual transfer size and not in the number of pages required to accommodate the frags/scatter gather list. Unfortunately Xen has a limit there that guests have to impose because otherwise the host side driver would shut down the connection completely. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Thanks for the continued help fixing this! I couldn't reproduce it using redis-benchmark on EC2, but that's okay. Scatter/gather IO is solely a performance flag in the driver? As in, it won't affect applications? The only effect I noticed after disabling it is that it's apparently required for jumbo framing: vif vif-0 eth0: Reducing MTU because no SG offload And it dropped to 1500. But I can live with that. Also, do you have a link to the upstream discussion? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscri
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Playing around with this, I actually found an even simpler way to trigger the issue: PV guest #1: Install redis-server (and enable eth0 ip in config) PV guest #2: Install redis-tools and run 'redis-benchmark -q -h -d 1000' The MTU size turns out to be irrelevant, this even happens with 1500 during the batch request tests. What does make a difference is to prevent scatter gather as it was reported in another bug about this (on any host that sees the "rides the rocket" message: sudo ethtool -K eth0 sg off I discussed the issue upstream and the driver should handle this case without dropping the request. It might be a bit complicated so I cannot give an ETA on the fix right now. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
So with the added debugging and running the reproducer with the outside bridge (and so the vifs) and the PV guests eth0 set to 9001 (as seen on EC2), I get the following (format is @): [ 698.108119] xen_netfront: xennet: skb rides the rocket: 19 slots [ 698.108134] header 1490@238 -> 1 slots [ 698.108139] frag #0 1614@2164 -> + 1 pages [ 698.108143] frag #1 3038@1296 -> + 2 pages [ 698.108147] frag #2 6076@1852 -> + 2 pages [ 698.108151] frag #3 6076@292 -> + 2 pages [ 698.108156] frag #4 6076@2828 -> + 3 pages [ 698.108160] frag #5 3038@1268 -> + 2 pages [ 698.108164] frag #6 2272@1824 -> + 1 pages [ 698.108168] frag #7 3804@0 -> + 1 pages [ 698.108172] frag #8 6076@264 -> + 2 pages [ 698.108177] frag #9 3946@2800 -> + 2 pages [ 698.108180] frags adding 18 slots So multiple frags can point to a compound page and also start at an offset. Which makes either the assumption about the size required to handle N frags is wrong or whatever creates that buffer... -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Er
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
** Changed in: linux (Ubuntu) Status: Confirmed => In Progress ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Stefan Bader (smb) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: In Progress Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Good news, the reproducer works on my local system, too. Thanks. :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Yes, the kernel would be a set of dpkg files to be installed via 'dpkg -i'. Of course I still have to code that up. If I can reproduce it with your instructions locally then even better (would cut down turnaround times). Otherwise I can start up some EC2 instances, too. Good to have a simple way to trigger it. Not like some other issues that only happen under production conditions. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHe
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
So I have a smaller test case. Basically, install Redis (from apt) on one machine, and Node.js (binaries from nodejs.org) with the below scripts on the other. Run pub.js once, and sub.js twice, this quickly triggers the error. The first arg to each script is the address of the redis machine; I use the internal 10.0.0.0/8 address. https://gist.github.com/stephank/764e3414d57bc3bcb6b3 I initially tried to do this using openbsd-inetd echo and several netcat processes, but that doesn't seem to trigger it. Maybe it's something specific about the way Redis distributes pubsub messages to its subscribers? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
If you'd like me to run this on EC2, I can give it a try. A custom kernel would simply be a replacement package? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Thanks for the additional info. Definitely the relation to MTU size sounds quite plausible. The checking is on traffic from the guest out and that I would expect to be affected by MTU together with GSO support. And yes, preferably we find a reproducer that does not require a production system to suffer. And ideally for me if I could do so on a local test system to understand the host side. I will try to figure out more details from a stock c3.large if that is possible and maybe whether something like iperf can trigger it there. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-pack
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
I can't comment on the driver implementation details, but I can give some further details about our experience. The app in question was a second screen app for the dutch public broadcasting network for the Eurovision Song Contest. The app was live for two semi-finals on tuesday the 6th and thursday the 8th, as well as the finals saturday the 10th. Load was lowest on the thursday, when the Netherlands did not perform, and highest saturday during the finals. We ran c3.large instances for all shows. During the first run on tuesday was when we first noticed the issue. Shortly before the second run on thursday we noticed the high MTU setting as a possible cause, and changed it to 1500 on half of our machines in the redundant setup. There was a clear difference in connection stability between these machines. For the third run on saturday, we had all machines on the normal MTU of 1500, as we adjusted our startup scripts to force the setting. We had zero connection issues that night, and clean kernel logs, even though this night saw the highest network load of all three. We have several m1.small instances running 24/7 as well, and these have clean kernel logs, but their network load is quite low. The MTU on these has always been untouched, and is a normal 1500, apparently by default. In the instance type list, EC2 shows Compute Optimized instances as having Enhanced Networking. Even though we don't qualify for it, perhaps the networking setup is different for these instances. https://aws.amazon.com/ec2/instance-types/ About a custom kernel, we'd have to look into deploying it, or reproducing the issue on a smaller test setup. I'd prefer looking into the latter, because maybe we can reproduce it between just two instances with stress tools. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-ups
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Thinking about this, I could build a debug kernel to which I add code to print out the layout of the socket buffer when the size check fails. Stéphan, would you be able to run that on a setup that shows the failures? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Could be interesting to find out whether on a m1.small the issue does not occur (although that still could be resulting from other differences in the setup than mtu). Not sure how AWS manages to cause the instance to come up with a different mtu either. In my experiments I had a normal bridge on the host set to 9000 and the guest still had 1500. Though I do not know how the network is set up in EC2 in detail (could be openvswitch). Generally the issue is that something seems to cause packets with a large data buffer. One slot in the xen-net driver is a 4k page. The limit is 18 slots. Anything above that causes the observed message and the packet to be dropped. The host side would have another limit of (usually) 20 slots on which it would assume a malicious guest and disrupts the connection. But since the guest drops at 17 or above the host should never see that number. Unfortunately I am not that deeply understanding the network code, so I will have to ask upstream. As far as I understand a socket buffer can consist of of multiple fragments (kind of a scatter gather list). There is a definition in the code that sets a limit to the number of fragments based on a maximum frame size of 64K. This results in 17 frags (for 4K pages that is 16 + 1 to handle data not starting at page boundary). The Xen driver counts the length of the memory area in all frags (if data in a frag starts at an offset that is added, the code does that for every frag, the question would be whether in theory each frag would be allowed to have an offset because that might add up to more than one page). To the number of pages needed for the frags, the driver then adds the number of pages (can that be more than one?) needed for the header. If that is bigger than 18 (17 for frag + 1 for header?) the rides the rocket error happens. This leaves a few question marks for me: the memory associated with a frag can be a compound page, so I would think that the length might be greater than 4K. I have no clue, yet, how compound pages exactly come into play. Is the 64K limit still enforced by a limit of the number of frags? Can each frag data begin at some offset (and end with more than one page of overall overhead)? Apparently the header can start at some offset, too. So worst case (assuming header length to be less than 4K), if the offset is quite big, that could end up requiring 2 pages. Then if the frag data happens to use up its 17 pages limit, we just would end up hitting the 19 pages failure size. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-bac
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
For what it's worth, the MTU appears to differ per instance type. At least c3.large has an MTU of 9000 by default, while m1.small has a normal MTU of 1500. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
** Changed in: linux (Ubuntu) Importance: Undecided => Medium -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:54 seq crw-rw 1 root audio 116, 33 May 9 09:54 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning DistroRelease: Ubuntu 14.04 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux (not installed) PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: trusty ec2-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
The machine is no longer running, but I can run apport-collect from a similar machine. The only difference being that we've since added a line to our startup script to reduce the MTU to 1500. ** Tags added: apport-collected ** Description changed: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines. Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.) The machines we run are started from ami-896c96fe. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: linux-image-3.13.0-24-generic 3.13.0-24.46 ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 Uname: Linux 3.13.0-24-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 May 9 09:01 seq crw-rw 1 root audio 116, 33 May 9 09:01 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.14.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory: 'iw' Date: Fri May 9 09:11:18 2014 Ec2AMI: ami-896c96fe Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: c3.large Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 May 9 09:54 seq + crw-rw 1 root audio 116, 33 May 9 09:54 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + CRDA: Error: [Errno 2] No such file or directory + CurrentDmesg: [ 24.724129] init: plymouth-upstart-bridge main process ended, respawning + DistroRelease: Ubuntu 14.04 + Ec2AMI: ami-896c96fe + Ec2AMIManifest: (unknown) + Ec2AvailabilityZone: eu-west-1c + Ec2InstanceType: c3.large + Ec2Kernel: aki-52a34525 + Ec2Ramdisk: unavailable + IwConfig: Error: [Errno 2] No such file or directory + Lspci: + + Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 + Package: linux (not installed) + PciMultimedia: + + ProcFB: + + ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 + ProcVersionSignature: User Name 3.13.0-24.46-generic 3.13.9 + RelatedPackageVersions: + linux-restricted-modules-3.13.0-24-generic N/A + linux-backports-modules-3.13.0-24-generic N/A + linux-firmware N/A + RfKill: Error: [Errno 2] No such file or directory + Tags: trusty ec2-images + Uname: Linux 3.13.0-24-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video + _MarkForUpload: True -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots" Status in “linux” package in Ubuntu: Confirmed Bug description: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout. The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent