[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"

xen_netfront: xennet: skb rides the rocket: 19 slots when i did dmesg from linux mechine i got this output, please sugest to what i have to do im using kernal 3.10.0-123.8.1.el7.x86_64 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"

ifconfig eth0: flags=4163 mtu 1500 inet 10.50.3.37 netmask 255.255.255.192 broadcast 10.50.3.63 inet6 fe80::c36:5fff:fea1:893b prefixlen 64 scopeid 0x20 ether 0e:36:5f:a1:89:3b txqueuelen 1000 (Ethernet) RX packets 10654784591

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"

ifconfig eth0: flags=4163 mtu 1500 inet 10.50.3.37 netmask 255.255.255.192 broadcast 10.50.3.63 inet6 fe80::c36:5fff:fea1:893b prefixlen 64 scopeid 0x20 ether 0e:36:5f:a1:89:3b txqueuelen 1000 (Ethernet) RX packets 10653395098

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

** Changed in: linux (Ubuntu Utopic) Status: Fix Committed = Fix Released ** Tags removed: kernel-bug-break-fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title:

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

This bug was fixed in the package linux - 3.13.0-46.75 --- linux (3.13.0-46.75) trusty; urgency=low [ Seth Forshee ] * Release Tracking Bug - LP: #1419963 [ Andy Whitcroft ] * [Debian] arm64 -- build ubuntu drivers - LP: #1411284 * hyper-v -- fix comment handing

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

This bug was fixed in the package linux - 3.13.0-46.75 --- linux (3.13.0-46.75) trusty; urgency=low [ Seth Forshee ] * Release Tracking Bug - LP: #1419963 [ Andy Whitcroft ] * [Debian] arm64 -- build ubuntu drivers - LP: #1411284 * hyper-v -- fix comment handing

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Turning off scatter gather disables the use of fragments in send buffers. So for the xen-netfront driver there is no chance that those fragments end up requiring more than the 19 pages that it can handle in one transaction. -- You received this bug notification because you are a member of Kernel

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

I think that is at least enough data to claim this issue verified. @Jon, you could check whether you have the same issue by looking at dmesg and ifconfig output on the sender side. If you see the rides the rocket messages together with an incrementing tx drop count, that is this bug. And it should

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

I can confirm that ethtool -K eth0 sg off did correct bacula backup problem: -bacula-sd - bacula-fd communication error that stops backup process saying Error: bsock.c:427 Write error sending reset by peer -so far no IPV6 trafic jam's whit aiccu, but single missing packet's should not stop

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Crazy idea, could there be situations when sender increases mtu over receiver side? Fundamental question, I tried to understan what ethtool -K eth0 sg off dose in protocoll level can anyone explain, it look's medicine at moment. -- You received this bug notification because you are a member

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

On the other hand, turning off scatter gather caused a rather constant 0.1% packet loss (transmit drops) in my setup (trusty on Xen). VPN gateways, with NAT and firewalling, relatively high throughput at times. Enabling scatter gather removes the constant small packet loss but then I do get the

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

hi, I have had two mysteriouse problem's, bacula stop's and says as reason connecyion lost due too big packets's ( tried to cange mtu no success) second have been aiccu that dose not recover time to time after connection get up againg, need's service stop start,... I just gave sudo ethtool -K

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Since yesterday morning I've had 3.13.0-46.75 running on 6 VMs. Those VMs haven't had any xen_netfront: xennet: skb rides the rocket messages. Meanwhile, 8 other VMs with 3.13.0-44 did have these errors. So, looks good to me. Not absolute proof, but looks good. -- You received this bug

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Installed 3.13.0-46.75 on 6 VMs which exhibited this problem daily. I'll confirm tomorrow evening, if it's gone away. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title:

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Doh! What I meant was the serialization of the skb may fail under memory pressure. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, xen_netfront:

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

I usually only do a redis-benchmark (with -d1000) which triggers the fragmentation on the server side. The benchmark itself never complained. Though I could verify that with the old kernel the tx.dropped count in ifconfig went up. And with the proposed kernel (I picked Utopic/3.16 since you and

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

I believe my test case is flawed, so I cannot verify with certainty if the issue is fixed or not. This is the same test case as I used before, for which I posted code in a gist: https://gist.github.com/stephank/764e3414d57bc3bcb6b3 Here's what I tried: - I started two new c3.large machines from

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- trusty' to 'verification-done-trusty'. If verification is not done by 5 working days from

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

** Changed in: linux (Ubuntu Trusty) Status: Confirmed = Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, xen_netfront: xennet:

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

** Changed in: linux (Ubuntu Utopic) Status: Confirmed = Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, xen_netfront: xennet:

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Unfortunately I cannot speed up the process. The fixes have been picked into our stable trees and got a chance to move over to the distro trees next week (which would get them into the next update). Meanwhile you could be working around it by disabling scatter gather (see comment #22). -- You

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

also waiting on this -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots Status in linux package

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Any updates on when will the fix come out ? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

This will be fixed in Utopic and Trusty. This was only delayed because the upstream fix was found to cause another regression just about when it would have been picked up. I just re-submitted that and the fix for the regression for getting picked up by our stable trees. -- You received this bug

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Just chiming in with Jon, I'm using 14.04.1 LTS on EC2 hs1.8xlarge with kernel 3.16.0-29 and I can still reliably produce this error.I thought that this was fixed in 3.14+ but no such luck. Under a particular load the server becomes unresponsive to network requests. I've tried turning off

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

So will this be fixed in 14.04 at all? I just upgraded to kernel 3.13.0-44 and I'm seeing more of these messages then before. This is on a system that does NAT and DNS. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Not surprising as we held back for Trusty and Utopic after being told that there was a regression. And as the task status show this only became fixed in current development (Vivid). But Now that both parts are there it is time to get back to stable. -- You received this bug notification because

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

** Description changed: Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log: xen_netfront: xennet: skb rides the rocket: 19 slots Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

I'm still seeing this issue in Ubuntu 14.04 on Ec2, despite using the latest kernel release: 3.13.0-44-generic #73-Ubuntu SMP Tue Dec 16 00:22:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Here are some of the logs I found: kern.log:1634:Jan 15 00:22:59

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

** Changed in: linux (Ubuntu) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, xen_netfront: xennet: skb

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

** Changed in: linux (Ubuntu Trusty) Status: Triaged = Confirmed ** Changed in: linux (Ubuntu Utopic) Status: Triaged = Confirmed ** Changed in: linux (Ubuntu) Status: In Progress = Fix Committed -- You received this bug notification because you are a member of Kernel

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

HVM instance would have the same issue when using PV network drivers (which usually they do for performance). However one also needs to cause fragmented skbs which contain multiple compound page fragments. And that depends on many factors which may not always be easy to meet. By now, there

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

HVM instances does not seem to have this issue, only PV/paravirtual instances. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, xen_netfront:

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

I am getting this error on Ubuntu 14.04 with latest kernel 3.13.0-30-generic running on c3.2xlarge instances on EC2 PV. Changing MTU to 1500 didn't help. Any other suggestions? We are getting 10-20 dropped packets a day (out of millions) -- You received this bug notification because you are a

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

As Stefan Bader mentions in #22, the current workaround is: sudo ethtool -K eth0 sg off -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2,

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Right, unfortunately a real fix without the need to disable scatter gather will unlikely happen soon. None of the approaches discussed until now seem to find the agreement of everybody as they all would not be perfect. -- You received this bug notification because you are a member of Kernel

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Thanks for the continued help fixing this! I couldn't reproduce it using redis-benchmark on EC2, but that's okay. Scatter/gather IO is solely a performance flag in the driver? As in, it won't affect applications? The only effect I noticed after disabling it is that it's apparently required for

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Oh, ok. It does work quite well on my local guests that come up with 1500 MTU. Maybe the EC2 guests would need a bigger data size value than 1000. But yeah, as long as I have some way to verify whatever comes up to fix this, it is ok. Yes, the loss of jumbo frames was expected. As long as high

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Playing around with this, I actually found an even simpler way to trigger the issue: PV guest #1: Install redis-server (and enable eth0 ip in config) PV guest #2: Install redis-tools and run 'redis-benchmark -q -h PV guest #1 IP -d 1000' The MTU size turns out to be irrelevant, this even

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Could be interesting to find out whether on a m1.small the issue does not occur (although that still could be resulting from other differences in the setup than mtu). Not sure how AWS manages to cause the instance to come up with a different mtu either. In my experiments I had a normal bridge

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Thinking about this, I could build a debug kernel to which I add code to print out the layout of the socket buffer when the size check fails. Stéphan, would you be able to run that on a setup that shows the failures? -- You received this bug notification because you are a member of Kernel

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

I can't comment on the driver implementation details, but I can give some further details about our experience. The app in question was a second screen app for the dutch public broadcasting network for the Eurovision Song Contest. The app was live for two semi-finals on tuesday the 6th and

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Thanks for the additional info. Definitely the relation to MTU size sounds quite plausible. The checking is on traffic from the guest out and that I would expect to be affected by MTU together with GSO support. And yes, preferably we find a reproducer that does not require a production system

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

So I have a smaller test case. Basically, install Redis (from apt) on one machine, and Node.js (binaries from nodejs.org) with the below scripts on the other. Run pub.js once, and sub.js twice, this quickly triggers the error. The first arg to each script is the address of the redis machine; I use

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Yes, the kernel would be a set of dpkg files to be installed via 'dpkg -i'. Of course I still have to code that up. If I can reproduce it with your instructions locally then even better (would cut down turnaround times). Otherwise I can start up some EC2 instances, too. Good to have a simple way

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

Good news, the reproducer works on my local system, too. Thanks. :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, xen_netfront: xennet: skb rides

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

** Changed in: linux (Ubuntu) Status: Confirmed = In Progress ** Changed in: linux (Ubuntu) Assignee: (unassigned) = Stefan Bader (smb) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

So with the added debugging and running the reproducer with the outside bridge (and so the vifs) and the PV guests eth0 set to 9001 (as seen on EC2), I get the following (format is length@offset): [ 698.108119] xen_netfront: xennet: skb rides the rocket: 19 slots [ 698.108134] header 1490@238 -

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

For what it's worth, the MTU appears to differ per instance type. At least c3.large has an MTU of 9000 by default, while m1.small has a normal MTU of 1500. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

The machine is no longer running, but I can run apport-collect from a similar machine. The only difference being that we've since added a line to our startup script to reduce the MTU to 1500. ** Tags added: apport-collected ** Description changed: Running Ubuntu 14.04 LTS on EC2, we see a lot

[Kernel-packages] [Bug 1317811] Re: Dropped packets on EC2, xen_netfront: xennet: skb rides the rocket: x slots

** Changed in: linux (Ubuntu) Importance: Undecided = Medium -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1317811 Title: Dropped packets on EC2, xen_netfront: xennet: skb rides the