Bug#592187: [stable] Bug#576838: virtio network crashes again
Am Donnerstag, den 09.09.2010, 04:23 +0100 schrieb Ben Hutchings: On Tue, 2010-09-07 at 13:25 +0200, Lukas Kolbe wrote: On Wed, 2010-09-01 at 05:26 +0100, Ben Hutchings wrote: On Tue, 2010-08-31 at 17:34 +0200, Lukas Kolbe wrote: On Tue, 2010-08-31 at 06:35 -0700, Greg KH wrote: [...] Then how about convincing the Debian kernel developers to accept these patches, and work through any regressions that might be found and after that, reporting back to us? Ben? The reason I contacted you was precisely because it went into 2.6.33.2, e.g. was already accepted into a -stalbe release. I didn't expect it to be such an issue. That's not likely if people spread FUD about the backlog patches! Dave, did you explicitly exclude these patches from 2.6.32 when you submitted them to stable, or is it just that 5534979 udp: use limited socket backlog depends on a1ab77f ipv6: udp: Optimise multicast reception? The former patch doesn't look too hard to backport to 2.6.32 (see below). Anybody? We've currently rolled out our own 2.6.32 kernel with these fixes applied, and they indeed fix a system crash under our nfs-load. What else can I do to get these fixes into either Debians' 2.6.32 or Greg's stable 2.6.32 series? [...] These patches will be included in Debian's version 2.6.32-22. We'll see how that goes. I owe you a few beers. Thanks a million! Ben. Lukas -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1284025201.15888.1.ca...@larosa.fritz.box
Bug#592187: [stable] Bug#576838: virtio network crashes again
On Tue, 2010-09-07 at 13:25 +0200, Lukas Kolbe wrote: On Wed, 2010-09-01 at 05:26 +0100, Ben Hutchings wrote: On Tue, 2010-08-31 at 17:34 +0200, Lukas Kolbe wrote: On Tue, 2010-08-31 at 06:35 -0700, Greg KH wrote: [...] Then how about convincing the Debian kernel developers to accept these patches, and work through any regressions that might be found and after that, reporting back to us? Ben? The reason I contacted you was precisely because it went into 2.6.33.2, e.g. was already accepted into a -stalbe release. I didn't expect it to be such an issue. That's not likely if people spread FUD about the backlog patches! Dave, did you explicitly exclude these patches from 2.6.32 when you submitted them to stable, or is it just that 5534979 udp: use limited socket backlog depends on a1ab77f ipv6: udp: Optimise multicast reception? The former patch doesn't look too hard to backport to 2.6.32 (see below). Anybody? We've currently rolled out our own 2.6.32 kernel with these fixes applied, and they indeed fix a system crash under our nfs-load. What else can I do to get these fixes into either Debians' 2.6.32 or Greg's stable 2.6.32 series? [...] These patches will be included in Debian's version 2.6.32-22. We'll see how that goes. Ben. -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. signature.asc Description: This is a digitally signed message part
Bug#592187: [stable] Bug#576838: virtio network crashes again
On Wed, 2010-09-01 at 05:26 +0100, Ben Hutchings wrote: On Tue, 2010-08-31 at 17:34 +0200, Lukas Kolbe wrote: On Tue, 2010-08-31 at 06:35 -0700, Greg KH wrote: [...] Then how about convincing the Debian kernel developers to accept these patches, and work through any regressions that might be found and after that, reporting back to us? Ben? The reason I contacted you was precisely because it went into 2.6.33.2, e.g. was already accepted into a -stalbe release. I didn't expect it to be such an issue. That's not likely if people spread FUD about the backlog patches! Dave, did you explicitly exclude these patches from 2.6.32 when you submitted them to stable, or is it just that 5534979 udp: use limited socket backlog depends on a1ab77f ipv6: udp: Optimise multicast reception? The former patch doesn't look too hard to backport to 2.6.32 (see below). Anybody? We've currently rolled out our own 2.6.32 kernel with these fixes applied, and they indeed fix a system crash under our nfs-load. What else can I do to get these fixes into either Debians' 2.6.32 or Greg's stable 2.6.32 series? Ben. From: Zhu Yi yi@intel.com Date: Thu, 4 Mar 2010 18:01:42 + Subject: [PATCH] udp: use limited socket backlog [ Upstream commit 55349790d7cbf0d381873a7ece1dcafcffd4aaa9 ] Make udp adapt to the limited socket backlog change. Cc: David S. Miller da...@davemloft.net Cc: Alexey Kuznetsov kuz...@ms2.inr.ac.ru Cc: Pekka Savola (ipv6) pek...@netcore.fi Cc: Patrick McHardy ka...@trash.net Signed-off-by: Zhu Yi yi@intel.com Acked-by: Eric Dumazet eric.duma...@gmail.com Signed-off-by: David S. Miller da...@davemloft.net Signed-off-by: Greg Kroah-Hartman gre...@suse.de [bwh: Backport to 2.6.32] Regards, Lukas -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1283858701.5722.78.ca...@quux.techfak.uni-bielefeld.de
Bug#592187: [stable] Bug#576838: virtio network crashes again
Am Montag, den 30.08.2010, 10:21 -0700 schrieb Greg KH: On Mon, Aug 30, 2010 at 09:46:36AM -0700, David Miller wrote: From: Greg KH g...@kroah.com Date: Mon, 30 Aug 2010 07:50:17 -0700 As I stated above, I need the ACK from David to be able to add these patches. David? I believe there were some regressions caused by these changes that were fixed later, a bit after those commites went into the tree. I'm only confortable ACK'ing this if someone does due diligence and checks for any such follow-on fixes to that series. It's a pretty non-trivial set of patches and has the potential to kill performance which would be a very serious regression. Fair enough. Yep, thanks! Who's done the checks to find out any problems with these patches? I'll skim the changelogs in 2.6.3[345].x to see if there are any related patches. And what is keeping you from moving to the .35 kernel tree instead? Basically, distribution support. Debian Squeeze will ship with 2.6.32, as Ubuntu already did for their current LTS - and I really want Debian's kernel to be as reliable and stable as possible (btw, that's why I initally reported this as a debian bug, because at that time I wasn't using vanilla kernels. Now that I know how git bisect works, it will hopefully be easier for me to pinpoint regressions in the future). Also, We do not really have enough hardware to test new upstream releases thouroughly before going into production (e.g., we only have one big tape library with one big disk pool, so no test if tape-on-mptsas and aacraid work properly and stable in a newer upstream releases. Kind regards, Lukas -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1283242616.14680.832.ca...@larosa.fritz.box
Bug#592187: [stable] Bug#576838: virtio network crashes again
Who's done the checks to find out any problems with these patches? I'll skim the changelogs in 2.6.3[345].x to see if there are any related patches. This is all I could find in current 2.6.36-rc2 (via git log | grep, minus rps/rfs patches). I don't know anything about these, but they sound related. If anybody with insight into the actual codebase could take a look there ... commit c377411f2494a931ff7facdbb3a6839b1266bcf6 Author: Eric Dumazet eric.duma...@gmail.com Date: Tue Apr 27 15:13:20 2010 -0700 net: sk_add_backlog() take rmem_alloc into account Current socket backlog limit is not enough to really stop DDOS attacks, because user thread spend many time to process a full backlog each round, and user might crazy spin on socket lock. We should add backlog size and receive_queue size (aka rmem_alloc) to pace writers, and let user run without being slow down too much. Introduce a sk_rcvqueues_full() helper, to avoid taking socket lock in stress situations. Under huge stress from a multiqueue/RPS enabled NIC, a single flow udp receiver can now process ~200.000 pps (instead of ~100 pps before the patch) on a 8 core machine. Signed-off-by: Eric Dumazet eric.duma...@gmail.com Signed-off-by: David S. Miller da...@davemloft.net commit 6cce09f87a04797fae5b947ef2626c14a78f0b49 Author: Eric Dumazet eric.duma...@gmail.com Date: Sun Mar 7 23:21:57 2010 + tcp: Add SNMP counters for backlog and min_ttl drops Commit 6b03a53a (tcp: use limited socket backlog) added the possibility of dropping frames when backlog queue is full. Commit d218d111 (tcp: Generalized TTL Security Mechanism) added the possibility of dropping frames when TTL is under a given limit. This patch adds new SNMP MIB entries, named TCPBacklogDrop and TCPMinTTLDrop, published in /proc/net/netstat in TcpExt: line netstat -s | egrep TCPBacklogDrop|TCPMinTTLDrop TCPBacklogDrop: 0 TCPMinTTLDrop: 0 Signed-off-by: Eric Dumazet eric.duma...@gmail.com Signed-off-by: David S. Miller da...@davemloft.net commit 4045635318538d3ddd2007720412fdc4b08f6a62 Author: Zhu Yi yi@intel.com Date: Sun Mar 7 16:21:39 2010 + net: add __must_check to sk_add_backlog Add the __must_check tag to sk_add_backlog() so that any failure to check and drop packets will be warned about. Signed-off-by: Zhu Yi yi@intel.com Signed-off-by: David S. Miller da...@davemloft.net commit b1faf5666438090a4dc4fceac8502edc7788b7e3 Author: Eric Dumazet eric.duma...@gmail.com Date: Mon May 31 23:44:05 2010 -0700 net: sock_queue_err_skb() dont mess with sk_forward_alloc Correct sk_forward_alloc handling for error_queue would need to use a backlog of frames that softirq handler could not deliver because socket is owned by user thread. Or extend backlog processing to be able to process normal and error packets. Another possibility is to not use mem charge for error queue, this is what I implemented in this patch. Note: this reverts commit 29030374 (net: fix sk_forward_alloc corruptions), since we dont need to lock socket anymore. Signed-off-by: Eric Dumazet eric.duma...@gmail.com Signed-off-by: David S. Miller da...@davemloft.net commit dee42870a423ad485129f43cddfe7275479f11d8 Author: Changli Gao xiao...@gmail.com Date: Sun May 2 05:42:16 2010 + net: fix softnet_stat Per cpu variable softnet_data.total was shared between IRQ and SoftIRQ context without any protection. And enqueue_to_backlog should update the netdev_rx_stat of the target CPU. This patch renames softnet_data.total to softnet_data.processed: the number of packets processed in uppper levels(IP stacks). softnet_stat data is moved into softnet_data. Signed-off-by: Changli Gao xiao...@gmail.com include/linux/netdevice.h | 17 +++-- net/core/dev.c| 26 -- net/sched/sch_generic.c |2 +- 3 files changed, 20 insertions(+), 25 deletions(-) Signed-off-by: Eric Dumazet eric.duma...@gmail.com Signed-off-by: David S. Miller da...@davemloft.net commit 4b0b72f7dd617b13abd1b04c947e15873e011a24 Author: Eric Dumazet eric.duma...@gmail.com Date: Wed Apr 28 14:35:48 2010 -0700 net: speedup udp receive path Since commit 95766fff ([UDP]: Add memory accounting.), each received packet needs one extra sock_lock()/sock_release() pair. This added latency because of possible backlog handling. Then later, ticket spinlocks added yet another latency source in case of DDOS. This patch introduces lock_sock_bh() and unlock_sock_bh() synchronization primitives, avoiding one atomic operation and backlog processing. skb_free_datagram_locked() uses them instead of full blown
Bug#592187: [stable] Bug#576838: virtio network crashes again
On Tue, Aug 31, 2010 at 10:16:56AM +0200, Lukas Kolbe wrote: Am Montag, den 30.08.2010, 10:21 -0700 schrieb Greg KH: On Mon, Aug 30, 2010 at 09:46:36AM -0700, David Miller wrote: From: Greg KH g...@kroah.com Date: Mon, 30 Aug 2010 07:50:17 -0700 As I stated above, I need the ACK from David to be able to add these patches. David? I believe there were some regressions caused by these changes that were fixed later, a bit after those commites went into the tree. I'm only confortable ACK'ing this if someone does due diligence and checks for any such follow-on fixes to that series. It's a pretty non-trivial set of patches and has the potential to kill performance which would be a very serious regression. Fair enough. Yep, thanks! Who's done the checks to find out any problems with these patches? I'll skim the changelogs in 2.6.3[345].x to see if there are any related patches. And what is keeping you from moving to the .35 kernel tree instead? Basically, distribution support. Debian Squeeze will ship with 2.6.32, as Ubuntu already did for their current LTS - and I really want Debian's kernel to be as reliable and stable as possible (btw, that's why I initally reported this as a debian bug, because at that time I wasn't using vanilla kernels. Now that I know how git bisect works, it will hopefully be easier for me to pinpoint regressions in the future). Also, We do not really have enough hardware to test new upstream releases thouroughly before going into production (e.g., we only have one big tape library with one big disk pool, so no test if tape-on-mptsas and aacraid work properly and stable in a newer upstream releases. Then how about convincing the Debian kernel developers to accept these patches, and work through any regressions that might be found and after that, reporting back to us? thanks, greg k-h -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100831133518.ga2...@kroah.com
Bug#592187: [stable] Bug#576838: virtio network crashes again
On Tue, 2010-08-31 at 06:35 -0700, Greg KH wrote: On Tue, Aug 31, 2010 at 10:16:56AM +0200, Lukas Kolbe wrote: Am Montag, den 30.08.2010, 10:21 -0700 schrieb Greg KH: On Mon, Aug 30, 2010 at 09:46:36AM -0700, David Miller wrote: From: Greg KH g...@kroah.com Date: Mon, 30 Aug 2010 07:50:17 -0700 As I stated above, I need the ACK from David to be able to add these patches. David? I believe there were some regressions caused by these changes that were fixed later, a bit after those commites went into the tree. I'm only confortable ACK'ing this if someone does due diligence and checks for any such follow-on fixes to that series. It's a pretty non-trivial set of patches and has the potential to kill performance which would be a very serious regression. Fair enough. Yep, thanks! Who's done the checks to find out any problems with these patches? I'll skim the changelogs in 2.6.3[345].x to see if there are any related patches. And what is keeping you from moving to the .35 kernel tree instead? Basically, distribution support. Debian Squeeze will ship with 2.6.32, as Ubuntu already did for their current LTS - and I really want Debian's kernel to be as reliable and stable as possible (btw, that's why I initally reported this as a debian bug, because at that time I wasn't using vanilla kernels. Now that I know how git bisect works, it will hopefully be easier for me to pinpoint regressions in the future). Also, We do not really have enough hardware to test new upstream releases thouroughly before going into production (e.g., we only have one big tape library with one big disk pool, so no test if tape-on-mptsas and aacraid work properly and stable in a newer upstream releases. Then how about convincing the Debian kernel developers to accept these patches, and work through any regressions that might be found and after that, reporting back to us? Ben? The reason I contacted you was precisely because it went into 2.6.33.2, e.g. was already accepted into a -stalbe release. I didn't expect it to be such an issue. thanks, greg k-h Regards, Lukas -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1283268890.5722.75.ca...@quux.techfak.uni-bielefeld.de
Bug#592187: [stable] Bug#576838: virtio network crashes again
On Tue, 2010-08-31 at 17:34 +0200, Lukas Kolbe wrote:
On Tue, 2010-08-31 at 06:35 -0700, Greg KH wrote:
[...]
Then how about convincing the Debian kernel developers to accept these
patches, and work through any regressions that might be found and after
that, reporting back to us?
Ben?
The reason I contacted you was precisely because it went into 2.6.33.2,
e.g. was already accepted into a -stalbe release. I didn't expect it to
be such an issue.
That's not likely if people spread FUD about the backlog patches!
Dave, did you explicitly exclude these patches from 2.6.32 when you
submitted them to stable, or is it just that 5534979 udp: use limited
socket backlog depends on a1ab77f ipv6: udp: Optimise multicast
reception? The former patch doesn't look too hard to backport to
2.6.32 (see below).
Ben.
From: Zhu Yi yi@intel.com
Date: Thu, 4 Mar 2010 18:01:42 +
Subject: [PATCH] udp: use limited socket backlog
[ Upstream commit 55349790d7cbf0d381873a7ece1dcafcffd4aaa9 ]
Make udp adapt to the limited socket backlog change.
Cc: David S. Miller da...@davemloft.net
Cc: Alexey Kuznetsov kuz...@ms2.inr.ac.ru
Cc: Pekka Savola (ipv6) pek...@netcore.fi
Cc: Patrick McHardy ka...@trash.net
Signed-off-by: Zhu Yi yi@intel.com
Acked-by: Eric Dumazet eric.duma...@gmail.com
Signed-off-by: David S. Miller da...@davemloft.net
Signed-off-by: Greg Kroah-Hartman gre...@suse.de
[bwh: Backport to 2.6.32]
---
net/ipv4/udp.c |6 --
net/ipv6/udp.c | 20 ++--
2 files changed, 18 insertions(+), 8 deletions(-)
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index c322f44..0ea57b1 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -1174,8 +1174,10 @@ int udp_queue_rcv_skb(struct sock *sk, struct sk_buff
*skb)
bh_lock_sock(sk);
if (!sock_owned_by_user(sk))
rc = __udp_queue_rcv_skb(sk, skb);
- else
- sk_add_backlog(sk, skb);
+ else if (sk_add_backlog_limited(sk, skb)) {
+ bh_unlock_sock(sk);
+ goto drop;
+ }
bh_unlock_sock(sk);
return rc;
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index cf538ed..154dd6b 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -470,16 +470,20 @@ static int __udp6_lib_mcast_deliver(struct net *net,
struct sk_buff *skb,
bh_lock_sock(sk2);
if (!sock_owned_by_user(sk2))
udpv6_queue_rcv_skb(sk2, buff);
- else
- sk_add_backlog(sk2, buff);
+ else if (sk_add_backlog_limited(sk2, buff)) {
+ atomic_inc(sk2-sk_drops);
+ kfree_skb(buff);
+ }
bh_unlock_sock(sk2);
}
}
bh_lock_sock(sk);
if (!sock_owned_by_user(sk))
udpv6_queue_rcv_skb(sk, skb);
- else
- sk_add_backlog(sk, skb);
+ else if (sk_add_backlog_limited(sk, skb)) {
+ atomic_inc(sk-sk_drops);
+ kfree_skb(skb);
+ }
bh_unlock_sock(sk);
out:
spin_unlock(hslot-lock);
@@ -598,8 +602,12 @@ int __udp6_lib_rcv(struct sk_buff *skb, struct udp_table
*udptable,
bh_lock_sock(sk);
if (!sock_owned_by_user(sk))
udpv6_queue_rcv_skb(sk, skb);
- else
- sk_add_backlog(sk, skb);
+ else if (sk_add_backlog_limited(sk, skb)) {
+ atomic_inc(sk-sk_drops);
+ bh_unlock_sock(sk);
+ sock_put(sk);
+ goto discard;
+ }
bh_unlock_sock(sk);
sock_put(sk);
return 0;
--
1.7.1
--
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
signature.asc
Description: This is a digitally signed message part
Bug#592187: [stable] Bug#576838: virtio network crashes again
On Thu, 2010-08-26 at 09:32 +0200, Lukas Kolbe wrote: Hi, I was finally able to identify the patch series that introduced the fix (they were introduced to -stable in 2.6.33.2): cb63112 net: add __must_check to sk_add_backlog a12a9a2 net: backlog functions rename 51c5db4 x25: use limited socket backlog c531ab2 tipc: use limited socket backlog 37d60aa sctp: use limited socket backlog 9b3d968 llc: use limited socket backlog 230401e udp: use limited socket backlog 20a92ec tcp: use limited socket backlog ab9dd05 net: add limit for socket backlog After applying these to 2.6.32.17, I wasn't able to trigger the failure anymore. What failure? 230401e didn't apply cleanly with git cherry-pick on top of 2.6.32.17, so there might be some additional work needed. @Greg: would it be possible to have these fixes in the next 2.6.32? See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592187#69 for details: they fix a guest network crash during heavy nfs-io using virtio. These are a lot of patches, looking like they are adding a new feature. I would need to get the ack of the network maintainer before I can add them. David? I don't mean to nag (hm well, maybe I do) and I know you were busy preparing the guard-page fixes, but what's the status of this? In the meantime, we triggered this bug also on barebone hardware using nfs over tcp with default [rw]sizes of about 1MiB. On the real hardware, the kernel oopsed, not only the network stack ... With these patches applied, everything works smoothly. I'd really love to see a stable 2.6.32 ... Is there anything I can do to help reaching a decision with this issue? Regards, Lukas -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1283176797.5722.13.ca...@quux.techfak.uni-bielefeld.de
Bug#592187: [stable] Bug#576838: virtio network crashes again
On Mon, Aug 30, 2010 at 03:59:57PM +0200, Lukas Kolbe wrote: On Thu, 2010-08-26 at 09:32 +0200, Lukas Kolbe wrote: Hi, I was finally able to identify the patch series that introduced the fix (they were introduced to -stable in 2.6.33.2): cb63112 net: add __must_check to sk_add_backlog a12a9a2 net: backlog functions rename 51c5db4 x25: use limited socket backlog c531ab2 tipc: use limited socket backlog 37d60aa sctp: use limited socket backlog 9b3d968 llc: use limited socket backlog 230401e udp: use limited socket backlog 20a92ec tcp: use limited socket backlog ab9dd05 net: add limit for socket backlog After applying these to 2.6.32.17, I wasn't able to trigger the failure anymore. What failure? 230401e didn't apply cleanly with git cherry-pick on top of 2.6.32.17, so there might be some additional work needed. @Greg: would it be possible to have these fixes in the next 2.6.32? See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592187#69 for details: they fix a guest network crash during heavy nfs-io using virtio. These are a lot of patches, looking like they are adding a new feature. I would need to get the ack of the network maintainer before I can add them. David? I don't mean to nag (hm well, maybe I do) and I know you were busy preparing the guard-page fixes, but what's the status of this? In the meantime, we triggered this bug also on barebone hardware using nfs over tcp with default [rw]sizes of about 1MiB. On the real hardware, the kernel oopsed, not only the network stack ... With these patches applied, everything works smoothly. I'd really love to see a stable 2.6.32 ... Is there anything I can do to help reaching a decision with this issue? As I stated above, I need the ACK from David to be able to add these patches. David? thanks, greg k-h -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100830145017.ga8...@kroah.com
Bug#592187: [stable] Bug#576838: virtio network crashes again
On Mon, Aug 30, 2010 at 09:46:36AM -0700, David Miller wrote: From: Greg KH g...@kroah.com Date: Mon, 30 Aug 2010 07:50:17 -0700 As I stated above, I need the ACK from David to be able to add these patches. David? I believe there were some regressions caused by these changes that were fixed later, a bit after those commites went into the tree. I'm only confortable ACK'ing this if someone does due diligence and checks for any such follow-on fixes to that series. It's a pretty non-trivial set of patches and has the potential to kill performance which would be a very serious regression. Fair enough. Who's done the checks to find out any problems with these patches? And what is keeping you from moving to the .35 kernel tree instead? thanks, greg k-h -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100830172107.ga10...@kroah.com
Bug#592187: [stable] Bug#576838: virtio network crashes again
Hi all, I was finally able to identify the patch series that introduced the fix (they were introduced to -stable in 2.6.33.2): cb63112 net: add __must_check to sk_add_backlog a12a9a2 net: backlog functions rename 51c5db4 x25: use limited socket backlog c531ab2 tipc: use limited socket backlog 37d60aa sctp: use limited socket backlog 9b3d968 llc: use limited socket backlog 230401e udp: use limited socket backlog 20a92ec tcp: use limited socket backlog ab9dd05 net: add limit for socket backlog After applying these to 2.6.32.17, I wasn't able to trigger the failure anymore. What failure? 230401e didn't apply cleanly with git cherry-pick on top of 2.6.32.17, so there might be some additional work needed. @Greg: would it be possible to have these fixes in the next 2.6.32? See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592187#69 for details: they fix a guest network crash during heavy nfs-io using virtio. These are a lot of patches, looking like they are adding a new feature. I would need to get the ack of the network maintainer before I can add them. David? I don't mean to nag (hm well, maybe I do) and I know you were busy preparing the guard-page fixes, but what's the status of this? In the meantime, we triggered this bug also on barebone hardware using nfs over tcp with default [rw]sizes of about 1MiB. On the real hardware, the kernel oopsed, not only the network stack ... With these patches applied, everything works smoothly. I'd really love to see a stable 2.6.32 ... thanks, greg k-h Regards, Lukas Kolbe -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1282807979.16456.10.ca...@larosa.fritz.box
Bug#592187: [stable] Bug#576838: virtio network crashes again
Hi all, I was finally able to identify the patch series that introduced the fix (they were introduced to -stable in 2.6.33.2): cb63112 net: add __must_check to sk_add_backlog a12a9a2 net: backlog functions rename 51c5db4 x25: use limited socket backlog c531ab2 tipc: use limited socket backlog 37d60aa sctp: use limited socket backlog 9b3d968 llc: use limited socket backlog 230401e udp: use limited socket backlog 20a92ec tcp: use limited socket backlog ab9dd05 net: add limit for socket backlog After applying these to 2.6.32.17, I wasn't able to trigger the failure anymore. What failure? From my other mail, for public reference: With 2.6.32.17 as a KVM guest using virtio_net, large nfs reads and writes cause the network to crash. Only rmmod virtio_net/modprobe virtio_net fixes it. I found that this bug was fixed in 2.6.33.2, and git bisect pointed me to the following patch series, which, when applied to 2.6.32.17, fixes the problem: I have to add that this also happens to bare-bone systems on real hardware - we just had a machine crash during it's nightly nfs backup with a slew of page allocation failures. 230401e didn't apply cleanly with git cherry-pick on top of 2.6.32.17, so there might be some additional work needed. @Greg: would it be possible to have these fixes in the next 2.6.32? See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592187#69 for details: they fix a guest network crash during heavy nfs-io using virtio. These are a lot of patches, looking like they are adding a new feature. I would need to get the ack of the network maintainer before I can add them. David? fyi, the comment of the original patch series applied to 2.6.33. It's not a new feature per se, but a fix to a general problem. Author: Zhu Yi yi@intel.com Date: Thu Mar 4 18:01:40 2010 + net: add limit for socket backlog [ Upstream commit 8eae939f1400326b06d0c9afe53d2a484a326871 ] We got system OOM while running some UDP netperf testing on the loopback device. The case is multiple senders sent stream UDP packets to a single receiver via loopback on local host. Of course, the receiver is not able to handle all the packets in time. But we surprisingly found that these packets were not discarded due to the receiver's sk-sk_rcvbuf limit. Instead, they are kept queuing to sk-sk_backlog and finally ate up all the memory. We believe this is a secure hole that a none privileged user can crash the system. The root cause for this problem is, when the receiver is doing __release_sock() (i.e. after userspace recv, kernel udp_recvmsg - skb_free_datagram_locked - release_sock), it moves skbs from backlog to sk_receive_queue with the softirq enabled. In the above case, multiple busy senders will almost make it an endless loop. The skbs in the backlog end up eat all the system memory. The issue is not only for UDP. Any protocols using socket backlog is potentially affected. The patch adds limit for socket backlog so that the backlog size cannot be expanded endlessly. Reported-by: Alex Shi alex@intel.com Cc: David Miller da...@davemloft.net Cc: Arnaldo Carvalho de Melo a...@ghostprotocols.net Cc: Alexey Kuznetsov kuz...@ms2.inr.ac.ru Cc: Pekka Savola (ipv6) pek...@netcore.fi Cc: Patrick McHardy ka...@trash.net Cc: Vlad Yasevich vladislav.yasev...@hp.com Cc: Sridhar Samudrala s...@us.ibm.com Cc: Jon Maloy jon.ma...@ericsson.com Cc: Allan Stephens allan.steph...@windriver.com Cc: Andrew Hendry andrew.hen...@gmail.com Signed-off-by: Zhu Yi yi@intel.com Signed-off-by: Eric Dumazet eric.duma...@gmail.com Acked-by: Arnaldo Carvalho de Melo a...@redhat.com Signed-off-by: David S. Miller da...@davemloft.net Signed-off-by: Greg Kroah-Hartman gre...@suse.de thanks, greg k-h Regards, Lukas -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1282310856.8635.7.ca...@quux.techfak.uni-bielefeld.de
Bug#592187: [stable] Bug#576838: virtio network crashes again
On Sun, Aug 15, 2010 at 09:37:34AM +0200, Lukas Kolbe wrote: Hi Ben, Greg, I was finally able to identify the patch series that introduced the fix (they were introduced to -stable in 2.6.33.2): cb63112 net: add __must_check to sk_add_backlog a12a9a2 net: backlog functions rename 51c5db4 x25: use limited socket backlog c531ab2 tipc: use limited socket backlog 37d60aa sctp: use limited socket backlog 9b3d968 llc: use limited socket backlog 230401e udp: use limited socket backlog 20a92ec tcp: use limited socket backlog ab9dd05 net: add limit for socket backlog After applying these to 2.6.32.17, I wasn't able to trigger the failure anymore. What failure? 230401e didn't apply cleanly with git cherry-pick on top of 2.6.32.17, so there might be some additional work needed. @Greg: would it be possible to have these fixes in the next 2.6.32? See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=592187#69 for details: they fix a guest network crash during heavy nfs-io using virtio. These are a lot of patches, looking like they are adding a new feature. I would need to get the ack of the network maintainer before I can add them. David? thanks, greg k-h -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100819142808.ga13...@kroah.com
