Re: TSO and IPoIB performance degradation

2006-04-26 Thread Troy Benjegerdes
On Mon, Mar 20, 2006 at 02:37:04AM -0800, David S. Miller wrote: From: Michael S. Tsirkin [EMAIL PROTECTED] Date: Mon, 20 Mar 2006 12:22:34 +0200 Quoting r. David S. Miller [EMAIL PROTECTED]: The path an SKB can take is opaque and unknown until the very last moment it is actually given

Re: Stretch ACKs, etc. (was TSO and IPoIB performance degradation)

2006-03-24 Thread Mark Butler
Wouldn't the appropriate place to add the tunable for Stretch ACKs be as a route attribute similar to RTAX_ADVMSS? Then system administrators who are aware of the local network topology, netfilters, etc, could use an ip route or whatever command to enable it on the route entry for the

Re: TSO and IPoIB performance degradation

2006-03-20 Thread Michael S. Tsirkin
Quoting r. David S. Miller [EMAIL PROTECTED]: well, there are stacks which do stretch acks (after a fashion) that make sure when they see packet loss to do the right thing wrt sending enough acks to allow cwnds to open again in a timely fashion. Once a loss happens, it's too late to

Re: TSO and IPoIB performance degradation

2006-03-20 Thread David S. Miller
From: Michael S. Tsirkin [EMAIL PROTECTED] Date: Mon, 20 Mar 2006 11:06:29 +0200 Is it the case then that this requirement is less essential on networks such as IP over InfiniBand, which are very low latency and essencially lossless (with explicit congestion contifications in hardware)? You

Re: TSO and IPoIB performance degradation

2006-03-20 Thread Michael S. Tsirkin
Quoting r. David S. Miller [EMAIL PROTECTED]: The path an SKB can take is opaque and unknown until the very last moment it is actually given to the device transmit function. Why, I was proposing looking at dst cache. If that's NULL, well, we won't stretch ACKs. Worst case we apply the wrong

Re: TSO and IPoIB performance degradation

2006-03-20 Thread David S. Miller
From: Michael S. Tsirkin [EMAIL PROTECTED] Date: Mon, 20 Mar 2006 12:22:34 +0200 Quoting r. David S. Miller [EMAIL PROTECTED]: The path an SKB can take is opaque and unknown until the very last moment it is actually given to the device transmit function. Why, I was proposing looking at

Re: TSO and IPoIB performance degradation

2006-03-20 Thread Lennert Buytenhek
On Mon, Mar 20, 2006 at 12:47:03PM +0100, Arjan van de Ven wrote: I disagree with Linux changing it's behavior. It would be great to turn off congestion control completely over local gigabit networks, but that isn't determinable in any way, so we don't do that. Interesting. Would

Re: TSO and IPoIB performance degradation

2006-03-20 Thread Arjan van de Ven
On Mon, 2006-03-20 at 12:49 +0100, Lennert Buytenhek wrote: On Mon, Mar 20, 2006 at 12:47:03PM +0100, Arjan van de Ven wrote: I disagree with Linux changing it's behavior. It would be great to turn off congestion control completely over local gigabit networks, but that isn't

Re: TSO and IPoIB performance degradation

2006-03-20 Thread Michael S. Tsirkin
Quoting r. Lennert Buytenhek [EMAIL PROTECTED]: I disagree with Linux changing it's behavior. It would be great to turn off congestion control completely over local gigabit networks, but that isn't determinable in any way, so we don't do that. Interesting. Would it make sense

Re: TSO and IPoIB performance degradation

2006-03-20 Thread Michael S. Tsirkin
Quoting Arjan van de Ven [EMAIL PROTECTED]: I read it as if he was proposing to have a sysctl knob to turn off TCP congestion control completely (which has so many issues it's not even funny.) owww that's so bad I didn't even consider that No, I think that comment was taken out of thread

Re: TSO and IPoIB performance degradation

2006-03-20 Thread Benjamin LaHaise
On Mon, Mar 20, 2006 at 02:04:07PM +0200, Michael S. Tsirkin wrote: does not stretch ACKs anymore. RFC 2581 does mention that it might be OK to stretch ACKs after careful consideration, and we are seeing that it helps IP over InfiniBand, so recent Linux kernels perform worse in that respect.

Re: TSO and IPoIB performance degradation

2006-03-20 Thread David S. Miller
From: Benjamin LaHaise [EMAIL PROTECTED] Date: Mon, 20 Mar 2006 10:09:42 -0500 Wouldn't it make sense to strech the ACK when the previous ACK is still in the TX queue of the device? I know that sort of behaviour was always an issue on modem links where you don't want to send out redundant

Re: TSO and IPoIB performance degradation

2006-03-10 Thread Rick Jones
David S. Miller wrote: From: Rick Jones [EMAIL PROTECTED] Date: Thu, 09 Mar 2006 16:21:05 -0800 well, there are stacks which do stretch acks (after a fashion) that make sure when they see packet loss to do the right thing wrt sending enough acks to allow cwnds to open again in a timely

Re: TSO and IPoIB performance degradation

2006-03-09 Thread David S. Miller
From: Michael S. Tsirkin [EMAIL PROTECTED] Date: Wed, 8 Mar 2006 14:53:11 +0200 What I was trying to figure out was, how can we re-enable the trick without hurting TSO? Could a solution be to simply look at the frame size, and call tcp_send_delayed_ack if the frame size is small? The change

Re: TSO and IPoIB performance degradation

2006-03-09 Thread Michael S. Tsirkin
Quoting David S. Miller [EMAIL PROTECTED]: Description To improve efficiency (both computer and network) a data receiver may refrain from sending an ACK for each incoming segment, according to [RFC1122]. However, an ACK should not be delayed an inordinate amount of

Re: TSO and IPoIB performance degradation

2006-03-09 Thread Rick Jones
David S. Miller wrote: From: Michael S. Tsirkin [EMAIL PROTECTED] Date: Wed, 8 Mar 2006 14:53:11 +0200 What I was trying to figure out was, how can we re-enable the trick without hurting TSO? Could a solution be to simply look at the frame size, and call tcp_send_delayed_ack if the frame size

Re: TSO and IPoIB performance degradation

2006-03-09 Thread Michael S. Tsirkin
Quoting r. Michael S. Tsirkin [EMAIL PROTECTED]: Or does __tcp_ack_snd_check delay until we have at least two full sized segments? What I'm trying to say, since RFC 2525, 2.13 talks about every second full-sized segment, so following the code from __tcp_ack_snd_check, why does it do

Re: TSO and IPoIB performance degradation

2006-03-09 Thread David S. Miller
From: Michael S. Tsirkin [EMAIL PROTECTED] Date: Fri, 10 Mar 2006 02:10:31 +0200 But with the change we are discussing, could an ack now be sent even sooner than we have at least two full sized segments? Or does __tcp_ack_snd_check delay until we have at least two full sized segments? David,

Re: TSO and IPoIB performance degradation

2006-03-09 Thread David S. Miller
From: Rick Jones [EMAIL PROTECTED] Date: Thu, 09 Mar 2006 16:21:05 -0800 well, there are stacks which do stretch acks (after a fashion) that make sure when they see packet loss to do the right thing wrt sending enough acks to allow cwnds to open again in a timely fashion. Once a loss

Re: Re: TSO and IPoIB performance degradation

2006-03-08 Thread Michael S. Tsirkin
Quoting r. David S. Miller [EMAIL PROTECTED]: Subject: Re: Re: TSO and IPoIB performance degradation From: Roland Dreier [EMAIL PROTECTED] Date: Tue, 07 Mar 2006 17:17:30 -0800 The reason TSO comes up is that reverting the patch described below helps (or helped at some point at least

Re: TSO and IPoIB performance degradation

2006-03-08 Thread David S. Miller
From: Michael S. Tsirkin [EMAIL PROTECTED] Date: Wed, 8 Mar 2006 14:53:11 +0200 What I was trying to figure out was, how can we re-enable the trick without hurting TSO? Could a solution be to simply look at the frame size, and call tcp_send_delayed_ack if the frame size is small? The problem

Re: [openib-general] Re: TSO and IPoIB performance degradation

2006-03-07 Thread Matt Leininger
On Mon, 2006-03-06 at 19:13 -0800, Shirley Ma wrote: More likely you are getting hit by the fact that TSO prevents the congestion window from increasing properly. This was fixed in 2.6.15 (around mid of Nov 2005). Yep, I noticed the same problem. After updating to the new kernel, the

Re: [openib-general] Re: TSO and IPoIB performance degradation

2006-03-07 Thread Matt Leininger
On Tue, 2006-03-07 at 13:49 -0800, Stephen Hemminger wrote: On Tue, 07 Mar 2006 13:44:51 -0800 Matt Leininger [EMAIL PROTECTED] wrote: On Mon, 2006-03-06 at 19:13 -0800, Shirley Ma wrote: More likely you are getting hit by the fact that TSO prevents the congestion window from

Re: [openib-general] Re: TSO and IPoIB performance degradation

2006-03-07 Thread David S. Miller
From: Matt Leininger [EMAIL PROTECTED] Date: Tue, 07 Mar 2006 16:11:37 -0800 I used the standard setting for tcp_rmem and tcp_wmem. Here are a few other runs that change those variables. I was able to improve performance by ~30MB/s to 403 MB/s, but this is still a ways from the 474 MB/s

Re: [openib-general] Re: TSO and IPoIB performance degradation

2006-03-07 Thread David S. Miller
From: Roland Dreier [EMAIL PROTECTED] Date: Tue, 07 Mar 2006 17:17:30 -0800 The reason TSO comes up is that reverting the patch described below helps (or helped at some point at least) IPoIB throughput quite a bit. I wish you had started the thread by mentioning this specific patch, we wasted

Re: [openib-general] Re: TSO and IPoIB performance degradation

2006-03-07 Thread Roland Dreier
David How limited are the IPoIB devices, TX descriptor wise? David One side effect of the TSO changes is that one extra David descriptor will be used for outgoing packets. This is David because we have to put the headers as well as the user David data, into page based buffers

Re: [openib-general] Re: TSO and IPoIB performance degradation

2006-03-07 Thread Roland Dreier
David I wish you had started the thread by mentioning this David specific patch, we wasted an enormous amount of precious David developer time speculating and asking for arbitrary tests David to be run in order to narrow down the problem, yet you knew David the specific change

TSO and IPoIB performance degradation

2006-03-06 Thread Michael S. Tsirkin
Hello, Dave! As you might know, the TSO patches merged into mainline kernel since 2.6.11 have hurt performance for the simple (non-TSO) high-speed netdevice that is IPoIB driver. This was discussed at length here http://openib.org/pipermail/openib-general/2005-October/012271.html I'm trying to

Re: TSO and IPoIB performance degradation

2006-03-06 Thread David S. Miller
From: Michael S. Tsirkin [EMAIL PROTECTED] Date: Tue, 7 Mar 2006 00:34:38 +0200 So I'm trying to get a handle on it: could a solution be to simply look at the frame size, and call tcp_send_delayed_ack from if the frame size is no larger than 1/8? Does this make sense? The comment you

Re: TSO and IPoIB performance degradation

2006-03-06 Thread Stephen Hemminger
On Tue, 7 Mar 2006 00:34:38 +0200 Michael S. Tsirkin [EMAIL PROTECTED] wrote: Hello, Dave! As you might know, the TSO patches merged into mainline kernel since 2.6.11 have hurt performance for the simple (non-TSO) high-speed netdevice that is IPoIB driver. This was discussed at length here