On Wednesday 14 May 2008 15:16:00 Olaf Kirch wrote: > I'll let you know as soon as I have something for you to test.
Okay, here we go. I have a whole stack of patches sitting in http://www.openfabrics.org/git/?p=~okir/ofed_1_3/linux-2.6.git on branch future-20080516 This patch stack contains everything I'm working on right now, but which isn't ready for OFED 1.3.1 yet (and I haven't started on 1.4 yet). So you get a lot more than you bargained for... but a fair bit of that is actually needed because it prepares the ground for the flow control stuff, so I didn't bother with ripping out the unneeded pieces and rediffing everything. I did some light testing with the code - it hasn't oopsed in a few hours, but I'm getting occasional errors with rds-stress right after reloading the module. They go away on the next attempt - so there's still something fishy about the code (or about rds-stress). I'm not completely happy with the performance yet. Early versions might have been more adequately dubbed "trickle control" - but I eventually managed to get something not-too-bad. However, I'm still seeing performance degradation of ~5% with some packet sizes. And that is *just* the overhead from exchanging the credit information and checking it - at some point we need to take a spinlock, and that seems to delay things just enough to make a dent in my throughput graph. In fact, I haven't yet found a test case where the sender had to slow down sending because it ran out of credits. Which confirms my suspicion that the current setup isn't so bad, at least for IB... If you're interested in the flow control code, the last commit on that branch is the one to look at - commit header appended below. I'm pretty sure that this is not exactly what iWARP needs, so please send comments/patches on how to beat it into shape for iWARP. Enjoyable weekend to everyone, Olaf commit e8a64b4f83df9df6617f75dff9e591b86174fa7c Author: Olaf Kirch <[EMAIL PROTECTED]> Date: Fri May 16 06:16:40 2008 -0700 RDS: Implement IB flow control Here it is - flow control for RDS/IB. This patch is still very much experimental. Here's the essentials - The approach chosen here uses a credit-based flow control mechanism. Every SEND WR (including ACKs) consumes one credit, and if the sender runs out of credits, it stalls. - As new receive buffers are posted, credits are transferred to the remote node (using yet another RDS header byte for this). - Flow control is negotiated during connection setup. Initial credits are exchanged in the rds_ib_connect_private sruct - sending a value of zero (which is also the default for older protocol versions) means no flow control. - We avoid deadlock (both nodes depleting their credits, and being unable to inform the peer of newly posted buffers) by requiring that the last credit can only be used if we're posting new credits to the peer. Flow control is configurable via sysctl. It only affects newly created connections however - so your best bet is to set this right after loading the RDS module. Signed-off-by: Olaf Kirch <[EMAIL PROTECTED]> -- Olaf Kirch | --- o --- Nous sommes du soleil we love when we play [EMAIL PROTECTED] | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
