On Sat, 2010-01-09 at 18:49 +0100, Bart Van Assche wrote: > On Sat, Jan 9, 2010 at 6:16 PM, David Dillow <d...@thedillows.org> wrote: > > Does SRPT support the RDMA'ing the indirect buffer descriptors from the > > Initiator such that it isn't constrained by the partial memory > > descriptor list in the command request? The initiator restricts itself > > to 255 SG entries because I've not found a target that implemented the > > spec fully, though it'd be nice to be able to guarantee the ability to > > send larger request sizes. I think this will also become important for > > running bidirectional commands, since the room for descriptors cached in > > the command is shared for both directions. > > At this time SRPT only supports indirect buffer descriptors that are > present entirely in the command request. Regarding the number of SG > entries: I'm not sure that I understand why you want to be able to > send SG-lists containing more than 255 SG entries. In the tests I have > run the throughput gain resulting from SG list sizes above 128 was > marginal (a few percent).
It depends on the hardware I suppose. It may not make sense for SRPT, but for some of the vendors I deal with, being able to guarantee 1 MB requests on IB is worth quite a bit of performance, and their tests on FC show that larger requests still show enough performance gains to make it worth it. I've done some experiments, and it looks like while IB is less happy with all the 4 KB requests, it still can get enough data to the device to saturate the RAID controller. For the controllers we're using, 1 MB requests perform better than 512 KB requests, and sending a 4 MB random write request stream (highly seek intensive) gets us to ~90% IIRC of the 1 MB pure sequential write performance, without using writeback cache. The controllers have a fair amount of overhead per request, so doing fewer large requests helps. Our workload is often largely sequential, with large volumes of data, but once enough clients start writing, it an become random in a hurry. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
