Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-11 Thread Vladislav Bolkhovitin
Chris Worley, on 01/09/2010 01:39 AM wrote: I thought if the device was opened with the O_DIRECT flag, then the scheduler should have nothing to coalesce. Depends on how many I/Os your application has in flight at once, assuming it is using AIO or threads. If you have more requests in flight tha

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-09 Thread Bart Van Assche
On Thu, Jan 7, 2010 at 1:16 AM, Chris Worley wrote: > 3) In my iSCSI (tgt) results using the HCA as a 10G interface (not > IPoIB, but mlnx4_en), comparing this to the results of using the same > HCA as IB under SRP, I get much better results with SRP when > benchmarking the raw device, as you'd ex

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-09 Thread David Dillow
On Sat, 2010-01-09 at 18:49 +0100, Bart Van Assche wrote: > On Sat, Jan 9, 2010 at 6:16 PM, David Dillow wrote: > > Does SRPT support the RDMA'ing the indirect buffer descriptors from the > > Initiator such that it isn't constrained by the partial memory > > descriptor list in the command request?

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-09 Thread Bart Van Assche
On Sat, Jan 9, 2010 at 6:16 PM, David Dillow wrote: > > On Sat, 2010-01-09 at 14:05 +0100, Bart Van Assche wrote: > > The SRP spec says that the target must specify the maximum message > > size in the SRP_LOGIN_RSP information unit. The largest value one can > > set the srp_sg_tablesize initiator

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-09 Thread David Dillow
On Sat, 2010-01-09 at 14:05 +0100, Bart Van Assche wrote: > The SRP spec says that the target must specify the maximum message > size in the SRP_LOGIN_RSP information unit. The largest value one can > set the srp_sg_tablesize initiator parameter to is (max. SRP message > size defined by the target

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-09 Thread Bart Van Assche
On Fri, Jan 8, 2010 at 11:39 PM, Chris Worley wrote: > On Fri, Jan 8, 2010 at 3:17 PM, David Dillow wrote: >> On Fri, 2010-01-08 at 14:40 -0700, Chris Worley wrote: >>> On Wed, Jan 6, 2010 at 6:57 PM, David Dillow wrote: >>> > On Wed, 2010-01-06 at 17:16 -0700, Chris Worley wrote: >>> >> 1) I'm

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-08 Thread David Dillow
On Fri, 2010-01-08 at 18:07 -0500, David Dillow wrote: > But this still isn't hurting you at the small request sizes we seem to > be talking about. Or do you mean 58 KB, which is believable -- the > default is 12, which guarantees a 48 KB request size is possible, and > you'd only need a few pages

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-08 Thread David Dillow
On Fri, 2010-01-08 at 15:39 -0700, Chris Worley wrote: > On Fri, Jan 8, 2010 at 3:17 PM, David Dillow wrote: > > On Fri, 2010-01-08 at 14:40 -0700, Chris Worley wrote: > >> I do set the ib_srp initiator "srp_sg_tablesize" to its maximum of 58. > > > > The max is 255, which will guarantee you can s

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-08 Thread Chris Worley
On Fri, Jan 8, 2010 at 3:17 PM, David Dillow wrote: > On Fri, 2010-01-08 at 14:40 -0700, Chris Worley wrote: >> On Wed, Jan 6, 2010 at 6:57 PM, David Dillow wrote: >> > On Wed, 2010-01-06 at 17:16 -0700, Chris Worley wrote: >> >> 1) I'm seeing small block random writes (32KB and smaller) get bett

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-08 Thread David Dillow
On Fri, 2010-01-08 at 14:40 -0700, Chris Worley wrote: > On Wed, Jan 6, 2010 at 6:57 PM, David Dillow wrote: > > On Wed, 2010-01-06 at 17:16 -0700, Chris Worley wrote: > >> 1) I'm seeing small block random writes (32KB and smaller) get better > >> performance over SRP than they do as a local drive

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-08 Thread Chris Worley
On Wed, Jan 6, 2010 at 6:57 PM, David Dillow wrote: > On Wed, 2010-01-06 at 17:16 -0700, Chris Worley wrote: >> 1) I'm seeing small block random writes (32KB and smaller) get better >> performance over SRP than they do as a local drive.  I'm guessing this >> is async behavior: once the written dat

Re: SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-06 Thread David Dillow
On Wed, 2010-01-06 at 17:16 -0700, Chris Worley wrote: > 1) I'm seeing small block random writes (32KB and smaller) get better > performance over SRP than they do as a local drive. I'm guessing this > is async behavior: once the written data is on the wire, it's deemed > complete, and setting a sy

SRP Q's: 1) When is asynchronous I/O complete, 2) Is sequential I/O coalesced, and 3) why is iSCSI faster than SRP in some instances

2010-01-06 Thread Chris Worley
In shifting through a great deal of benchmark data collected from two identical machines (including the attached drive array), I see the following SRP anomalies: 1) I'm seeing small block random writes (32KB and smaller) get better performance over SRP than they do as a local drive. I'm guessing