On Tue, Jan 05, 2010 at 02:05:03AM -0800, Jack Z wrote:
> Hi Pasi,
> 
> Thank you very much for your reply.
> 
> > > I was testing the performance of open-iscsi initiator with IET target
> > > over a 100Mbps Ethernet link with emulated rtt.  What I did was to do
> > > raw disk sequential write by
> >
> > > $ dd if=/dev/zero of=/dev/sdb bs=1024 count=1048576
> >
> > Did you also try with bigger block sizes? 1k blocks are pretty small.
> >
> > try bs=1024k to see if it makes a difference.
> 
> 
> I tried bs = 1024k and the throughput is improved, but not much... It
> goes from 7.2MB/s to 8.0MB/s at a rtt of 16ms. And again, over 90% of
> the TCP segments on the wire was only of 1448 bytes...
>

Ok..


> 
> > dd will use only one outstanding IO, so you have wait for rtt
> > milliseconds after every IO for the ack.. so that definitely slows you
> > down a lot when rtt gets bigger.
> >
> > Try using some benchmarking tool that can do multiple outstanding IOs..
> > for example ltp disktest.
> >
> 
> And I tried ltp disktest, too. But I'm not sure whether I used it
> right because the result was a little surprising...
> 
> I did
> 
> disktest -w -S0:1k -B 1024 /dev/sdb
> 
> (/dev/sdb is the iSCSI device file, no partition or file system on it)
> 
> And the result was:
> 
> | 2010/01/05-02:58:26 | START | 27293 | v1.4.2 | /dev/sdb | Start
> args: -w -S0:1024k -B 1024 -PA (-I b) (-N 8385867) (-K 4) (-c) (-p R)
> (-L 1048577) (-D 0:100) (-t 0:2m) (-o 0)
> | 2010/01/05-02:58:26 | INFO  | 27293 | v1.4.2 | /dev/sdb | Starting
> pass
> ^C| 2010/01/05-03:00:58 | STAT  | 27293 | v1.4.2 | /dev/sdb | Total
> bytes written in 85578 transfers: 87631872
> | 2010/01/05-03:00:58 | STAT  | 27293 | v1.4.2 | /dev/sdb | Total
> write throughput: 701055.0B/s (0.67MB/s), IOPS 684.6/s.
> | 2010/01/05-03:00:58 | STAT  | 27293 | v1.4.2 | /dev/sdb | Total
> Write Time: 125 seconds (0d0h2m5s)
> | 2010/01/05-03:00:58 | STAT  | 27293 | v1.4.2 | /dev/sdb | Total
> overall runtime: 152 seconds (0d0h2m32s)
> | 2010/01/05-03:00:58 | END   | 27293 | v1.4.2 | /dev/sdb | User
> Interrupt: Test Done (Passed)
> 
> As you can see, the throughput was only 0.67MB/s and only 85578
> written in 87631872 transfers...
> I also tweaked the options with "-p l" and/or "-I bd" (change seek
> pattern to linear and/or speficy IO type as block and direct IO) but
> no improvement happened...
> 

Hmm.. so it does 684 IO operations per second (IOPS), and each IO was 1k
in size, so it makes 684 kB/sec of throughput.

1000 milliseconds (1 second) divided by 684 IOPS is 1.46 milliseconds per IO..

Are you sure you had 16ms of rtt? 

> There must be something I've done wrong... Could you maybe help me out
> here?
> 
> Thanks a lot!
> 

Try to play and experiment with these options:

-B 64k  (blocksize 64k, try also 4k)
-I BD (block device, direct IO (O_DIRECT))
-K 16 (16 threads, aka 16 outstanding IOs. -K 1 should be the same as dd)

Examples:

Sequential (linear) reads using blocksize 4k and 4 simultaneous threads, for 60 
seconds:
disktest -B 4k -h 1 -I BD -K 4 -p l -P T -T 60 -r /dev/sdX

Random writes:
disktest -B 4k -h 1 -I BD -K 4 -p r -P T -T 60 -w /dev/sdX

30% random reads, 70% random writes:
disktest -r -w -D30:70 -K2 -E32 -B 8k -T 60 -pR -Ibd -PA /dev/md4

Hopefully that helps..

-- Pasi

-- 
You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To post to this group, send email to open-is...@googlegroups.com.
To unsubscribe from this group, send email to 
open-iscsi+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/open-iscsi?hl=en.


Reply via email to