Hi Pasi, Thank you very much for your help. I really appreciate it!
On Jan 5, 12:58 pm, Pasi Kärkkäinen <pa...@iki.fi> wrote: > On Tue, Jan 05, 2010 at 02:05:03AM -0800, Jack Z wrote: > > > Try using some benchmarking tool that can do multiple outstanding IOs.. > > > for example ltp disktest. > > And I tried ltp disktest, too. But I'm not sure whether I used it > > right because the result was a little surprising... > > I did > > disktest -w -S0:1k -B 1024 /dev/sdb > > (/dev/sdb is the iSCSI device file, no partition or file system on it) > > And the result was: > > | 2010/01/05-02:58:26 | START | 27293 | v1.4.2 | /dev/sdb | Start > > args: -w -S0:1024k -B 1024 -PA (-I b) (-N 8385867) (-K 4) (-c) (-p R) > > (-L 1048577) (-D 0:100) (-t 0:2m) (-o 0) > > | 2010/01/05-02:58:26 | INFO | 27293 | v1.4.2 | /dev/sdb | Starting > > pass > > ^C| 2010/01/05-03:00:58 | STAT | 27293 | v1.4.2 | /dev/sdb | Total > > bytes written in 85578 transfers: 87631872 > > | 2010/01/05-03:00:58 | STAT | 27293 | v1.4.2 | /dev/sdb | Total > > write throughput: 701055.0B/s (0.67MB/s), IOPS 684.6/s. > > | 2010/01/05-03:00:58 | STAT | 27293 | v1.4.2 | /dev/sdb | Total > > Write Time: 125 seconds (0d0h2m5s) > > | 2010/01/05-03:00:58 | STAT | 27293 | v1.4.2 | /dev/sdb | Total > > overall runtime: 152 seconds (0d0h2m32s) > > | 2010/01/05-03:00:58 | END | 27293 | v1.4.2 | /dev/sdb | User > > Interrupt: Test Done (Passed) > > As you can see, the throughput was only 0.67MB/s and only 85578 > > written in 87631872 transfers... > > I also tweaked the options with "-p l" and/or "-I bd" (change seek > > pattern to linear and/or speficy IO type as block and direct IO) but > > no improvement happened... > Hmm.. so it does 684 IO operations per second (IOPS), and each IO was 1k > in size, so it makes 684 kB/sec of throughput. > 1000 milliseconds (1 second) divided by 684 IOPS is 1.46 milliseconds per IO.. > Are you sure you had 16ms of rtt? Actually that was probably the output from 0.2 ms rtt instead of 16 ms... I'm sorry for the mistake. I tried again the same command on a 16ms RTT, and the IOPS was mostly around 150. > Try to play and experiment with these options: > > -B 64k (blocksize 64k, try also 4k) > -I BD (block device, direct IO (O_DIRECT)) > -K 16 (16 threads, aka 16 outstanding IOs. -K 1 should be the same as dd) > > Examples: > > Sequential (linear) reads using blocksize 4k and 4 simultaneous threads, for > 60 seconds: > disktest -B 4k -h 1 -I BD -K 4 -p l -P T -T 60 -r /dev/sdX > > Random writes: > > disktest -B 4k -h 1 -I BD -K 4 -p r -P T -T 60 -w /dev/sdX > > 30% random reads, 70% random writes: > disktest -r -w -D30:70 -K2 -E32 -B 8k -T 60 -pR -Ibd -PA /dev/md4 > > Hopefully that helps.. That did help! I tried the following combinations of -B -K and -p at 20 ms RTT and the other options were -h 30 -I BD -P T -S0:(1 GB size) -B 4k/64k -K 4/64 -p l When I put -p l there the performance went down drastically... -B 4k -K 4/64 -p r The disk throughput was similar to the one I used in the previous post "disktest -w -S0:1k -B 1024 /dev/sdb " and it was much lower than dd could get. -B 64k -K 4 -p r The disk throughput was higher than the last one but still not as high as dd could get. -B 64k -K 64 -p r The disk throughput was boosted to 8.06 MB/s and the IOPS was 129.0. At the link layer, the traffic rate was 70.536 Mbps (the TCP baseline was 96.202 Mbps). At the same time, dd ( bs=64K count=(1 GB size)) got a throughput of 6.7 MB/s and the traffic rate on the link layer was 57.749 Mbps. Although not much, it was still an improvement :) and it was the first improvement I have ever seen since I started my experiments! Thank you very much! As for > Oh, also make sure you have 'oflag=direct' for dd. The result was surprisingly low again... Do you think the reason might be that I was running dd on a device file (/dev/sdb), which did not have any partitions/file systems on it? Thanks a lot! Jack
-- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.