On Wed, Jun 10, 2015 at 6:18 PM, Nick Fisk <[email protected]> wrote:
>> -----Original Message-----
>> From: Ilya Dryomov [mailto:[email protected]]
>> Sent: 10 June 2015 14:06
>> To: Nick Fisk
>> Cc: ceph-users
>> Subject: Re: [ceph-users] krbd splitting large IO's into smaller IO's
>>
>> On Wed, Jun 10, 2015 at 2:47 PM, Nick Fisk <[email protected]> wrote:
>> > Hi,
>> >
>> > Using Kernel RBD client with Kernel 4.03 (I have also tried some older
>> > kernels with the same effect) and IO is being split into smaller IO's
>> > which is having a negative impact on performance.
>> >
>> > cat /sys/block/sdc/queue/max_hw_sectors_kb
>> > 4096
>> >
>> > cat /sys/block/rbd0/queue/max_sectors_kb
>> > 4096
>> >
>> > Using DD
>> > dd if=/dev/rbd0 of=/dev/null bs=4M
>> >
>> > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
>> > avgqu-sz await r_await w_await svctm %util
>> > rbd0 0.00 0.00 201.50 0.00 25792.00 0.00 256.00
>> > 1.99 10.15 10.15 0.00 4.96 100.00
>> >
>> >
>> > Using FIO with 4M blocks
>> > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
>> > avgqu-sz await r_await w_await svctm %util
>> > rbd0 0.00 0.00 232.00 0.00 118784.00 0.00 1024.00
>> > 11.29 48.58 48.58 0.00 4.31 100.00
>> >
>> > Any ideas why IO sizes are limited to 128k (256 blocks) in DD's case
>> > and 512k in Fio's case?
>>
>> 128k vs 512k is probably buffered vs direct IO - add iflag=direct to your dd
>> invocation.
>
> Yes, thanks for this, that was the case
>
>>
>> As for the 512k - I'm pretty sure it's a regression in our switch to blk-mq.
>> I
>> tested it around 3.18-3.19 and saw steady 4M IOs. I hope we are just missing
>> a knob - I'll take a look.
>
> I've tested both 4.03 and 3.16 and both seem to be split into 512k. Let me
> know if you need me to test any other particular version.
With 3.16 you are going to need to adjust max_hw_sectors_kb /
max_sectors_kb as discussed in Dan's thread. The patch that fixed that
in the block layer went into 3.19, blk-mq into 4.0 - try 3.19.
Thanks,
Ilya
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
