>From the looks of it you are reaching the maxing out your OSD’s. Some of them >are pushing over 500 iops, which is a lot for a 7.2k disk and at the high >queue depths IO’s will have to wait a long time to reach the front of the >queue. The only real thing I can suggest is to add more OSD’s which will >spread your workload over more disks. It’s possible creating more PG’s may >distribute the workload a little better, but I don’t see it making a massive >difference.
From: ceph-users [mailto:[email protected]] On Behalf Of German Anders Sent: 10 June 2015 17:20 To: Ilya Dryomov Cc: ceph-users; Nick Fisk Subject: Re: [ceph-users] krbd splitting large IO's into smaller IO's hi guys, sorry that I hang on this email, I've four OSD servers with Ubuntu 14.04.1 LTS with 9 osd daemons each, 3TB drive size, and 3 ssd journal drives (each journal holds 3 osd daemons), the kernel version that I'm using is 3.18.3-031803-generic, and ceph version 0.82, I would like to know what would be the 'best' parameters in term of io for my 3TB devices, I've: scheduler: deadline max_hw_sectors_kb: 16383 max_sectors_kb: 4096 read_ahead_kb: 128 nr_requests: 128 I'm experience some high io waits on all the OSD servers: avg-cpu: %user %nice %system %iowait %steal %idle 1.74 0.00 15.43 64.80 0.00 18.03 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 1610.40 322.20 374.80 11.00 7940.80 1330.00 48.06 0.08 0.21 0.20 0.44 0.20 7.68 sdb 130.60 322.20 55.00 11.00 742.40 1330.00 62.80 0.02 0.23 0.17 0.51 0.19 1.28 md0 0.00 0.00 2170.80 332.40 8683.20 1329.60 8.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 2170.80 332.40 8683.20 1329.60 8.00 0.87 0.35 0.21 1.26 0.03 7.84 sdd 0.00 0.00 11.80 384.40 4217.60 33197.60 188.87 75.17 189.72 130.78 191.53 1.88 74.64 sdc 0.00 0.00 18.80 313.40 581.60 33154.40 203.11 78.09 235.08 66.85 245.17 2.16 71.84 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdf 0.00 0.80 78.20 181.40 10400.80 19204.80 228.09 31.75 110.93 43.09 140.18 2.99 77.52 sdg 0.00 0.00 1.60 304.60 51.20 31647.20 207.04 64.05 209.19 73.50 209.90 1.90 58.32 sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdi 0.00 0.00 6.60 17.20 159.20 2784.80 247.39 0.27 9.14 12.12 8.00 3.19 7.60 sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 0.00 13.40 120.00 428.80 8487.20 133.67 23.91 203.37 36.18 222.04 2.64 35.28 sdl 0.00 0.80 12.40 524.20 2088.80 40842.40 160.01 93.53 168.27 183.35 167.91 1.64 88.24 sdn 0.00 1.40 4.00 433.80 92.80 35926.40 164.55 88.72 196.29 299.40 195.33 1.71 74.96 sdm 0.00 0.00 0.60 544.60 19.20 40348.00 148.08 118.31 217.00 17.33 217.22 1.67 90.80 Thanks in advance, Best regards, German Anders Storage System Engineer Leader Despegar | IT Team office +54 11 4894 3500 x3408 mobile +54 911 3493 7262 mail [email protected] <mailto:[email protected]>
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
