[ceph-users] ceph df showing wrong MAX AVAIL for hybrid CRUSH Rule
Hi, I have the following configuration of OSDs: ID CLASS WEIGHT REWEIGHT SIZE USEAVAIL %USE VAR PGS 0 hdd 5.45599 1.0 5587G 2259G 3327G 40.45 1.10 234 1 hdd 5.45599 1.0 5587G 2295G 3291G 41.08 1.11 231 2 hdd 5.45599 1.0 5587G 2321G 3265G 41.56 1.13 232 3 hdd 5.45599 1.0 5587G 2313G 3273G 41.42 1.12 234 4 hdd 5.45599 1.0 5587G 2105G 3481G 37.68 1.02 212 5 hdd 5.45599 1.0 5587G 2231G 3355G 39.94 1.08 218 41 ssd 0.87299 1.0 894G 9637M 884G 1.05 0.03 31 42 ssd 0.87299 1.0 894G 13361M 881G 1.46 0.04 41 6 hdd 5.45599 1.0 5587G 2404G 3182G 43.03 1.17 239 7 hdd 5.45599 1.0 5587G 2226G 3360G 39.85 1.08 222 8 hdd 5.45599 1.0 5587G 2668G 2918G 47.76 1.30 256 9 hdd 5.45599 1.0 5587G 2366G 3220G 42.36 1.15 236 10 hdd 5.45599 1.0 5587G 2454G 3132G 43.92 1.19 253 11 hdd 5.45599 1.0 5587G 2405G 3181G 43.06 1.17 245 43 ssd 0.87299 1.0 894G 15498M 879G 1.69 0.05 47 44 ssd 0.87299 1.0 894G 10104M 884G 1.10 0.03 27 12 hdd 5.45599 1.0 5587G 2242G 3344G 40.14 1.09 229 13 hdd 5.45599 1.0 5587G 2551G 3035G 45.67 1.24 247 14 hdd 5.45599 1.0 5587G 2513G 3074G 44.98 1.22 245 15 hdd 5.45599 1.0 5587G 2014G 3572G 36.06 0.98 209 16 hdd 5.45599 1.0 5587G 2586G 3000G 46.29 1.26 249 17 hdd 5.45599 1.0 5587G 2459G 3127G 44.02 1.19 243 45 ssd 0.87299 1.0 894G 9697M 884G 1.06 0.03 35 46 ssd 0.87299 1.0 894G 12975M 881G 1.42 0.04 37 18 hdd 3.63699 1.0 3724G 1595G 2128G 42.84 1.16 156 19 hdd 3.63699 1.0 3724G 1387G 2336G 37.25 1.01 147 20 hdd 3.63699 1.0 3724G 1551G 2172G 41.67 1.13 157 21 hdd 3.63699 1.0 3724G 1535G 2189G 41.22 1.12 155 22 hdd 3.63699 1.0 3724G 1459G 2264G 39.20 1.06 155 23 hdd 3.63699 1.0 3724G 1395G 2329G 37.46 1.02 147 24 hdd 3.63699 1.0 3724G 1489G 2234G 40.00 1.08 160 25 hdd 3.63699 1.0 3724G 1634G 2090G 43.88 1.19 159 26 hdd 3.63699 1.0 3724G 1566G 2157G 42.06 1.14 154 49 ssd 0.87299 1.0 894G 9385M 884G 1.03 0.03 32 50 ssd 0.87299 1.0 894G 12757M 881G 1.39 0.04 36 27 hdd 5.45599 1.0 5587G 2462G 3124G 44.08 1.20 244 28 hdd 5.45599 1.0 5587G 2314G 3272G 41.43 1.12 237 29 hdd 5.45599 1.0 5587G 2166G 3420G 38.79 1.05 221 30 hdd 5.45599 1.0 5587G 2484G 3102G 44.47 1.21 242 31 hdd 5.45599 1.0 5587G 2292G 3294G 41.03 1.11 225 32 hdd 5.45599 1.0 5587G 1982G 3604G 35.49 0.96 208 47 ssd 0.87299 1.0 894G 12015M 882G 1.31 0.04 39 48 ssd 0.87299 1.0 894G 14820M 879G 1.62 0.04 46 33 hdd 5.45599 1.0 5587G 2002G 3584G 35.85 0.97 205 34 hdd 5.45599 1.0 5587G 2069G 3517G 37.04 1.00 209 35 hdd 5.45599 1.0 5587G 2187G 3399G 39.16 1.06 226 36 hdd 5.45599 1.0 5587G 1821G 3765G 32.60 0.88 185 37 hdd 5.45599 1.0 5587G 2123G 3463G 38.01 1.03 205 38 hdd 5.45599 1.0 5587G 2197G 3390G 39.32 1.07 228 39 hdd 5.45599 1.0 5587G 2180G 3406G 39.02 1.06 217 40 hdd 5.45599 1.0 5587G 2232G 3354G 39.97 1.08 228 51 ssd 0.87320 1.0 894G 14747M 879G 1.61 0.04 38 52 ssd 0.87320 1.0 894G 7716M 886G 0.84 0.02 18 53 ssd 0.87320 1.0 894G 12660M 881G 1.38 0.04 33 54 ssd 0.87320 1.0 894G 11155M 883G 1.22 0.03 31 55 ssd 0.87320 1.0 894G 9350M 885G 1.02 0.03 24 56 ssd 0.87320 1.0 894G 13816M 880G 1.51 0.04 38 57 ssd 0.87320 1.0 894G 10122M 884G 1.11 0.03 28 58 ssd 0.87320 1.0 894G 10096M 884G 1.10 0.03 32 59 ssd 0.87320 1.0 894G 13750M 880G 1.50 0.04 36 60 ssd 0.87320 1.0 894G 16168M 878G 1.77 0.05 35 61 ssd 0.87320 1.0 894G 11401M 883G 1.25 0.03 31 62 ssd 0.87320 1.0 894G 12105M 882G 1.32 0.04 32 63 ssd 0.87320 1.0 894G 13998M 880G 1.53 0.04 40 64 ssd 0.87320 1.0 894G 17468M 877G 1.91 0.05 42 65 ssd 0.87320 1.0 894G 9540M 884G 1.04 0.03 25 66 ssd 0.87320 1.0 894G 15109M 879G 1.65 0.04 42 TOTAL 230T 86866G 145T 36.88 MIN/MAX VAR: 0.02/1.30 STDDEV: 22.48 With the following crush ruleset: # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd device 6 osd.6 class hdd device 7 osd.7 class hdd device 8 osd.8 class hdd device 9 osd.9 class hdd device 10 osd.10 class hdd device 11 osd.11 class hdd device 12 osd.12 class hdd device 13 osd.13 class hdd device 14 osd.14 class hdd device 15 osd.15 class hdd device 16 osd.16 class hdd device 17 osd.17 class hdd device 18
[ceph-users] Recommendations for I/O (blk-mq) scheduler for HDDs and SSDs?
Hi, after reading a lot about I/O schedulers and performance gains with blk-mq, I switched to a custom 4.14.5 kernel with CONFIG_SCSI_MQ_DEFAULT enabled to have blk-mq for all devices on my cluster. This allows me to use the following schedulers for HDDs and SSDs: mq-deadline, kyber, bfq, none I've currently set the HDD scheduler to bfq and the SSD scheduler to none, however I'm still not sure if this is the best solution performance-wise. Does anyone have more experience with this and can maybe give me a recommendation? I'm not even sure if blk-mq is a good idea for ceph, since I haven't really found anything on the topic. Best, Patrick ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CRUSH - adding device class to existing rule without causing complete rebalance
Hi everyone,
I only have a single rule in my crushmap and only OSDs classed as hdd (after
the luminous update):
rule replicated_ruleset {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
Since luminous added device classes, I tried updating the rule to
step take default class hdd
so I could later add a rule for ssds without the existing pools trying to
balance on those.
Since all existing OSDs are classed as hdd I thought nothing should change with
this rule change (since the OSDs to be used are the same before and after the
change), however after inserting the new crushmap, the whole cluster started
rebalancing, after which I quickly reinserted the old crushmap.
So, is there any way to limit my existing pools to HDDs without causing a
complete rebalance (at least that's what it looked like)?
Best,
Patrick
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
