Den mån 4 mars 2024 kl 11:30 skrev Ml Ml <mliebher...@googlemail.com>: > > Hello, > > i wonder why my autobalancer is not working here:
I think the short answer is "because you have so wildly varying sizes both for drives and hosts". If your drive sizes span from 0.5 to 9.5, there will naturally be skewed data, and it is not a huge surprise that the automation has some troubles getting it "good". When the balancer places a PG on a 0.5-sized drive compared to a 9.5-sized one, it eats up 19x more of the "free space" on the smaller one, so there are very few good options when the sizes are so different. Even if you placed all PGs correctly due to size, the 9.5-sized disk would end up getting 19x more IO than the small drive and for hdd, it seldom is possible to gracefully handle a 19-fold increase in IO, most of the time will probably be spent on seeks. > root@ceph01:~# ceph -s > cluster: > id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df > health: HEALTH_ERR > 1 backfillfull osd(s) > 1 full osd(s) > 1 nearfull osd(s) > 4 pool(s) full > > => osd.17 was too full (92% or something like that) > > root@ceph01:~# ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE ... %USE ... PGS TYPE NAME > -25 209.50084 - 213 TiB ... 69.56 ... - datacenter > xxx-dc-root > -19 84.59369 - 86 TiB ... 56.97 ... - rack > RZ1.Reihe4.R10 > -3 35.49313 - 37 TiB ... 57.88 ... - host > ceph02 > 2 hdd 1.70000 1.00000 1.7 TiB ... 58.77 ... 44 osd.2 > 3 hdd 1.00000 1.00000 2.7 TiB ... 22.14 ... 25 osd.3 > 7 hdd 2.50000 1.00000 2.7 TiB ... 58.84 ... 70 osd.7 > 9 hdd 9.50000 1.00000 9.5 TiB ... 63.07 ... 268 osd.9 > 13 hdd 2.67029 1.00000 2.7 TiB ... 53.59 ... 65 osd.13 > 16 hdd 2.89999 1.00000 2.7 TiB ... 59.35 ... 71 osd.16 > 19 hdd 1.70000 1.00000 1.7 TiB ... 48.98 ... 37 osd.19 > 23 hdd 2.38419 1.00000 2.4 TiB ... 59.33 ... 64 osd.23 > 24 hdd 1.39999 1.00000 1.7 TiB ... 51.23 ... 39 osd.24 > 28 hdd 3.63869 1.00000 3.6 TiB ... 64.17 ... 104 osd.28 > 31 hdd 2.70000 1.00000 2.7 TiB ... 64.73 ... 76 osd.31 > 32 hdd 3.39999 1.00000 3.3 TiB ... 67.28 ... 101 osd.32 > -9 22.88817 - 23 TiB ... 56.96 ... - host > ceph06 > 35 hdd 7.15259 1.00000 7.2 TiB ... 55.71 ... 182 osd.35 > 36 hdd 5.24519 1.00000 5.2 TiB ... 53.75 ... 128 osd.36 > 45 hdd 5.24519 1.00000 5.2 TiB ... 60.91 ... 144 osd.45 > 48 hdd 5.24519 1.00000 5.2 TiB ... 57.94 ... 139 osd.48 > -17 26.21239 - 26 TiB ... 55.67 ... - host > ceph08 > 37 hdd 6.67569 1.00000 6.7 TiB ... 58.17 ... 174 osd.37 > 40 hdd 9.53670 1.00000 9.5 TiB ... 58.54 ... 250 osd.40 > 46 hdd 5.00000 1.00000 5.0 TiB ... 52.39 ... 116 osd.46 > 47 hdd 5.00000 1.00000 5.0 TiB ... 50.05 ... 112 osd.47 > -20 59.11053 - 60 TiB ... 82.47 ... - rack > RZ1.Reihe4.R9 > -4 23.09996 - 24 TiB ... 79.92 ... - host > ceph03 > 5 hdd 1.70000 0.75006 1.7 TiB ... 87.24 ... 66 osd.5 > 6 hdd 1.70000 0.44998 1.7 TiB ... 47.30 ... 36 osd.6 > 10 hdd 2.70000 0.85004 2.7 TiB ... 83.23 ... 100 osd.10 > 15 hdd 2.70000 0.75006 2.7 TiB ... 74.26 ... 88 osd.15 > 17 hdd 0.50000 0.85004 1.6 TiB ... 91.44 ... 67 osd.17 > 20 hdd 2.00000 0.85004 1.7 TiB ... 88.41 ... 68 osd.20 > 21 hdd 2.79999 0.75006 2.7 TiB ... 77.25 ... 91 osd.21 > 25 hdd 1.70000 0.90002 1.7 TiB ... 78.31 ... 60 osd.25 > 26 hdd 2.70000 1.00000 2.7 TiB ... 82.75 ... 99 osd.26 > 27 hdd 2.70000 0.90002 2.7 TiB ... 84.26 ... 101 osd.27 > 63 hdd 1.89999 0.90002 1.7 TiB ... 84.15 ... 65 osd.63 > -13 36.01057 - 36 TiB ... 84.12 ... - host > ceph05 > 11 hdd 7.15259 0.90002 7.2 TiB ... 85.45 ... 273 osd.11 > 39 hdd 7.20000 0.85004 7.2 TiB ... 80.90 ... 257 osd.39 > 41 hdd 7.20000 0.75006 7.2 TiB ... 74.95 ... 239 osd.41 > 42 hdd 9.00000 1.00000 9.5 TiB ... 92.00 ... 392 osd.42 > 43 hdd 5.45799 1.00000 5.5 TiB ... 84.84 ... 207 osd.43 > -21 65.79662 - 66 TiB ... 74.29 ... - rack > RZ3.Reihe3.R10 > -2 28.49664 - 29 TiB ... 74.79 ... - host > ceph01 > 0 hdd 2.70000 1.00000 2.7 TiB ... 73.82 ... 88 osd.0 > 1 hdd 3.63869 1.00000 3.6 TiB ... 73.47 ... 121 osd.1 > 4 hdd 2.70000 1.00000 2.7 TiB ... 74.63 ... 89 osd.4 > 8 hdd 2.70000 1.00000 2.7 TiB ... 77.10 ... 92 osd.8 > 12 hdd 2.70000 1.00000 2.7 TiB ... 78.76 ... 94 osd.12 > 14 hdd 5.45799 1.00000 5.5 TiB ... 78.86 ... 193 osd.14 > 18 hdd 1.89999 1.00000 2.7 TiB ... 63.79 ... 76 osd.18 > 22 hdd 1.70000 1.00000 1.7 TiB ... 74.85 ... 57 osd.22 > 30 hdd 1.70000 1.00000 1.7 TiB ... 76.34 ... 59 osd.30 > 64 hdd 3.29999 1.00000 3.3 TiB ... 73.48 ... 110 osd.64 > -11 12.39999 - 12 TiB ... 73.40 ... - host > ceph04 > 34 hdd 5.20000 1.00000 5.2 TiB ... 72.81 ... 171 osd.34 > 44 hdd 7.20000 1.00000 7.2 TiB ... 73.83 ... 236 osd.44 > -15 24.89998 - 25 TiB ... 74.15 ... - host > ceph07 > 66 hdd 7.20000 1.00000 7.2 TiB ... 74.07 ... 236 osd.66 > 67 hdd 7.20000 1.00000 7.2 TiB ... 73.74 ... 236 osd.67 > 68 hdd 3.29999 1.00000 3.3 TiB ... 72.99 ... 110 osd.68 > 69 hdd 7.20000 1.00000 7.2 TiB ... 75.18 ... 241 osd.69 > -1 0 - 0 B ... 0 ... - root default > TOTAL 213 TiB ... 69.56 > -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
