I haven't had crush-compat do anything helpful for balancing my clusters. upmap has been amazing and balanced my clusters far better than anything else I've ever seen. I would go so far as to say that upmap can achieve a perfect balance.
It seems to evenly distribute the PGs for each pool onto all OSDs that pool is on. It does that with a maximum difference of 1PG based on how divisible the number of PGs are with the number of OSDs you have. As a side note, your OSD CRUSH weights should be the default weights for their size for upmap to be as effective as it can be. On Sat, Oct 20, 2018 at 3:58 PM Oliver Freyermuth < [email protected]> wrote: > Ok, I'll try out the balancer end of the upcoming week then (after we've > fixed a HW-issue with one of our mons > and the cooling system). > > Until then, any further advice and whether upmap is recommended over > crush-compat (all clients are Luminous) are welcome ;-). > > Cheers, > Oliver > > Am 20.10.18 um 21:26 schrieb Janne Johansson: > > Ok, can't say "why" then, I'd reweigh them somewhat to even it out, > > 1.22 -vs- 0.74 in variance is a lot, so either a balancer plugin for > > the MGRs, a script or just a few manual tweaks might be in order. > > > > Den lör 20 okt. 2018 kl 21:02 skrev Oliver Freyermuth > > <[email protected]>: > >> > >> All OSDs are of the very same size. One OSD host has slightly more > disks (33 instead of 31), though. > >> So also that that can't explain the hefty difference. > >> > >> I attach the output of "ceph osd tree" and "ceph osd df". > >> > >> The crush rule for the ceph_data pool is: > >> rule cephfs_data { > >> id 2 > >> type erasure > >> min_size 3 > >> max_size 6 > >> step set_chooseleaf_tries 5 > >> step set_choose_tries 100 > >> step take default class hdd > >> step chooseleaf indep 0 type host > >> step emit > >> } > >> So that only considers the hdd device class. EC is done with k=4 m=2. > >> > >> So I don't see any imbalance on the hardware level, but only a somewhat > uneven distribution of PGs. > >> Am I missing something, or is this really just a case for the ceph > balancer plugin? > >> I'm just a bit astonished this effect is so huge. > >> Maybe our 4096 PGs for the ceph_data pool are not enough to get an even > distribution without balancing? > >> But it yields about 100 PGs per OSD, as you can see... > >> > >> -------------------------------------------------------------- > >> # ceph osd tree > >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > >> -1 826.26428 root default > >> -3 0.43700 host mon001 > >> 0 ssd 0.21799 osd.0 up 1.00000 1.00000 > >> 1 ssd 0.21799 osd.1 up 1.00000 1.00000 > >> -5 0.43700 host mon002 > >> 2 ssd 0.21799 osd.2 up 1.00000 1.00000 > >> 3 ssd 0.21799 osd.3 up 1.00000 1.00000 > >> -31 1.81898 host mon003 > >> 230 ssd 0.90999 osd.230 up 1.00000 1.00000 > >> 231 ssd 0.90999 osd.231 up 1.00000 1.00000 > >> -10 116.64600 host osd001 > >> 4 hdd 3.64499 osd.4 up 1.00000 1.00000 > >> 5 hdd 3.64499 osd.5 up 1.00000 1.00000 > >> 6 hdd 3.64499 osd.6 up 1.00000 1.00000 > >> 7 hdd 3.64499 osd.7 up 1.00000 1.00000 > >> 8 hdd 3.64499 osd.8 up 1.00000 1.00000 > >> 9 hdd 3.64499 osd.9 up 1.00000 1.00000 > >> 10 hdd 3.64499 osd.10 up 1.00000 1.00000 > >> 11 hdd 3.64499 osd.11 up 1.00000 1.00000 > >> 12 hdd 3.64499 osd.12 up 1.00000 1.00000 > >> 13 hdd 3.64499 osd.13 up 1.00000 1.00000 > >> 14 hdd 3.64499 osd.14 up 1.00000 1.00000 > >> 15 hdd 3.64499 osd.15 up 1.00000 1.00000 > >> 16 hdd 3.64499 osd.16 up 1.00000 1.00000 > >> 17 hdd 3.64499 osd.17 up 1.00000 1.00000 > >> 18 hdd 3.64499 osd.18 up 1.00000 1.00000 > >> 19 hdd 3.64499 osd.19 up 1.00000 1.00000 > >> 20 hdd 3.64499 osd.20 up 1.00000 1.00000 > >> 21 hdd 3.64499 osd.21 up 1.00000 1.00000 > >> 22 hdd 3.64499 osd.22 up 1.00000 1.00000 > >> 23 hdd 3.64499 osd.23 up 1.00000 1.00000 > >> 24 hdd 3.64499 osd.24 up 1.00000 1.00000 > >> 25 hdd 3.64499 osd.25 up 1.00000 1.00000 > >> 26 hdd 3.64499 osd.26 up 1.00000 1.00000 > >> 27 hdd 3.64499 osd.27 up 1.00000 1.00000 > >> 28 hdd 3.64499 osd.28 up 1.00000 1.00000 > >> 29 hdd 3.64499 osd.29 up 1.00000 1.00000 > >> 30 hdd 3.64499 osd.30 up 1.00000 1.00000 > >> 31 hdd 3.64499 osd.31 up 1.00000 1.00000 > >> 32 hdd 3.64499 osd.32 up 1.00000 1.00000 > >> 33 hdd 3.64499 osd.33 up 1.00000 1.00000 > >> 34 hdd 3.64499 osd.34 up 1.00000 1.00000 > >> 35 hdd 3.64499 osd.35 up 1.00000 1.00000 > >> -13 116.64600 host osd002 > >> 36 hdd 3.64499 osd.36 up 1.00000 1.00000 > >> 37 hdd 3.64499 osd.37 up 1.00000 1.00000 > >> 38 hdd 3.64499 osd.38 up 1.00000 1.00000 > >> 39 hdd 3.64499 osd.39 up 1.00000 1.00000 > >> 40 hdd 3.64499 osd.40 up 1.00000 1.00000 > >> 41 hdd 3.64499 osd.41 up 1.00000 1.00000 > >> 42 hdd 3.64499 osd.42 up 1.00000 1.00000 > >> 43 hdd 3.64499 osd.43 up 1.00000 1.00000 > >> 44 hdd 3.64499 osd.44 up 1.00000 1.00000 > >> 45 hdd 3.64499 osd.45 up 1.00000 1.00000 > >> 46 hdd 3.64499 osd.46 up 1.00000 1.00000 > >> 47 hdd 3.64499 osd.47 up 1.00000 1.00000 > >> 48 hdd 3.64499 osd.48 up 1.00000 1.00000 > >> 49 hdd 3.64499 osd.49 up 1.00000 1.00000 > >> 50 hdd 3.64499 osd.50 up 1.00000 1.00000 > >> 51 hdd 3.64499 osd.51 up 1.00000 1.00000 > >> 52 hdd 3.64499 osd.52 up 1.00000 1.00000 > >> 53 hdd 3.64499 osd.53 up 1.00000 1.00000 > >> 54 hdd 3.64499 osd.54 up 1.00000 1.00000 > >> 55 hdd 3.64499 osd.55 up 1.00000 1.00000 > >> 56 hdd 3.64499 osd.56 up 1.00000 1.00000 > >> 57 hdd 3.64499 osd.57 up 1.00000 1.00000 > >> 58 hdd 3.64499 osd.58 up 1.00000 1.00000 > >> 59 hdd 3.64499 osd.59 up 1.00000 1.00000 > >> 60 hdd 3.64499 osd.60 up 1.00000 1.00000 > >> 61 hdd 3.64499 osd.61 up 1.00000 1.00000 > >> 62 hdd 3.64499 osd.62 up 1.00000 1.00000 > >> 63 hdd 3.64499 osd.63 up 1.00000 1.00000 > >> 64 hdd 3.64499 osd.64 up 1.00000 1.00000 > >> 65 hdd 3.64499 osd.65 up 1.00000 1.00000 > >> 66 hdd 3.64499 osd.66 up 1.00000 1.00000 > >> 67 hdd 3.64499 osd.67 up 1.00000 1.00000 > >> -16 116.63899 host osd003 > >> 68 hdd 3.64499 osd.68 up 1.00000 1.00000 > >> 69 hdd 3.64499 osd.69 up 1.00000 1.00000 > >> 70 hdd 3.64499 osd.70 up 1.00000 1.00000 > >> 71 hdd 3.64499 osd.71 up 1.00000 1.00000 > >> 72 hdd 3.64499 osd.72 up 1.00000 1.00000 > >> 73 hdd 3.64499 osd.73 up 1.00000 1.00000 > >> 74 hdd 3.64499 osd.74 up 1.00000 1.00000 > >> 75 hdd 3.64499 osd.75 up 1.00000 1.00000 > >> 76 hdd 3.64499 osd.76 up 1.00000 1.00000 > >> 77 hdd 3.64499 osd.77 up 1.00000 1.00000 > >> 78 hdd 3.64499 osd.78 up 1.00000 1.00000 > >> 79 hdd 3.64499 osd.79 up 1.00000 1.00000 > >> 80 hdd 3.63899 osd.80 up 1.00000 1.00000 > >> 81 hdd 3.64499 osd.81 up 1.00000 1.00000 > >> 82 hdd 3.64499 osd.82 up 1.00000 1.00000 > >> 83 hdd 3.64499 osd.83 up 1.00000 1.00000 > >> 84 hdd 3.64499 osd.84 up 1.00000 1.00000 > >> 85 hdd 3.64499 osd.85 up 1.00000 1.00000 > >> 86 hdd 3.64499 osd.86 up 1.00000 1.00000 > >> 87 hdd 3.64499 osd.87 up 1.00000 1.00000 > >> 88 hdd 3.64499 osd.88 up 1.00000 1.00000 > >> 89 hdd 3.64499 osd.89 up 1.00000 1.00000 > >> 90 hdd 3.64499 osd.90 up 1.00000 1.00000 > >> 91 hdd 3.64499 osd.91 up 1.00000 1.00000 > >> 92 hdd 3.64499 osd.92 up 1.00000 1.00000 > >> 93 hdd 3.64499 osd.93 up 1.00000 1.00000 > >> 94 hdd 3.64499 osd.94 up 1.00000 1.00000 > >> 95 hdd 3.64499 osd.95 up 1.00000 1.00000 > >> 96 hdd 3.64499 osd.96 up 1.00000 1.00000 > >> 97 hdd 3.64499 osd.97 up 1.00000 1.00000 > >> 98 hdd 3.64499 osd.98 up 1.00000 1.00000 > >> 99 hdd 3.64499 osd.99 up 1.00000 1.00000 > >> -19 116.64600 host osd004 > >> 100 hdd 3.64499 osd.100 up 1.00000 1.00000 > >> 101 hdd 3.64499 osd.101 up 1.00000 1.00000 > >> 102 hdd 3.64499 osd.102 up 1.00000 1.00000 > >> 103 hdd 3.64499 osd.103 up 1.00000 1.00000 > >> 104 hdd 3.64499 osd.104 up 1.00000 1.00000 > >> 105 hdd 3.64499 osd.105 up 1.00000 1.00000 > >> 106 hdd 3.64499 osd.106 up 1.00000 1.00000 > >> 107 hdd 3.64499 osd.107 up 1.00000 1.00000 > >> 108 hdd 3.64499 osd.108 up 1.00000 1.00000 > >> 109 hdd 3.64499 osd.109 up 1.00000 1.00000 > >> 110 hdd 3.64499 osd.110 up 1.00000 1.00000 > >> 111 hdd 3.64499 osd.111 up 1.00000 1.00000 > >> 112 hdd 3.64499 osd.112 up 1.00000 1.00000 > >> 113 hdd 3.64499 osd.113 up 1.00000 1.00000 > >> 114 hdd 3.64499 osd.114 up 1.00000 1.00000 > >> 115 hdd 3.64499 osd.115 up 1.00000 1.00000 > >> 116 hdd 3.64499 osd.116 up 1.00000 1.00000 > >> 117 hdd 3.64499 osd.117 up 1.00000 1.00000 > >> 118 hdd 3.64499 osd.118 up 1.00000 1.00000 > >> 119 hdd 3.64499 osd.119 up 1.00000 1.00000 > >> 120 hdd 3.64499 osd.120 up 1.00000 1.00000 > >> 121 hdd 3.64499 osd.121 up 1.00000 1.00000 > >> 122 hdd 3.64499 osd.122 up 1.00000 1.00000 > >> 123 hdd 3.64499 osd.123 up 1.00000 1.00000 > >> 124 hdd 3.64499 osd.124 up 1.00000 1.00000 > >> 125 hdd 3.64499 osd.125 up 1.00000 1.00000 > >> 126 hdd 3.64499 osd.126 up 1.00000 1.00000 > >> 127 hdd 3.64499 osd.127 up 1.00000 1.00000 > >> 128 hdd 3.64499 osd.128 up 1.00000 1.00000 > >> 129 hdd 3.64499 osd.129 up 1.00000 1.00000 > >> 130 hdd 3.64499 osd.130 up 1.00000 1.00000 > >> 131 hdd 3.64499 osd.131 up 1.00000 1.00000 > >> -22 116.64600 host osd005 > >> 132 hdd 3.64499 osd.132 up 1.00000 1.00000 > >> 133 hdd 3.64499 osd.133 up 1.00000 1.00000 > >> 134 hdd 3.64499 osd.134 up 1.00000 1.00000 > >> 135 hdd 3.64499 osd.135 up 1.00000 1.00000 > >> 136 hdd 3.64499 osd.136 up 1.00000 1.00000 > >> 137 hdd 3.64499 osd.137 up 1.00000 1.00000 > >> 138 hdd 3.64499 osd.138 up 1.00000 1.00000 > >> 139 hdd 3.64499 osd.139 up 1.00000 1.00000 > >> 140 hdd 3.64499 osd.140 up 1.00000 1.00000 > >> 141 hdd 3.64499 osd.141 up 1.00000 1.00000 > >> 142 hdd 3.64499 osd.142 up 1.00000 1.00000 > >> 143 hdd 3.64499 osd.143 up 1.00000 1.00000 > >> 144 hdd 3.64499 osd.144 up 1.00000 1.00000 > >> 145 hdd 3.64499 osd.145 up 1.00000 1.00000 > >> 146 hdd 3.64499 osd.146 up 1.00000 1.00000 > >> 147 hdd 3.64499 osd.147 up 1.00000 1.00000 > >> 148 hdd 3.64499 osd.148 up 1.00000 1.00000 > >> 149 hdd 3.64499 osd.149 up 1.00000 1.00000 > >> 150 hdd 3.64499 osd.150 up 1.00000 1.00000 > >> 151 hdd 3.64499 osd.151 up 1.00000 1.00000 > >> 152 hdd 3.64499 osd.152 up 1.00000 1.00000 > >> 153 hdd 3.64499 osd.153 up 1.00000 1.00000 > >> 154 hdd 3.64499 osd.154 up 1.00000 1.00000 > >> 155 hdd 3.64499 osd.155 up 1.00000 1.00000 > >> 156 hdd 3.64499 osd.156 up 1.00000 1.00000 > >> 157 hdd 3.64499 osd.157 up 1.00000 1.00000 > >> 158 hdd 3.64499 osd.158 up 1.00000 1.00000 > >> 159 hdd 3.64499 osd.159 up 1.00000 1.00000 > >> 160 hdd 3.64499 osd.160 up 1.00000 1.00000 > >> 161 hdd 3.64499 osd.161 up 1.00000 1.00000 > >> 162 hdd 3.64499 osd.162 up 1.00000 1.00000 > >> 163 hdd 3.64499 osd.163 up 1.00000 1.00000 > >> -25 116.63335 host osd006 > >> 164 hdd 3.64499 osd.164 up 1.00000 1.00000 > >> 165 hdd 3.64499 osd.165 up 1.00000 1.00000 > >> 166 hdd 3.64499 osd.166 up 1.00000 1.00000 > >> 167 hdd 3.64499 osd.167 up 1.00000 1.00000 > >> 168 hdd 3.64499 osd.168 up 1.00000 1.00000 > >> 169 hdd 3.64499 osd.169 up 1.00000 1.00000 > >> 170 hdd 3.64499 osd.170 up 1.00000 1.00000 > >> 171 hdd 3.64499 osd.171 up 1.00000 1.00000 > >> 172 hdd 3.64499 osd.172 up 1.00000 1.00000 > >> 173 hdd 3.64499 osd.173 up 1.00000 1.00000 > >> 174 hdd 3.64499 osd.174 up 1.00000 1.00000 > >> 175 hdd 3.64499 osd.175 up 1.00000 1.00000 > >> 176 hdd 3.64499 osd.176 up 1.00000 1.00000 > >> 177 hdd 3.64499 osd.177 up 1.00000 1.00000 > >> 178 hdd 3.64499 osd.178 up 1.00000 1.00000 > >> 179 hdd 3.64499 osd.179 up 1.00000 1.00000 > >> 180 hdd 3.64499 osd.180 up 1.00000 1.00000 > >> 181 hdd 3.64499 osd.181 up 1.00000 1.00000 > >> 182 hdd 3.64499 osd.182 up 1.00000 1.00000 > >> 183 hdd 3.64499 osd.183 up 1.00000 1.00000 > >> 184 hdd 3.64499 osd.184 up 1.00000 1.00000 > >> 185 hdd 3.64499 osd.185 up 1.00000 1.00000 > >> 186 hdd 3.64499 osd.186 up 1.00000 1.00000 > >> 187 hdd 3.64499 osd.187 up 1.00000 1.00000 > >> 188 hdd 3.64499 osd.188 up 1.00000 1.00000 > >> 189 hdd 3.64499 osd.189 up 1.00000 1.00000 > >> 190 hdd 3.64499 osd.190 up 1.00000 1.00000 > >> 191 hdd 3.64499 osd.191 up 1.00000 1.00000 > >> 192 hdd 3.63869 osd.192 up 1.00000 1.00000 > >> 193 hdd 3.64499 osd.193 up 1.00000 1.00000 > >> 194 hdd 3.64499 osd.194 up 1.00000 1.00000 > >> 195 hdd 3.64499 osd.195 up 1.00000 1.00000 > >> -28 123.71500 host osd007 > >> 196 hdd 3.63899 osd.196 up 1.00000 1.00000 > >> 197 hdd 3.63899 osd.197 up 1.00000 1.00000 > >> 198 hdd 3.63899 osd.198 up 1.00000 1.00000 > >> 199 hdd 3.63899 osd.199 up 1.00000 1.00000 > >> 200 hdd 3.63899 osd.200 up 1.00000 1.00000 > >> 201 hdd 3.63899 osd.201 up 1.00000 1.00000 > >> 202 hdd 3.63899 osd.202 up 1.00000 1.00000 > >> 203 hdd 3.63899 osd.203 up 1.00000 1.00000 > >> 204 hdd 3.63899 osd.204 up 1.00000 1.00000 > >> 205 hdd 3.63899 osd.205 up 1.00000 1.00000 > >> 206 hdd 3.63899 osd.206 up 1.00000 1.00000 > >> 207 hdd 3.63899 osd.207 up 1.00000 1.00000 > >> 208 hdd 3.63899 osd.208 up 1.00000 1.00000 > >> 209 hdd 3.63899 osd.209 up 1.00000 1.00000 > >> 210 hdd 3.63899 osd.210 up 1.00000 1.00000 > >> 211 hdd 3.63899 osd.211 up 1.00000 1.00000 > >> 212 hdd 3.63899 osd.212 up 1.00000 1.00000 > >> 213 hdd 3.63899 osd.213 up 1.00000 1.00000 > >> 214 hdd 3.63899 osd.214 up 1.00000 1.00000 > >> 215 hdd 3.63899 osd.215 up 1.00000 1.00000 > >> 216 hdd 3.63899 osd.216 up 1.00000 1.00000 > >> 217 hdd 3.63899 osd.217 up 1.00000 1.00000 > >> 218 hdd 3.63899 osd.218 up 1.00000 1.00000 > >> 219 hdd 3.63899 osd.219 up 1.00000 1.00000 > >> 220 hdd 3.63899 osd.220 up 1.00000 1.00000 > >> 221 hdd 3.63899 osd.221 up 1.00000 1.00000 > >> 222 hdd 3.63899 osd.222 up 1.00000 1.00000 > >> 223 hdd 3.63899 osd.223 up 1.00000 1.00000 > >> 224 hdd 3.63899 osd.224 up 1.00000 1.00000 > >> 225 hdd 3.63899 osd.225 up 1.00000 1.00000 > >> 226 hdd 3.63899 osd.226 up 1.00000 1.00000 > >> 227 hdd 3.63899 osd.227 up 1.00000 1.00000 > >> 228 hdd 3.63899 osd.228 up 1.00000 1.00000 > >> 229 hdd 3.63899 osd.229 up 1.00000 1.00000 > >> > >> # ceph osd df > >> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > >> 0 ssd 0.21799 1.00000 223G 4305M 219G 1.88 0.04 64 > >> 1 ssd 0.21799 1.00000 223G 3353M 220G 1.46 0.03 64 > >> 2 ssd 0.21799 1.00000 223G 1836M 221G 0.80 0.02 56 > >> 3 ssd 0.21799 1.00000 223G 2276M 221G 0.99 0.02 72 > >> 230 ssd 0.90999 1.00000 931G 2775M 928G 0.29 0.01 60 > >> 231 ssd 0.90999 1.00000 931G 3279M 928G 0.34 0.01 68 > >> 4 hdd 3.64499 1.00000 3726G 1916G 1809G 51.45 1.02 111 > >> 5 hdd 3.64499 1.00000 3726G 2144G 1581G 57.54 1.15 124 > >> 6 hdd 3.64499 1.00000 3726G 1887G 1838G 50.66 1.01 109 > >> 7 hdd 3.64499 1.00000 3726G 1642G 2083G 44.09 0.88 95 > >> 8 hdd 3.64499 1.00000 3726G 1969G 1756G 52.86 1.05 114 > >> 9 hdd 3.64499 1.00000 3726G 1577G 2148G 42.34 0.84 91 > >> 10 hdd 3.64499 1.00000 3726G 2043G 1682G 54.85 1.09 118 > >> 11 hdd 3.64499 1.00000 3726G 1957G 1768G 52.54 1.05 113 > >> 12 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114 > >> 13 hdd 3.64499 1.00000 3726G 1676G 2049G 45.00 0.90 97 > >> 14 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102 > >> 15 hdd 3.64499 1.00000 3726G 2161G 1564G 58.02 1.15 125 > >> 16 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101 > >> 17 hdd 3.64499 1.00000 3726G 2022G 1703G 54.29 1.08 117 > >> 18 hdd 3.64499 1.00000 3726G 1784G 1941G 47.89 0.95 103 > >> 19 hdd 3.64499 1.00000 3726G 1783G 1942G 47.85 0.95 103 > >> 20 hdd 3.64499 1.00000 3726G 1731G 1994G 46.47 0.92 100 > >> 21 hdd 3.64499 1.00000 3726G 1939G 1786G 52.06 1.04 112 > >> 22 hdd 3.64499 1.00000 3726G 2061G 1664G 55.34 1.10 119 > >> 23 hdd 3.64499 1.00000 3726G 1538G 2187G 41.29 0.82 89 > >> 24 hdd 3.64499 1.00000 3726G 2168G 1557G 58.19 1.16 125 > >> 25 hdd 3.64499 1.00000 3726G 2023G 1702G 54.32 1.08 117 > >> 26 hdd 3.64499 1.00000 3726G 1900G 1825G 51.00 1.02 110 > >> 27 hdd 3.64499 1.00000 3726G 2126G 1599G 57.08 1.14 123 > >> 28 hdd 3.64499 1.00000 3726G 1938G 1787G 52.02 1.04 112 > >> 29 hdd 3.64499 1.00000 3726G 1663G 2062G 44.65 0.89 96 > >> 30 hdd 3.64499 1.00000 3726G 2163G 1562G 58.07 1.16 125 > >> 31 hdd 3.64499 1.00000 3726G 1919G 1806G 51.52 1.03 111 > >> 32 hdd 3.64499 1.00000 3726G 1696G 2029G 45.52 0.91 98 > >> 33 hdd 3.64499 1.00000 3726G 1438G 2287G 38.61 0.77 83 > >> 34 hdd 3.64499 1.00000 3726G 2073G 1652G 55.64 1.11 120 > >> 35 hdd 3.64499 1.00000 3726G 2079G 1646G 55.81 1.11 120 > >> 36 hdd 3.64499 1.00000 3726G 1955G 1770G 52.50 1.04 113 > >> 37 hdd 3.64499 1.00000 3726G 2180G 1545G 58.51 1.16 126 > >> 38 hdd 3.64499 1.00000 3726G 1664G 2061G 44.66 0.89 96 > >> 39 hdd 3.64499 1.00000 3726G 2003G 1722G 53.77 1.07 116 > >> 40 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103 > >> 41 hdd 3.64499 1.00000 3726G 1903G 1822G 51.08 1.02 110 > >> 42 hdd 3.64499 1.00000 3726G 1747G 1978G 46.89 0.93 101 > >> 43 hdd 3.64499 1.00000 3726G 2021G 1704G 54.26 1.08 117 > >> 44 hdd 3.64499 1.00000 3726G 1852G 1873G 49.71 0.99 107 > >> 45 hdd 3.64499 1.00000 3726G 1522G 2203G 40.87 0.81 88 > >> 46 hdd 3.64499 1.00000 3726G 1851G 1874G 49.69 0.99 107 > >> 47 hdd 3.64499 1.00000 3726G 1698G 2027G 45.59 0.91 98 > >> 48 hdd 3.64499 1.00000 3726G 1940G 1785G 52.08 1.04 112 > >> 49 hdd 3.64499 1.00000 3726G 1746G 1979G 46.88 0.93 101 > >> 50 hdd 3.64499 1.00000 3726G 1728G 1997G 46.40 0.92 100 > >> 51 hdd 3.64499 1.00000 3726G 2093G 1632G 56.18 1.12 121 > >> 52 hdd 3.64499 1.00000 3726G 2235G 1490G 60.01 1.19 129 > >> 53 hdd 3.64499 1.00000 3726G 1871G 1854G 50.23 1.00 108 > >> 54 hdd 3.64499 1.00000 3726G 2094G 1631G 56.22 1.12 121 > >> 55 hdd 3.64499 1.00000 3726G 2422G 1303G 65.02 1.29 140 > >> 56 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108 > >> 57 hdd 3.64499 1.00000 3726G 1729G 1996G 46.43 0.92 100 > >> 58 hdd 3.64499 1.00000 3726G 1976G 1749G 53.05 1.06 114 > >> 59 hdd 3.64499 1.00000 3726G 1954G 1771G 52.46 1.04 113 > >> 60 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107 > >> 61 hdd 3.64499 1.00000 3726G 2042G 1683G 54.82 1.09 118 > >> 62 hdd 3.64499 1.00000 3726G 1921G 1804G 51.58 1.03 111 > >> 63 hdd 3.64499 1.00000 3726G 2128G 1597G 57.13 1.14 123 > >> 64 hdd 3.64499 1.00000 3726G 2037G 1688G 54.67 1.09 118 > >> 65 hdd 3.64499 1.00000 3726G 1970G 1755G 52.89 1.05 114 > >> 66 hdd 3.64499 1.00000 3726G 1539G 2186G 41.32 0.82 89 > >> 67 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108 > >> 68 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109 > >> 69 hdd 3.64499 1.00000 3726G 2092G 1633G 56.16 1.12 121 > >> 70 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104 > >> 71 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129 > >> 72 hdd 3.64499 1.00000 3726G 1557G 2168G 41.79 0.83 90 > >> 73 hdd 3.64499 1.00000 3726G 2300G 1425G 61.75 1.23 133 > >> 74 hdd 3.64499 1.00000 3726G 2132G 1593G 57.24 1.14 123 > >> 75 hdd 3.64499 1.00000 3726G 1818G 1907G 48.80 0.97 105 > >> 76 hdd 3.64499 1.00000 3726G 2231G 1494G 59.89 1.19 129 > >> 77 hdd 3.64499 1.00000 3726G 1749G 1976G 46.96 0.93 101 > >> 78 hdd 3.64499 1.00000 3726G 1766G 1959G 47.41 0.94 102 > >> 79 hdd 3.64499 1.00000 3726G 1678G 2047G 45.04 0.90 97 > >> 80 hdd 3.63899 1.00000 3726G 1744G 1981G 46.82 0.93 102 > >> 81 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109 > >> 82 hdd 3.64499 1.00000 3726G 1711G 2014G 45.93 0.91 99 > >> 83 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100 > >> 84 hdd 3.64499 1.00000 3726G 1834G 1891G 49.22 0.98 106 > >> 85 hdd 3.64499 1.00000 3726G 1709G 2016G 45.88 0.91 99 > >> 86 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112 > >> 87 hdd 3.64499 1.00000 3726G 2131G 1594G 57.19 1.14 123 > >> 88 hdd 3.64499 1.00000 3726G 1559G 2166G 41.85 0.83 90 > >> 89 hdd 3.64499 1.00000 3726G 1645G 2080G 44.16 0.88 95 > >> 90 hdd 3.64499 1.00000 3726G 2045G 1680G 54.89 1.09 118 > >> 91 hdd 3.64499 1.00000 3726G 2164G 1561G 58.09 1.16 125 > >> 92 hdd 3.64499 1.00000 3726G 1922G 1803G 51.59 1.03 111 > >> 93 hdd 3.64499 1.00000 3726G 1938G 1787G 52.03 1.04 112 > >> 94 hdd 3.64499 1.00000 3726G 1971G 1754G 52.91 1.05 114 > >> 95 hdd 3.64499 1.00000 3726G 1917G 1808G 51.47 1.02 111 > >> 96 hdd 3.64499 1.00000 3726G 1814G 1911G 48.70 0.97 105 > >> 97 hdd 3.64499 1.00000 3726G 1660G 2065G 44.57 0.89 96 > >> 98 hdd 3.64499 1.00000 3726G 1904G 1821G 51.10 1.02 110 > >> 99 hdd 3.64499 1.00000 3726G 2112G 1613G 56.70 1.13 122 > >> 100 hdd 3.64499 1.00000 3726G 1869G 1856G 50.18 1.00 108 > >> 101 hdd 3.64499 1.00000 3726G 2005G 1720G 53.84 1.07 116 > >> 102 hdd 3.64499 1.00000 3726G 2039G 1686G 54.73 1.09 118 > >> 103 hdd 3.64499 1.00000 3726G 2093G 1632G 56.20 1.12 121 > >> 104 hdd 3.64499 1.00000 3726G 1884G 1841G 50.58 1.01 109 > >> 105 hdd 3.64499 1.00000 3726G 1867G 1858G 50.13 1.00 108 > >> 106 hdd 3.64499 1.00000 3726G 1956G 1769G 52.51 1.05 113 > >> 107 hdd 3.64499 1.00000 3726G 2077G 1648G 55.74 1.11 120 > >> 108 hdd 3.64499 1.00000 3726G 1990G 1735G 53.43 1.06 115 > >> 109 hdd 3.64499 1.00000 3726G 1781G 1944G 47.80 0.95 103 > >> 110 hdd 3.64499 1.00000 3726G 1869G 1856G 50.17 1.00 108 > >> 111 hdd 3.64499 1.00000 3726G 2023G 1702G 54.30 1.08 117 > >> 112 hdd 3.64499 1.00000 3726G 2007G 1718G 53.89 1.07 116 > >> 113 hdd 3.64499 1.00000 3726G 1851G 1874G 49.70 0.99 107 > >> 114 hdd 3.64499 1.00000 3726G 1832G 1893G 49.18 0.98 106 > >> 115 hdd 3.64499 1.00000 3726G 2098G 1627G 56.32 1.12 121 > >> 116 hdd 3.64499 1.00000 3726G 1866G 1859G 50.10 1.00 108 > >> 117 hdd 3.64499 1.00000 3726G 1764G 1961G 47.36 0.94 102 > >> 118 hdd 3.64499 1.00000 3726G 1816G 1909G 48.76 0.97 105 > >> 119 hdd 3.64499 1.00000 3726G 1733G 1992G 46.53 0.93 100 > >> 120 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115 > >> 121 hdd 3.64499 1.00000 3726G 1657G 2068G 44.48 0.89 96 > >> 122 hdd 3.64499 1.00000 3726G 1715G 2010G 46.04 0.92 99 > >> 123 hdd 3.64499 1.00000 3726G 1974G 1751G 53.00 1.05 114 > >> 124 hdd 3.64499 1.00000 3726G 1731G 1994G 46.48 0.93 100 > >> 125 hdd 3.64499 1.00000 3726G 1897G 1828G 50.94 1.01 110 > >> 126 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115 > >> 127 hdd 3.64499 1.00000 3726G 1972G 1753G 52.93 1.05 114 > >> 128 hdd 3.64499 1.00000 3726G 1888G 1837G 50.68 1.01 109 > >> 129 hdd 3.64499 1.00000 3726G 2025G 1700G 54.35 1.08 117 > >> 130 hdd 3.64499 1.00000 3726G 1747G 1978G 46.90 0.93 101 > >> 131 hdd 3.64499 1.00000 3726G 2061G 1664G 55.32 1.10 119 > >> 132 hdd 3.64499 1.00000 3726G 2185G 1540G 58.65 1.17 126 > >> 133 hdd 3.64499 1.00000 3726G 1764G 1961G 47.35 0.94 102 > >> 134 hdd 3.64499 1.00000 3726G 1782G 1943G 47.84 0.95 103 > >> 135 hdd 3.64499 1.00000 3726G 1675G 2050G 44.96 0.89 97 > >> 136 hdd 3.64499 1.00000 3726G 1889G 1836G 50.70 1.01 109 > >> 137 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113 > >> 138 hdd 3.64499 1.00000 3726G 1694G 2031G 45.49 0.91 98 > >> 139 hdd 3.64499 1.00000 3726G 2114G 1611G 56.76 1.13 122 > >> 140 hdd 3.64499 1.00000 3726G 1726G 1999G 46.35 0.92 100 > >> 141 hdd 3.64499 1.00000 3726G 1988G 1737G 53.36 1.06 115 > >> 142 hdd 3.64499 1.00000 3726G 2181G 1544G 58.55 1.17 126 > >> 143 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107 > >> 144 hdd 3.64499 1.00000 3726G 2144G 1581G 57.56 1.15 124 > >> 145 hdd 3.64499 1.00000 3726G 1868G 1857G 50.14 1.00 108 > >> 146 hdd 3.64499 1.00000 3726G 2078G 1647G 55.77 1.11 120 > >> 147 hdd 3.64499 1.00000 3726G 1992G 1733G 53.49 1.06 115 > >> 148 hdd 3.64499 1.00000 3726G 1728G 1997G 46.39 0.92 100 > >> 149 hdd 3.64499 1.00000 3726G 2162G 1563G 58.03 1.15 125 > >> 150 hdd 3.64499 1.00000 3726G 2005G 1720G 53.83 1.07 116 > >> 151 hdd 3.64499 1.00000 3726G 1918G 1807G 51.49 1.02 111 > >> 152 hdd 3.64499 1.00000 3726G 1921G 1804G 51.57 1.03 111 > >> 153 hdd 3.64499 1.00000 3726G 1783G 1942G 47.86 0.95 103 > >> 154 hdd 3.64499 1.00000 3726G 2005G 1720G 53.82 1.07 116 > >> 155 hdd 3.64499 1.00000 3726G 1853G 1873G 49.73 0.99 107 > >> 156 hdd 3.64499 1.00000 3726G 1626G 2099G 43.65 0.87 94 > >> 157 hdd 3.64499 1.00000 3726G 1802G 1923G 48.38 0.96 104 > >> 158 hdd 3.64499 1.00000 3726G 1765G 1960G 47.39 0.94 102 > >> 159 hdd 3.64499 1.00000 3726G 1990G 1735G 53.42 1.06 115 > >> 160 hdd 3.64499 1.00000 3726G 1901G 1824G 51.02 1.02 110 > >> 161 hdd 3.64499 1.00000 3726G 1906G 1819G 51.16 1.02 110 > >> 162 hdd 3.64499 1.00000 3726G 1763G 1962G 47.33 0.94 102 > >> 163 hdd 3.64499 1.00000 3726G 1800G 1925G 48.33 0.96 104 > >> 164 hdd 3.64499 1.00000 3726G 1922G 1803G 51.61 1.03 111 > >> 165 hdd 3.64499 1.00000 3726G 1731G 1994G 46.46 0.92 100 > >> 166 hdd 3.64499 1.00000 3726G 2027G 1699G 54.40 1.08 117 > >> 167 hdd 3.64499 1.00000 3726G 1956G 1769G 52.50 1.04 113 > >> 168 hdd 3.64499 1.00000 3726G 1853G 1872G 49.74 0.99 107 > >> 169 hdd 3.64499 1.00000 3726G 1884G 1841G 50.59 1.01 109 > >> 170 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115 > >> 171 hdd 3.64499 1.00000 3726G 1898G 1827G 50.96 1.01 110 > >> 172 hdd 3.64499 1.00000 3726G 1781G 1944G 47.82 0.95 103 > >> 173 hdd 3.64499 1.00000 3726G 1628G 2097G 43.71 0.87 94 > >> 174 hdd 3.64499 1.00000 3726G 1992G 1733G 53.48 1.06 115 > >> 175 hdd 3.64499 1.00000 3726G 1905G 1820G 51.15 1.02 110 > >> 176 hdd 3.64499 1.00000 3726G 1709G 2016G 45.89 0.91 99 > >> 177 hdd 3.64499 1.00000 3726G 1991G 1734G 53.44 1.06 115 > >> 178 hdd 3.64499 1.00000 3726G 1973G 1752G 52.97 1.05 114 > >> 179 hdd 3.64499 1.00000 3726G 1574G 2151G 42.26 0.84 91 > >> 180 hdd 3.64499 1.00000 3726G 1917G 1808G 51.46 1.02 111 > >> 181 hdd 3.64499 1.00000 3726G 1937G 1788G 52.00 1.03 112 > >> 182 hdd 3.64499 1.00000 3726G 1989G 1736G 53.40 1.06 115 > >> 183 hdd 3.64499 1.00000 3726G 2039G 1686G 54.74 1.09 118 > >> 184 hdd 3.64499 1.00000 3726G 1868G 1857G 50.16 1.00 108 > >> 185 hdd 3.64499 1.00000 3726G 1954G 1771G 52.45 1.04 113 > >> 186 hdd 3.64499 1.00000 3726G 1850G 1875G 49.66 0.99 107 > >> 187 hdd 3.64499 1.00000 3726G 1851G 1874G 49.68 0.99 107 > >> 188 hdd 3.64499 1.00000 3726G 1814G 1911G 48.69 0.97 105 > >> 189 hdd 3.64499 1.00000 3726G 2286G 1439G 61.36 1.22 132 > >> 190 hdd 3.64499 1.00000 3726G 2060G 1665G 55.31 1.10 119 > >> 191 hdd 3.64499 1.00000 3726G 1836G 1889G 49.29 0.98 106 > >> 192 hdd 3.63869 1.00000 3726G 1721G 2004G 46.21 0.92 101 > >> 193 hdd 3.64499 1.00000 3726G 1953G 1772G 52.43 1.04 113 > >> 194 hdd 3.64499 1.00000 3726G 1541G 2184G 41.37 0.82 89 > >> 195 hdd 3.64499 1.00000 3726G 1837G 1888G 49.32 0.98 106 > >> 196 hdd 3.63899 1.00000 3726G 1705G 2020G 45.79 0.91 100 > >> 197 hdd 3.63899 1.00000 3726G 1743G 1982G 46.78 0.93 102 > >> 198 hdd 3.63899 1.00000 3726G 1982G 1743G 53.21 1.06 116 > >> 199 hdd 3.63899 1.00000 3726G 1382G 2343G 37.11 0.74 81 > >> 200 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111 > >> 201 hdd 3.63899 1.00000 3726G 1845G 1880G 49.54 0.99 108 > >> 202 hdd 3.63899 1.00000 3726G 1658G 2067G 44.51 0.89 97 > >> 203 hdd 3.63899 1.00000 3726G 2066G 1659G 55.47 1.10 121 > >> 204 hdd 3.63899 1.00000 3726G 1672G 2053G 44.90 0.89 98 > >> 205 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89 > >> 206 hdd 3.63899 1.00000 3726G 1796G 1929G 48.22 0.96 105 > >> 207 hdd 3.63899 1.00000 3726G 1555G 2170G 41.75 0.83 91 > >> 208 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102 > >> 209 hdd 3.63899 1.00000 3726G 2016G 1709G 54.11 1.08 118 > >> 210 hdd 3.63899 1.00000 3726G 2136G 1589G 57.34 1.14 125 > >> 211 hdd 3.63899 1.00000 3726G 1656G 2069G 44.46 0.88 97 > >> 212 hdd 3.63899 1.00000 3726G 1553G 2172G 41.71 0.83 91 > >> 213 hdd 3.63899 1.00000 3726G 1758G 1967G 47.19 0.94 103 > >> 214 hdd 3.63899 1.00000 3726G 1603G 2122G 43.03 0.86 94 > >> 215 hdd 3.63899 1.00000 3726G 1809G 1916G 48.57 0.97 106 > >> 216 hdd 3.63899 1.00000 3726G 1760G 1965G 47.26 0.94 103 > >> 217 hdd 3.63899 1.00000 3726G 1742G 1983G 46.77 0.93 102 > >> 218 hdd 3.63899 1.00000 3726G 1880G 1845G 50.46 1.00 110 > >> 219 hdd 3.63899 1.00000 3726G 1693G 2032G 45.45 0.90 99 > >> 220 hdd 3.63899 1.00000 3726G 2032G 1693G 54.56 1.09 119 > >> 221 hdd 3.63899 1.00000 3726G 1842G 1883G 49.44 0.98 108 > >> 222 hdd 3.63899 1.00000 3726G 1521G 2204G 40.84 0.81 89 > >> 223 hdd 3.63899 1.00000 3726G 1470G 2255G 39.46 0.79 86 > >> 224 hdd 3.63899 1.00000 3726G 1808G 1917G 48.53 0.97 106 > >> 225 hdd 3.63899 1.00000 3726G 1893G 1832G 50.83 1.01 111 > >> 226 hdd 3.63899 1.00000 3726G 1776G 1949G 47.69 0.95 104 > >> 227 hdd 3.63899 1.00000 3726G 1929G 1796G 51.78 1.03 113 > >> 228 hdd 3.63899 1.00000 3726G 1657G 2068G 44.48 0.89 97 > >> 229 hdd 3.63899 1.00000 3726G 1843G 1882G 49.47 0.98 108 > >> TOTAL 825T 414T 410T 50.24 > >> MIN/MAX VAR: 0.01/1.29 STDDEV: 9.22 > >> -------------------------------------------------------------- > >> > >> Am 20.10.18 um 20:35 schrieb Janne Johansson: > >>> Yes, if you have uneven sizes I guess you could end up in a situation > >>> where you have > >>> lots of 1TB OSDs and a number of 2TB OSD but pool replication forces > >>> the pool to have one > >>> PG replica on the 1TB OSD, then it would be possible to state "this > >>> pool cant write more than X G" > >>> but when it is full, there would be free space left on some of the > >>> 2TB-OSDs, but which the pool > >>> cant utilize. Probably same for uneven OSD hosts if you have those. > >>> > >>> Den lör 20 okt. 2018 kl 20:28 skrev Oliver Freyermuth > >>> <[email protected]>: > >>>> > >>>> Dear Janne, > >>>> > >>>> yes, of course. But since we only have two pools here, this can not > explain the difference. > >>>> The metadata is replicated (3 copies) across ssd drives, and we have > < 3 TB of total raw storage for that. > >>>> So looking at the raw space usage, we can ignore that. > >>>> > >>>> All the rest is used for the ceph_data pool. So the ceph_data pool, > in terms of raw storage, is about 50 % used. > >>>> > >>>> But in terms of storage shown for that pool, it's almost 63 % %USED. > >>>> So I guess this can purely be from bad balancing, correct? > >>>> > >>>> Cheers, > >>>> Oliver > >>>> > >>>> Am 20.10.18 um 19:49 schrieb Janne Johansson: > >>>>> Do mind that drives may have more than one pool on them, so RAW space > >>>>> is what it says, how much free space there is. Then the avail and > >>>>> %USED on per-pool stats will take replication into account, it can > >>>>> tell how much data you may write into that particular pool, given > that > >>>>> pools replication or EC settings. > >>>>> > >>>>> Den lör 20 okt. 2018 kl 19:09 skrev Oliver Freyermuth > >>>>> <[email protected]>: > >>>>>> > >>>>>> Dear Cephalopodians, > >>>>>> > >>>>>> as many others, I'm also a bit confused by "ceph df" output > >>>>>> in a pretty straightforward configuration. > >>>>>> > >>>>>> We have a CephFS (12.2.7) running, with 4+2 EC profile. > >>>>>> > >>>>>> I get: > >>>>>> > ---------------------------------------------------------------------------- > >>>>>> # ceph df > >>>>>> GLOBAL: > >>>>>> SIZE AVAIL RAW USED %RAW USED > >>>>>> 824T 410T 414T 50.26 > >>>>>> POOLS: > >>>>>> NAME ID USED %USED MAX AVAIL > OBJECTS > >>>>>> cephfs_metadata 1 452M 0.05 860G > 365774 > >>>>>> cephfs_data 2 275T 62.68 164T > 75056403 > >>>>>> > ---------------------------------------------------------------------------- > >>>>>> > >>>>>> So about 50 % of raw space are used, but already ~63 % of > filesystem space are used. > >>>>>> Is this purely from imperfect balancing? > >>>>>> In "ceph osd df", I do indeed see OSD usages spreading from 65.02 % > usage down to 37.12 %. > >>>>>> > >>>>>> We did not yet use the balancer plugin. > >>>>>> We don't have any pre-luminous clients. > >>>>>> In that setup, I take it that "upmap" mode would be recommended - > correct? > >>>>>> Any "gotchas" using that on luminous? > >>>>>> > >>>>>> Cheers, > >>>>>> Oliver > >>>>>> > >>>>>> _______________________________________________ > >>>>>> ceph-users mailing list > >>>>>> [email protected] > >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > > > > > > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
