What does your crush map look like? Also a `ceph df` output. You're optimizing your map for pool #5, if there are other pools with a significant amount of data, then your going to be off on your cluster balance.
A big question for balancing a cluster is how big are your PGs? If your primary data pool has PGs that are 100GB in size, then even if you balanced the cluster such that all of the osds were within 1 PG of each other (assuming your average OSD size <1TB), then your OSDs would be 10% apart in disk usage from each other and 100GB. On Thu, Oct 26, 2017 at 4:44 AM Stefan Priebe - Profihost AG < [email protected]> wrote: > Hello, > > while trying to optimize a ceph cluster running jewel i get the > following output: > 2017-10-26 10:43:27,615 argv = optimize --crushmap > /home/spriebe/ceph.report --out-path /home/spriebe/optimized.crush > --pool 5 --pool=5 --choose-args=5 --replication-count=3 --pg-num=4096 > --pgp-num=4096 --rule=data --out-version=j --no-positions > 2017-10-26 10:43:27,646 root optimizing > 2017-10-26 10:43:29,329 root already optimized > 2017-10-26 10:43:29,337 cloud1-1475 optimizing > 2017-10-26 10:43:29,348 cloud1-1474 optimizing > 2017-10-26 10:43:29,353 cloud1-1473 optimizing > 2017-10-26 10:43:29,361 cloud1-1467 optimizing > 2017-10-26 10:43:30,118 cloud1-1473 already optimized > 2017-10-26 10:43:30,126 cloud1-1472 optimizing > 2017-10-26 10:43:30,177 cloud1-1474 already optimized > 2017-10-26 10:43:30,178 cloud1-1467 already optimized > 2017-10-26 10:43:30,185 cloud1-1471 optimizing > 2017-10-26 10:43:30,193 cloud1-1470 optimizing > 2017-10-26 10:43:30,301 cloud1-1475 already optimized > 2017-10-26 10:43:30,310 cloud1-1469 optimizing > 2017-10-26 10:43:30,855 cloud1-1472 already optimized > 2017-10-26 10:43:30,864 cloud1-1468 optimizing > 2017-10-26 10:43:31,020 cloud1-1470 already optimized > 2017-10-26 10:43:31,075 cloud1-1471 already optimized > 2017-10-26 10:43:31,079 cloud1-1469 already optimized > 2017-10-26 10:43:31,460 cloud1-1468 already optimized > > > But this one is heavily inbalanced if you look at the AVAIL GB. > > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 31 1.00000 1.00000 931G 756G 174G 81.24 1.24 204 > 32 1.00000 1.00000 931G 750G 180G 80.58 1.23 202 > 33 1.00000 1.00000 931G 774G 157G 83.13 1.27 209 > 34 1.00000 1.00000 931G 743G 187G 79.89 1.22 200 > 48 1.00000 1.00000 931G 780G 150G 83.79 1.28 210 > 49 1.00000 1.00000 931G 778G 152G 83.64 1.27 209 > 50 0.79999 1.00000 819G 631G 188G 77.00 1.17 170 > 21 1.00000 0.95001 931G 729G 201G 78.32 1.19 197 > 22 1.00000 1.00000 931G 756G 174G 81.28 1.24 204 > 23 1.00000 1.00000 931G 778G 152G 83.57 1.27 210 > 24 1.00000 0.95001 931G 760G 171G 81.63 1.24 205 > 46 1.00000 1.00000 931G 756G 174G 81.30 1.24 203 > 47 1.00000 1.00000 931G 768G 162G 82.57 1.26 207 > 55 0.79999 1.00000 819G 602G 217G 73.46 1.12 162 > 11 1.00000 1.00000 931G 628G 302G 67.47 1.03 169 > 12 1.00000 1.00000 931G 630G 300G 67.68 1.03 170 > 13 1.00000 1.00000 931G 629G 301G 67.63 1.03 170 > 14 1.00000 1.00000 931G 644G 286G 69.20 1.05 173 > 45 1.00000 1.00000 931G 635G 295G 68.23 1.04 171 > 56 0.79999 1.00000 819G 509G 309G 62.18 0.95 137 > 40 1.72670 1.00000 1768G 774G 994G 43.78 0.67 209 > 51 1.00000 0.95001 931G 784G 146G 84.26 1.28 212 > 52 1.00000 1.00000 931G 768G 162G 82.52 1.26 207 > 53 1.00000 1.00000 931G 752G 178G 80.78 1.23 203 > 54 1.00000 1.00000 931G 750G 180G 80.58 1.23 202 > 38 1.00000 1.00000 931G 768G 162G 82.58 1.26 207 > 39 1.00000 1.00000 931G 768G 162G 82.51 1.26 207 > 57 0.79999 1.00000 819G 616G 203G 75.17 1.15 167 > 43 1.00000 1.00000 931G 636G 294G 68.41 1.04 171 > 44 1.00000 1.00000 931G 630G 300G 67.70 1.03 170 > 36 1.00000 1.00000 931G 626G 304G 67.33 1.03 170 > 37 1.00000 1.00000 931G 637G 293G 68.43 1.04 171 > 58 0.79999 1.00000 819G 508G 311G 62.03 0.95 137 > 41 0.87000 1.00000 888G 549G 339G 61.79 0.94 148 > 42 1.72670 1.00000 1768G 382G 1385G 21.65 0.33 103 > 65 1.72670 1.00000 1768G 383G 1384G 21.69 0.33 103 > 1 1.00000 1.00000 931G 760G 170G 81.70 1.24 205 > 2 1.00000 1.00000 931G 768G 162G 82.57 1.26 207 > 3 1.00000 0.95001 931G 732G 198G 78.71 1.20 197 > 30 1.00000 1.00000 931G 771G 159G 82.84 1.26 209 > 35 1.00000 1.00000 931G 751G 179G 80.70 1.23 202 > 59 0.79999 1.00000 819G 599G 220G 73.08 1.11 162 > 0 0.84999 1.00000 874G 662G 211G 75.81 1.16 179 > 4 1.00000 1.00000 931G 633G 297G 68.07 1.04 171 > 5 1.00000 1.00000 931G 632G 298G 67.94 1.04 171 > 60 0.79999 1.00000 819G 509G 310G 62.12 0.95 137 > 29 0.87000 1.00000 888G 553G 335G 62.29 0.95 149 > 7 0.87000 1.00000 888G 554G 333G 62.44 0.95 149 > 28 0.87000 1.00000 888G 551G 336G 62.09 0.95 149 > 64 1.73000 1.00000 1768G 1088G 679G 61.57 0.94 293 > 6 1.72670 1.00000 1768G 472G 1295G 26.71 0.41 127 > 66 1.72670 1.00000 1768G 475G 1292G 26.88 0.41 128 > 8 1.00000 1.00000 931G 625G 305G 67.14 1.02 169 > 9 1.00000 1.00000 931G 631G 299G 67.82 1.03 170 > 61 0.79999 1.00000 819G 507G 312G 61.90 0.94 137 > 15 0.87000 1.00000 888G 550G 338G 61.89 0.94 148 > 26 0.87000 1.00000 888G 551G 337G 62.04 0.95 149 > 27 0.84999 1.00000 874G 541G 333G 61.90 0.94 146 > 63 1.73000 1.00000 1768G 1101G 666G 62.31 0.95 297 > 10 1.72670 1.00000 1768G 483G 1284G 27.34 0.42 131 > 67 1.72670 1.00000 1768G 482G 1285G 27.31 0.42 130 > 16 1.00000 1.00000 931G 786G 144G 84.46 1.29 212 > 17 1.00000 1.00000 931G 783G 147G 84.12 1.28 211 > 18 1.00000 1.00000 931G 747G 183G 80.28 1.22 201 > 19 1.00000 1.00000 931G 758G 172G 81.50 1.24 205 > 20 1.00000 1.00000 931G 759G 171G 81.59 1.24 205 > 25 1.00000 1.00000 931G 766G 164G 82.35 1.25 207 > 62 0.79999 1.00000 819G 620G 199G 75.68 1.15 166 > 31 1.00000 1.00000 931G 756G 174G 81.24 1.24 204 > 32 1.01900 1.00000 931G 750G 180G 80.58 1.23 202 > 33 1.00000 1.00000 931G 774G 157G 83.13 1.27 209 > 34 0.92699 1.00000 931G 743G 187G 79.89 1.22 200 > 48 1.00000 1.00000 931G 780G 150G 83.79 1.28 210 > 49 1.02399 1.00000 931G 778G 152G 83.64 1.27 209 > 50 0.82999 1.00000 819G 631G 188G 77.00 1.17 170 > 21 1.12793 0.95001 931G 729G 201G 78.32 1.19 197 > 22 0.93199 1.00000 931G 756G 174G 81.28 1.24 204 > 23 0.94606 1.00000 931G 778G 152G 83.57 1.27 210 > 24 0.95599 0.95001 931G 760G 171G 81.63 1.24 205 > 46 1.01500 1.00000 931G 756G 174G 81.30 1.24 203 > 47 0.99500 1.00000 931G 768G 162G 82.57 1.26 207 > 55 0.82899 1.00000 819G 602G 217G 73.46 1.12 162 > 11 0.78575 1.00000 931G 628G 302G 67.47 1.03 169 > 12 0.89369 1.00000 931G 630G 300G 67.68 1.03 170 > 13 0.84279 1.00000 931G 629G 301G 67.63 1.03 170 > 14 0.77834 1.00000 931G 644G 286G 69.20 1.05 173 > 45 0.83331 1.00000 931G 635G 295G 68.23 1.04 171 > 56 0.74358 1.00000 819G 509G 309G 62.18 0.95 137 > 40 0.96149 1.00000 1768G 774G 994G 43.78 0.67 209 > 51 1.03899 0.95001 931G 784G 146G 84.26 1.28 212 > 52 0.98000 1.00000 931G 768G 162G 82.52 1.26 207 > 53 1.01900 1.00000 931G 752G 178G 80.78 1.23 203 > 54 0.98000 1.00000 931G 750G 180G 80.58 1.23 202 > 38 0.94199 1.00000 931G 768G 162G 82.58 1.26 207 > 39 1.03899 1.00000 931G 768G 162G 82.51 1.26 207 > 57 0.79999 1.00000 819G 616G 203G 75.17 1.15 167 > 43 0.77654 1.00000 931G 636G 294G 68.41 1.04 171 > 44 0.85941 1.00000 931G 630G 300G 67.70 1.03 170 > 36 0.85104 1.00000 931G 626G 304G 67.33 1.03 170 > 37 0.80946 1.00000 931G 637G 293G 68.43 1.04 171 > 58 0.67404 1.00000 819G 508G 311G 62.03 0.95 137 > 41 0.68245 1.00000 888G 549G 339G 61.79 0.94 148 > 42 0.49387 1.00000 1768G 382G 1385G 21.65 0.33 103 > 65 0.52313 1.00000 1768G 383G 1384G 21.69 0.33 103 > 1 0.96100 1.00000 931G 760G 170G 81.70 1.24 205 > 2 0.94199 1.00000 931G 768G 162G 82.57 1.26 207 > 3 1.03499 0.95001 931G 732G 198G 78.71 1.20 197 > 30 1.00000 1.00000 931G 771G 159G 82.84 1.26 209 > 35 1.07399 1.00000 931G 751G 179G 80.70 1.23 202 > 59 0.87000 1.00000 819G 599G 220G 73.08 1.11 162 > 0 0.76900 1.00000 874G 662G 211G 75.81 1.16 179 > 4 0.73238 1.00000 931G 633G 297G 68.07 1.04 171 > 5 0.88098 1.00000 931G 632G 298G 67.94 1.04 171 > 60 0.67899 1.00000 819G 509G 310G 62.12 0.95 137 > 29 0.72182 1.00000 888G 553G 335G 62.29 0.95 149 > 7 0.71684 1.00000 888G 554G 333G 62.44 0.95 149 > 28 0.64395 1.00000 888G 551G 336G 62.09 0.95 149 > 64 1.46759 1.00000 1768G 1088G 679G 61.57 0.94 293 > 6 0.59332 1.00000 1768G 472G 1295G 26.71 0.41 127 > 66 0.68506 1.00000 1768G 475G 1292G 26.88 0.41 128 > 8 0.89154 1.00000 931G 625G 305G 67.14 1.02 169 > 9 0.83815 1.00000 931G 631G 299G 67.82 1.03 170 > 61 0.64093 1.00000 819G 507G 312G 61.90 0.94 137 > 15 0.74059 1.00000 888G 550G 338G 61.89 0.94 148 > 26 0.65033 1.00000 888G 551G 337G 62.04 0.95 149 > 27 0.61040 1.00000 874G 541G 333G 61.90 0.94 146 > 63 1.44260 1.00000 1768G 1101G 666G 62.31 0.95 297 > 10 0.65889 1.00000 1768G 483G 1284G 27.34 0.42 131 > 67 0.60854 1.00000 1768G 482G 1285G 27.31 0.42 130 > 16 0.98000 1.00000 931G 786G 144G 84.46 1.29 212 > 17 1.11299 1.00000 931G 783G 147G 84.12 1.28 211 > 18 1.12399 1.00000 931G 747G 183G 80.28 1.22 201 > 19 0.87099 1.00000 931G 758G 172G 81.50 1.24 205 > 20 0.95099 1.00000 931G 759G 171G 81.59 1.24 205 > 25 1.04700 1.00000 931G 766G 164G 82.35 1.25 207 > 62 0.71399 1.00000 819G 620G 199G 75.68 1.15 166 > TOTAL 69475G 45596G 23878G 65.63 > MIN/MAX VAR: 0.33/1.29 STDDEV: 16.64 > > crush tool is 1.0.35 > > Greets, > Stefan > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
