On 12/3/19 1:30 PM, Lars Täuber wrote:
here it comes:
$ ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS TYPE NAME
-1 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB
66.45 1.00 - root default
-25 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB
66.45 1.00 - room PRZ
-26 195.40730 - 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB
66.45 1.00 - row rechts
-27 83.74599 - 84 TiB 57 TiB 56 TiB 25 GiB 206 GiB 27 TiB
67.51 1.02 - rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 8.4 GiB 66 GiB 10 TiB
62.80 0.95 - host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB
62.48 0.94 163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB
62.47 0.94 163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB
62.53 0.94 163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB
62.90 0.95 164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB
67.32 1.01 176 up osd.37
4 ssd 0.34239 1.00000 351 GiB 11 GiB 187 MiB 8.3 GiB 2.0 GiB 340 GiB
3.01 0.05 110 up osd.4
-13 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 10 TiB
62.64 0.94 - host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB
62.49 0.94 163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB
62.49 0.94 163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB
62.43 0.94 163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB
62.13 0.94 162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB
67.36 1.01 176 up osd.40
33 ssd 0.34239 1.00000 351 GiB 11 GiB 201 MiB 8.2 GiB 2.2 GiB 340 GiB
3.02 0.05 110 up osd.33
-22 27.91533 - 28 TiB 22 TiB 21 TiB 8.1 GiB 74 GiB 6.4 TiB
77.10 1.16 - host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB
77.59 1.17 203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB
78.40 1.18 205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB
75.80 1.14 198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB
76.13 1.15 199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB
82.18 1.24 215 up osd.30
36 ssd 0.34239 1.00000 351 GiB 10 GiB 184 MiB 8.1 GiB 2.0 GiB 340 GiB
2.92 0.04 110 up osd.36
-28 55.83066 - 56 TiB 35 TiB 34 TiB 17 GiB 132 GiB 21 TiB
62.36 0.94 - rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 8.2 GiB 66 GiB 11 TiB
62.27 0.94 - host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB
62.08 0.93 162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB
62.13 0.93 162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB
61.77 0.93 161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB
61.75 0.93 161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB
67.31 1.01 176 up osd.38
31 ssd 0.34239 1.00000 351 GiB 11 GiB 166 MiB 8.1 GiB 2.4 GiB 340 GiB
3.04 0.05 110 up osd.31
-16 27.91533 - 28 TiB 17 TiB 17 TiB 8.7 GiB 66 GiB 10 TiB
62.44 0.94 - host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB
62.15 0.94 162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB
62.16 0.94 162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB
62.14 0.94 162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB
62.12 0.93 162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB
67.31 1.01 176 up osd.41
34 ssd 0.34239 1.00000 351 GiB 11 GiB 192 MiB 8.7 GiB 1.8 GiB 340 GiB
3.04 0.05 109 up osd.34
-29 55.83066 - 56 TiB 38 TiB 38 TiB 16 GiB 138 GiB 17 TiB
68.95 1.04 - rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 8.1 GiB 63 GiB 11 TiB
61.02 0.92 - host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB
60.63 0.91 158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB
60.19 0.91 157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB
60.27 0.91 157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB
60.28 0.91 157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB
67.34 1.01 176 up osd.39
32 ssd 0.34239 1.00000 351 GiB 10 GiB 271 MiB 8.1 GiB 1.8 GiB 341 GiB
2.88 0.04 109 up osd.32
-19 27.91533 - 28 TiB 21 TiB 21 TiB 8.1 GiB 74 GiB 6.5 TiB
76.89 1.16 - host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB
72.79 1.10 190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB
81.79 1.23 214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB
80.29 1.21 210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB
77.31 1.16 202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB
76.86 1.16 201 up osd.29
35 ssd 0.34239 1.00000 351 GiB 10 GiB 208 MiB 8.1 GiB 1.9 GiB 340 GiB
2.89 0.04 110 up osd.35
TOTAL 195 TiB 130 TiB 128 TiB 58 GiB 476 GiB 66 TiB
66.45
MIN/MAX VAR: 0.04/1.24 STDDEV: 26.74
better only for the class hdd
$ ceph osd df tree class hdd
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS TYPE NAME
-1 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB
67.24 1.00 - root default
-25 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB
67.24 1.00 - room PRZ
-26 195.40730 - 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB
67.24 1.00 - row rechts
-27 83.74599 - 83 TiB 57 TiB 56 TiB 81 MiB 200 GiB 26 TiB
68.31 1.02 - rack 1-eins
-3 27.91533 - 28 TiB 18 TiB 17 TiB 31 MiB 64 GiB 10 TiB
63.54 0.94 - host onode1
0 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.8 MiB 14 GiB 2.1 TiB
62.48 0.93 163 up osd.0
1 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 12 GiB 2.1 TiB
62.47 0.93 163 up osd.1
2 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 7.1 MiB 12 GiB 2.1 TiB
62.53 0.93 163 up osd.2
3 hdd 5.51459 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 7.5 MiB 12 GiB 2.0 TiB
62.90 0.94 164 up osd.3
37 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 6.4 MiB 13 GiB 1.8 TiB
67.32 1.00 176 up osd.37
-13 27.91533 - 28 TiB 17 TiB 17 TiB 21 MiB 64 GiB 10 TiB
63.38 0.94 - host onode4
13 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB
62.49 0.93 163 up osd.13
14 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.2 MiB 13 GiB 2.1 TiB
62.49 0.93 163 up osd.14
15 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.4 MiB 12 GiB 2.1 TiB
62.43 0.93 163 up osd.15
16 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 12 GiB 2.1 TiB
62.13 0.92 162 up osd.16
40 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.6 MiB 13 GiB 1.8 TiB
67.36 1.00 176 up osd.40
-22 27.91533 - 28 TiB 22 TiB 21 TiB 28 MiB 72 GiB 6.1 TiB
78.02 1.16 - host onode7
25 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 7.2 MiB 14 GiB 1.2 TiB
77.59 1.15 203 up osd.25
26 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.7 MiB 14 GiB 1.2 TiB
78.40 1.17 205 up osd.26
27 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 3.8 MiB 14 GiB 1.3 TiB
75.80 1.13 198 up osd.27
28 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.1 TiB 4.5 MiB 14 GiB 1.3 TiB
76.13 1.13 199 up osd.28
30 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 8.2 MiB 15 GiB 1006 GiB
82.18 1.22 215 up osd.30
-28 55.83066 - 55 TiB 35 TiB 34 TiB 38 MiB 128 GiB 20 TiB
63.09 0.94 - rack 2-zwei
-7 27.91533 - 28 TiB 17 TiB 17 TiB 18 MiB 63 GiB 10 TiB
63.01 0.94 - host onode2
5 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 12 GiB 2.1 TiB
62.08 0.92 162 up osd.5
6 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.9 MiB 13 GiB 2.1 TiB
62.13 0.92 162 up osd.6
7 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.7 MiB 12 GiB 2.1 TiB
61.77 0.92 161 up osd.7
8 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 3.2 MiB 12 GiB 2.1 TiB
61.75 0.92 161 up osd.8
38 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.7 MiB 14 GiB 1.8 TiB
67.31 1.00 176 up osd.38
-16 27.91533 - 28 TiB 17 TiB 17 TiB 20 MiB 65 GiB 10 TiB
63.18 0.94 - host onode5
17 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4 MiB 12 GiB 2.1 TiB
62.15 0.92 162 up osd.17
18 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.0 MiB 13 GiB 2.1 TiB
62.16 0.92 162 up osd.18
19 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 4.5 MiB 13 GiB 2.1 TiB
62.14 0.92 162 up osd.19
20 hdd 5.51459 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 3.5 MiB 13 GiB 2.1 TiB
62.12 0.92 162 up osd.20
41 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 3.9 MiB 14 GiB 1.8 TiB
67.31 1.00 176 up osd.41
-29 55.83066 - 55 TiB 38 TiB 38 TiB 49 MiB 134 GiB 17 TiB
69.77 1.04 - rack 3-drei
-10 27.91533 - 28 TiB 17 TiB 17 TiB 22 MiB 62 GiB 11 TiB
61.74 0.92 - host onode3
9 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 12 GiB 2.2 TiB
60.63 0.90 158 up osd.9
10 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.1 MiB 12 GiB 2.2 TiB
60.19 0.90 157 up osd.10
11 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 6.7 MiB 12 GiB 2.2 TiB
60.27 0.90 157 up osd.11
12 hdd 5.51459 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4.1 MiB 12 GiB 2.2 TiB
60.28 0.90 157 up osd.12
39 hdd 5.51459 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4.6 MiB 13 GiB 1.8 TiB
67.34 1.00 176 up osd.39
-19 27.91533 - 28 TiB 21 TiB 21 TiB 27 MiB 72 GiB 6.1 TiB
77.81 1.16 - host onode6
21 hdd 5.51459 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6.2 MiB 13 GiB 1.5 TiB
72.79 1.08 190 up osd.21
22 hdd 5.51459 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 5.1 MiB 16 GiB 1.0 TiB
81.79 1.22 214 up osd.22
23 hdd 5.51459 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.4 MiB 16 GiB 1.1 TiB
80.29 1.19 210 up osd.23
24 hdd 5.51459 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6.7 MiB 14 GiB 1.3 TiB
77.31 1.15 202 up osd.24
29 hdd 5.51459 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4.6 MiB 14 GiB 1.3 TiB
76.86 1.14 201 up osd.29
TOTAL 193 TiB 130 TiB 128 TiB 169 MiB 462 GiB 63 TiB
67.24
MIN/MAX VAR: 0.90/1.22 STDDEV: 7.17
ceph osd pool ls detail
pool 1 'cephfs_data' erasure size 6 min_size 5 crush_rule 1 object_hash
rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 20353 lfor
0/0/2366 flags hashpspool,ec_overwrites,selfmanaged_snaps max_bytes
119457034600410 stripe_width 16384 target_size_ratio 0.85 application cephfs
removed_snaps
[2~4,7~27,2f~1e,4f~1f,6f~39,a9~5,af~1,b1~1,b3~1,b5~1,b7~1,b9~1,bb~1,bd~1,bf~1,c1~1,c3~1,c5~1,c7~1,c9~1]
pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 2 object_hash
rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 261 lfor 0/0/259
flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16
recovery_priority 5 target_size_ratio 0.3 application cephfs
$ ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "cephfs_data",
"ruleset": 1,
"type": 3,
"min_size": 3,
"max_size": 6,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -2,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_indep",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
},
{
"rule_id": 2,
"rule_name": "rep_3_ssd",
"ruleset": 2,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -6,
"item_name": "default~ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
Lars, you have too much PG's for this OSD's. I suggest to disable PG
autoscaler and:
- reduce number of PG's for cephfs_metada pool to something like 16 PG's.
- reduce number of PG's for cephfs_data to something like 512.
- update crush rule for cephfs_metadata pool - set domain to 'rack'
instead 'host'.
Also please paste your `ceph osd tree`.
k
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
