Hello,
I'm seeing something that seems to be odd behavior when reweighting
OSDs. I've just upgraded to 12.2.5 and am adding in a new osd server to
the cluster. I gradually weight the 10TB OSDs into the cluster by doing
a +1, letting things backfill for a while, then +1 until I reach my
desired weight. This hasn't been a problem in the past, a proportionate
amount of PGs would get remapped, peer and activate across this cluster.
Now on 12.2.5 when I do this, almost all PGs peer and reactivate.
Sometimes it recovers within a minute, other times it takes longer, this
last time actually saw some OSDs on the new node crash and caused a
longer time for peering/activating. Regardless of recovery time, this is
a fairly violent reaction to reweighting.
Has anyone else seen behavior like this or have any ideas what's going on?
For example...
[root@sephmon1 ~]# ceph -s
cluster:
id: bc2a1488-74f8-4d87-b2f6-615ae26bf7c9
health: HEALTH_OK
services:
mon: 3 daemons, quorum sephmon1,sephmon2,sephmon3
mgr: sephmon2(active), standbys: sephmon1, sephmon3
mds: cephfs1-1/1/1 up {0=sephmon1=up:active}, 1 up:standby
osd: 789 osds: 789 up, 789 in
data:
pools: 7 pools, 39168 pgs
objects: 74046k objects, 2989 TB
usage: 3756 TB used, 1517 TB / 5273 TB avail
pgs: 39137 active+clean
26 active+clean+scrubbing+deep
5 active+clean+scrubbing
io:
client: 3522 MB/s rd, 118 MB/s wr, 1295 op/s rd, 833 op/s wr
[root@sephmon1 ~]# for i in {771..779}; do ceph osd crush reweight
osd.${i} 6.5; done
reweighted item id 771 name 'osd.771' to 6.5 in crush map
reweighted item id 772 name 'osd.772' to 6.5 in crush map
reweighted item id 773 name 'osd.773' to 6.5 in crush map
reweighted item id 774 name 'osd.774' to 6.5 in crush map
reweighted item id 775 name 'osd.775' to 6.5 in crush map
reweighted item id 776 name 'osd.776' to 6.5 in crush map
reweighted item id 777 name 'osd.777' to 6.5 in crush map
reweighted item id 778 name 'osd.778' to 6.5 in crush map
reweighted item id 779 name 'osd.779' to 6.5 in crush map
[root@sephmon1 ~]# ceph -s
cluster:
id: bc2a1488-74f8-4d87-b2f6-615ae26bf7c9
health: HEALTH_WARN
2 osds down
78219/355096089 objects misplaced (0.022%)
Reduced data availability: 668 pgs inactive, 1920 pgs down,
551 pgs peering, 29 pgs incomplete
Degraded data redundancy: 803425/355096089 objects degraded
(0.226%), 204 pgs degraded
3 slow requests are blocked > 32 sec
services:
mon: 3 daemons, quorum sephmon1,sephmon2,sephmon3
mgr: sephmon2(active), standbys: sephmon1, sephmon3
mds: cephfs1-1/1/1 up {0=sephmon1=up:active}, 1 up:standby
osd: 789 osds: 787 up, 789 in; 257 remapped pgs
data:
pools: 7 pools, 39168 pgs
objects: 73964k objects, 2985 TB
usage: 3756 TB used, 1517 TB / 5273 TB avail
pgs: 0.028% pgs unknown
94.904% pgs not active
803425/355096089 objects degraded (0.226%)
78219/355096089 objects misplaced (0.022%)
20215 peering
14335 activating
1882 active+clean
1788 down
205 remapped+peering
167 stale+peering
142 activating+undersized+degraded
127 activating+undersized
126 stale+down
57 active+undersized+degraded
39 stale+active+clean
27 incomplete
17 activating+remapped
11 unknown
7 stale+activating
6 down+remapped
3 stale+activating+undersized
2 stale+incomplete
2 active+undersized
2 stale+activating+undersized+degraded
1 activating+undersized+degraded+remapped
1 stale+active+undersized+degraded
1 remapped
1 active+clean+scrubbing
1 active+undersized+degraded+remapped+backfill_wait
1 stale+remapped+peering
1 active+clean+remapped
1 active+remapped+backfilling
io:
client: 3896 GB/s rd, 339 GB/s wr, 8004 kop/s rd, 320 kop/s wr
recovery: 726 GB/s, 11172 objects/s
Thanks,
Kevin
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
