Same here,
Nautilus 14.2.2.
Evacuate one host and join another one at the same time and all is
unbalance.
Best
De: ceph-users <ceph-users-boun...@lists.ceph.com> En nombre de David
Herselman
Enviado el: lunes, 29 de julio de 2019 11:31
Para: ceph-users@lists.ceph.com
Asunto: [ceph-users] Ceph Nautilus - can't balance due to degraded state
Hi,
We appear to be stuck in a proverbial chicken and egg situation. Degraded
placement groups won't backfill as OSDs are near full and we can't run the
balancer as some placement groups are degraded.
We upgraded Ceph from Luminous 12.2.12 to Nautilus 14.2.1 on a cluster used
for backup services. We are in the process of migrating data (nearly
complete), after which we'll be able to repurpose the old systems as
additional Ceph OSD nodes. Our cluster was subsequently at about 75%
utilisation and the balancer module together with upmap did a great job.
We've historically been very conservative with placement group numbering,
considering that smaller drives generally get replaced with much larger ones
and the PGs per OSD subsequently grow to problematic levels.
The upgrade process was so extremely painless that we also enabled the
pg_autoscaler module which subsequently marked 75% of the data as miss
placed, but also degraded various placement groups. The result is now that
we have many placement groups marked as nearfull, but can't run the balancer
as some placement groups are in a degraded state.
Is there a way we can override the degraded check and force the balancer to
redistribute PGs; or could we manually adjust OSDs to have the same effect?
Is there alternatively a way that we can get Ceph to first heal the degraded
PGs and only then work on the miss placed ones?
There are only 3 RBD images in this cluster, a 80GB operating system image
in a replicated SSD pool, a 150TB erasure coded image and a relatively tiny
replicated SSD caching tier for the EC pool.
[admin@kvm7e ~]# ceph osd lspools
1 rbd_ssd
5 cephfs_data
6 cephfs_metadata
7 rbd_hdd
8 ec_hdd
9 rbd_hdd_cache
10 ec_hdd_cache
[admin@kvm7e ~]# for f in `ceph osd lspools | cut -d\ -f2`; do ceph osd
pool set $f pg_autoscale_mode on; done;
set pool 1 pg_autoscale_mode to on
set pool 5 pg_autoscale_mode to on
set pool 6 pg_autoscale_mode to on
set pool 7 pg_autoscale_mode to on
set pool 8 pg_autoscale_mode to on
set pool 9 pg_autoscale_mode to on
set pool 10 pg_autoscale_mode to on
Concerning was that Ceph marked OSDs are near full although this is by
default only when an OSD reaches 85% utilisation. I presume Ceph projects
the resulting storage utilisation based on the weighting set by the
balancer?
[admin@kvm7e ~]# ceph health detail
HEALTH_ERR noout flag(s) set; 6 nearfull osd(s); 4 pool(s) nearfull; Reduced
data availability: 2 pgs inactive; Degraded data redundancy (low space): 4
pgs backfill_toofull
OSDMAP_FLAGS noout flag(s) set
OSD_NEARFULL 6 nearfull osd(s)
osd.100 is near full
osd.101 is near full
osd.102 is near full
osd.103 is near full
osd.104 is near full
osd.105 is near full
POOL_NEARFULL 4 pool(s) nearfull
pool 'cephfs_data' is nearfull
pool 'cephfs_metadata' is nearfull
pool 'rbd_hdd' is nearfull
pool 'ec_hdd' is nearfull
PG_AVAILABILITY Reduced data availability: 2 pgs inactive
pg 7.1e is stuck inactive for 437.102346, current state
clean+premerge+peered, last acting [303,104,405]
pg 7.3e is stuck inactive for 436.965670, current state
remapped+premerge+backfill_wait+peered, last acting [405,104,301]
PG_DEGRADED_FULL Degraded data redundancy (low space): 4 pgs
backfill_toofull
pg 8.c8 is active+remapped+backfill_wait+backfill_toofull, acting
[305,104,404,504,203]
pg 8.1bb is active+remapped+backfill_wait+backfill_toofull, acting
[505,204,102,304,404]
pg 8.326 is active+remapped+backfill_wait+backfill_toofull, acting
[302,504,402,103,202]
pg 8.3e0 is active+remapped+backfill_wait+backfill_toofull, acting
[202,402,103,305,505]
[admin@kvm7e ~]# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS
100 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 9.2 MiB 2.5 GiB 490
GiB 73.71 1.01 59 up
101 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 4.2 MiB 2.5 GiB 489
GiB 73.77 1.01 58 up
102 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 21 MiB 7.4 GiB 1.4
TiB 73.63 1.01 175 up
103 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 20 MiB 7.4 GiB 1.4
TiB 73.49 1.00 176 up
104 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 28 MiB 13 GiB 2.2
TiB 75.68 1.03 304 up
105 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 23 MiB 13 GiB 2.2
TiB 75.63 1.03 301 up
200 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 4.1 MiB 2.5 GiB 492
GiB 73.61 1.01 59 up
201 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 4.2 MiB 2.5 GiB 491
GiB 73.65 1.01 58 up
202 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 21 MiB 7.4 GiB 1.4
TiB 73.60 1.01 175 up
203 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 18 MiB 7.4 GiB 1.4
TiB 73.51 1.00 175 up
204 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2
TiB 75.65 1.03 301 up
205 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 41 MiB 13 GiB 2.2
TiB 75.71 1.03 302 up
300 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 6.1 MiB 2.5 GiB 490
GiB 73.68 1.01 58 up
301 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 13 MiB 2.5 GiB 490
GiB 73.72 1.01 59 up
302 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 13 MiB 7.4 GiB 1.4
TiB 73.57 1.01 174 up
303 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 22 MiB 7.4 GiB 1.4
TiB 73.56 1.01 177 up
304 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 42 MiB 13 GiB 2.2
TiB 75.68 1.03 302 up
305 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2
TiB 75.67 1.03 304 up
400 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 8.6 MiB 2.5 GiB 489
GiB 73.75 1.01 59 up
401 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 8.7 MiB 2.5 GiB 493
GiB 73.56 1.01 59 up
402 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 25 MiB 7.4 GiB 1.4
TiB 73.51 1.00 176 up
403 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 18 MiB 7.4 GiB 1.4
TiB 73.59 1.01 175 up
404 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2
TiB 75.68 1.03 298 up
405 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 MiB 13 GiB 2.2
TiB 75.69 1.03 301 up
500 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 8.8 MiB 2.5 GiB 491
GiB 73.66 1.01 59 up
501 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 10 MiB 2.5 GiB 491
GiB 73.63 1.01 58 up
502 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 22 MiB 7.4 GiB 1.4
TiB 73.62 1.01 174 up
503 hdd 5.45789 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 21 MiB 7.4 GiB 1.4
TiB 73.54 1.01 177 up
504 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 38 MiB 13 GiB 2.2
TiB 75.67 1.03 301 up
505 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 42 MiB 13 GiB 2.2
TiB 75.67 1.03 300 up
10100 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 28 MiB 996 MiB 376
GiB 9.60 0.13 35 up
10101 ssd 0.40630 1.00000 416 GiB 44 GiB 43 GiB 20 MiB 1004 MiB 372
GiB 10.57 0.14 37 up
10200 ssd 0.40630 1.00000 416 GiB 32 GiB 31 GiB 15 MiB 1009 MiB 384
GiB 7.80 0.11 36 up
10201 ssd 0.40630 1.00000 416 GiB 32 GiB 31 GiB 16 MiB 1008 MiB 384
GiB 7.67 0.10 37 up
10300 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 17 MiB 1007 MiB 376
GiB 9.71 0.13 36 up
10301 ssd 0.40630 1.00000 416 GiB 43 GiB 42 GiB 18 MiB 1006 MiB 373
GiB 10.31 0.14 37 up
10400 ssd 0.40630 1.00000 416 GiB 34 GiB 33 GiB 17 MiB 1007 MiB 382
GiB 8.14 0.11 35 up
10401 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 19 MiB 1005 MiB 376
GiB 9.64 0.13 36 up
10500 ssd 0.40630 1.00000 416 GiB 38 GiB 37 GiB 18 MiB 1006 MiB 378
GiB 9.11 0.12 37 up
10501 ssd 0.40630 1.00000 416 GiB 46 GiB 45 GiB 18 MiB 1006 MiB 370
GiB 10.98 0.15 37 up
TOTAL 168 TiB 123 TiB 123 TiB 839 MiB 234 GiB 45
TiB 73.16
[admin@kvm7e ~]# for f in /var/run/ceph/ceph-osd.*.asok; do ceph
--admin-daemon $f config show; done | grep 'full'
"mon_cache_target_full_warn_ratio": "0.660000",
"mon_osd_backfillfull_ratio": "0.900000",
"mon_osd_full_ratio": "0.950000",
"mon_osd_nearfull_ratio": "0.850000",
"mon_osdmap_full_prune_enabled": "true",
"mon_osdmap_full_prune_interval": "10",
"mon_osdmap_full_prune_min": "10000",
"mon_osdmap_full_prune_txsize": "100",
"osd_debug_skip_full_check_in_backfill_reservation": "false",
"osd_debug_skip_full_check_in_recovery": "false",
"osd_failsafe_full_ratio": "0.970000",
"osd_pool_default_cache_target_full_ratio": "0.800000",
"paxos_stash_full_interval": "25",
<snip>
An hour later:
[admin@kvm7e ~]# ceph health detail
HEALTH_ERR noout flag(s) set; 6 nearfull osd(s); 4 pool(s) nearfull;
Degraded data redundancy: 2891706/93291241 objects degraded (3.100%), 165
pgs degraded, 165 pgs undersized; Degraded data redundancy (low space): 647
pgs backfill_toofull
OSDMAP_FLAGS noout flag(s) set
OSD_NEARFULL 6 nearfull osd(s)
osd.100 is near full
osd.101 is near full
osd.102 is near full
osd.103 is near full
osd.104 is near full
osd.105 is near full
POOL_NEARFULL 4 pool(s) nearfull
pool 'cephfs_data' is nearfull
pool 'cephfs_metadata' is nearfull
pool 'rbd_hdd' is nearfull
pool 'ec_hdd' is nearfull
PG_DEGRADED Degraded data redundancy: 2891706/93291241 objects degraded
(3.100%), 165 pgs degraded, 165 pgs undersized
pg 8.304 is active+undersized+degraded+remapped+backfill_toofull, acting
[2147483647,503,304,402,203]
pg 8.307 is stuck undersized for 7968.693589, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,205,503,2147483647,401]
pg 8.30c is stuck undersized for 7968.739427, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[501,2147483647,202,304,402]
pg 8.311 is stuck undersized for 7968.732720, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[405,2147483647,502,204,305]
pg 8.314 is stuck undersized for 7968.732950, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,302,503,205,405]
pg 8.319 is stuck undersized for 7968.716745, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,403,204,505,305]
pg 8.31c is stuck undersized for 7968.713635, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[500,2147483647,402,305,204]
pg 8.327 is stuck undersized for 7968.664546, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[202,305,504,2147483647,405]
pg 8.32f is stuck undersized for 7968.682409, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,402,204,504,302]
pg 8.332 is stuck undersized for 7968.732504, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[302,405,2147483647,205,502]
pg 8.334 is stuck undersized for 7968.694182, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,205,404,302,503]
pg 8.33c is stuck undersized for 7968.734577, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[302,505,405,2147483647,201]
pg 8.33f is stuck undersized for 7968.552298, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[400,504,2147483647,304,204]
pg 8.348 is stuck undersized for 7968.696137, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,2147483647,404,504,203]
pg 8.34c is stuck undersized for 7968.768111, current state
active+undersized+degraded+remapped+backfilling, last acting
[504,204,2147483647,303,403]
pg 8.350 is stuck undersized for 7968.734046, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[405,502,305,2147483647,205]
pg 8.35a is stuck undersized for 7968.685123, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,201,404,504,304]
pg 8.35c is stuck undersized for 7968.700947, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[404,2147483647,303,505,205]
pg 8.35e is stuck undersized for 7968.683728, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[402,2147483647,304,500,203]
pg 8.361 is stuck undersized for 7968.798644, current state
active+undersized+degraded+remapped+backfilling, last acting
[505,404,304,2147483647,204]
pg 8.365 is stuck undersized for 7968.731458, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[405,303,2147483647,205,504]
pg 8.368 is stuck undersized for 7968.799312, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,304,404,502,205]
pg 8.36b is stuck undersized for 7968.736514, current state
active+undersized+degraded+remapped+backfill_wait, last acting
[300,403,204,2147483647,505]
pg 8.36f is stuck undersized for 7968.695546, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,503,2147483647,405,205]
pg 8.373 is stuck undersized for 7968.717140, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[403,303,2147483647,202,502]
pg 8.379 is stuck undersized for 7968.732125, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,405,205,302,504]
pg 8.37c is stuck undersized for 7968.712063, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[401,500,205,2147483647,302]
pg 8.37d is stuck undersized for 7968.740233, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[501,302,2147483647,205,404]
pg 8.384 is stuck undersized for 7968.796821, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,503,304,402,203]
pg 8.387 is stuck undersized for 7968.639604, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,205,503,2147483647,401]
pg 8.392 is stuck undersized for 7968.771812, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[504,305,405,202,2147483647]
pg 8.394 is stuck undersized for 7968.734314, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,302,503,205,405]
pg 8.399 is stuck undersized for 7968.722090, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,403,204,505,305]
pg 8.39c is stuck undersized for 7968.712907, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[500,2147483647,402,305,204]
pg 8.39d is stuck undersized for 7968.725227, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[403,2147483647,502,305,203]
pg 8.3a5 is stuck undersized for 7968.644387, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[204,2147483647,304,400,505]
pg 8.3a7 is stuck undersized for 7968.668449, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[202,305,504,2147483647,405]
pg 8.3af is stuck undersized for 7968.683191, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,402,204,504,302]
pg 8.3b2 is stuck undersized for 7968.733004, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[302,405,2147483647,205,502]
pg 8.3b4 is stuck undersized for 7968.694415, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,205,404,302,503]
pg 8.3bc is stuck undersized for 7968.732598, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[302,505,405,2147483647,201]
pg 8.3cc is stuck undersized for 7968.771592, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[504,204,2147483647,303,403]
pg 8.3d0 is stuck undersized for 7968.733520, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[405,502,305,2147483647,205]
pg 8.3de is stuck undersized for 7968.681841, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[402,2147483647,304,500,203]
pg 8.3df is stuck undersized for 7968.726621, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,200,303,500,404]
pg 8.3e1 is stuck undersized for 7968.796332, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[505,404,304,2147483647,204]
pg 8.3ec is stuck undersized for 7968.776206, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[502,302,405,2147483647,204]
pg 8.3ef is stuck undersized for 7968.690806, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,503,2147483647,405,205]
pg 8.3f7 is stuck undersized for 7968.672395, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[205,505,404,305,2147483647]
pg 8.3fc is stuck undersized for 7968.711543, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[401,500,205,2147483647,302]
pg 8.3fd is stuck undersized for 7968.740233, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[501,302,2147483647,205,404]
PG_DEGRADED_FULL Degraded data redundancy (low space): 647 pgs
backfill_toofull
pg 8.3bc is active+undersized+degraded+remapped+backfill_toofull, acting
[302,505,405,2147483647,201]
pg 8.3bd is active+remapped+backfill_wait+backfill_toofull, acting
[202,402,105,503,304]
pg 8.3be is active+remapped+backfill_toofull, acting
[504,303,205,100,403]
pg 8.3c0 is active+remapped+backfill_toofull, acting
[402,304,503,105,205]
pg 8.3c1 is active+remapped+backfill_toofull, acting
[301,101,405,504,204]
pg 8.3c2 is active+remapped+backfill_toofull, acting
[104,204,305,503,405]
pg 8.3c3 is active+remapped+backfill_toofull, acting
[405,105,204,303,505]
pg 8.3c4 is active+remapped+backfill_toofull, acting
[504,305,101,403,201]
pg 8.3c6 is active+remapped+backfill_toofull, acting
[404,304,205,504,105]
pg 8.3c8 is active+remapped+backfill_toofull, acting
[305,104,404,504,203]
pg 8.3c9 is active+remapped+backfill_toofull, acting
[404,505,102,301,203]
pg 8.3cb is active+remapped+backfill_wait+backfill_toofull, acting
[105,200,402,505,304]
pg 8.3cc is active+undersized+degraded+remapped+backfill_toofull, acting
[504,204,2147483647,303,403]
pg 8.3cd is active+remapped+backfill_wait+backfill_toofull, acting
[105,305,403,504,205]
pg 8.3cf is active+remapped+backfill_toofull, acting
[502,400,304,105,202]
pg 8.3d0 is active+undersized+degraded+remapped+backfill_toofull, acting
[405,502,305,2147483647,205]
pg 8.3d1 is active+remapped+backfill_toofull, acting
[205,103,404,502,304]
pg 8.3d2 is active+remapped+backfill_toofull, acting
[202,505,304,403,103]
pg 8.3d3 is active+remapped+backfill_toofull, acting
[204,101,405,505,302]
pg 8.3d4 is active+remapped+backfill_toofull, acting
[503,305,404,100,205]
pg 8.3d5 is active+remapped+backfill_wait+backfill_toofull, acting
[105,503,203,401,304]
pg 8.3d7 is active+remapped+backfill_toofull, acting
[504,305,404,200,102]
pg 8.3d9 is active+remapped+backfill_toofull, acting
[202,302,402,105,504]
pg 8.3da is active+remapped+backfill_toofull, acting
[104,201,404,504,304]
pg 8.3dc is active+remapped+backfill_wait+backfill_toofull, acting
[404,104,303,505,205]
pg 8.3dd is active+remapped+backfill_toofull, acting
[404,204,505,302,105]
pg 8.3de is active+undersized+degraded+remapped+backfill_toofull, acting
[402,2147483647,304,500,203]
pg 8.3df is active+undersized+degraded+remapped+backfill_toofull, acting
[2147483647,200,303,500,404]
pg 8.3e0 is active+remapped+backfill_toofull, acting
[202,402,103,305,505]
pg 8.3e1 is active+undersized+degraded+remapped+backfill_toofull, acting
[505,404,304,2147483647,204]
pg 8.3e2 is active+remapped+backfill_toofull, acting
[304,102,505,401,203]
pg 8.3e4 is active+remapped+backfill_toofull, acting
[303,102,503,202,405]
pg 8.3e5 is active+remapped+backfill_toofull, acting
[405,303,104,205,504]
pg 8.3e6 is active+remapped+backfill_wait+backfill_toofull, acting
[504,105,204,404,305]
pg 8.3e7 is active+remapped+backfill_toofull, acting
[103,202,505,405,304]
pg 8.3eb is active+remapped+backfill_toofull, acting
[300,403,204,104,505]
pg 8.3ec is active+undersized+degraded+remapped+backfill_toofull, acting
[502,302,405,2147483647,204]
pg 8.3ee is active+remapped+backfill_toofull, acting
[403,300,204,503,100]
pg 8.3ef is active+undersized+degraded+remapped+backfill_toofull, acting
[305,503,2147483647,405,205]
pg 8.3f0 is active+remapped+backfill_toofull, acting
[102,203,500,304,403]
pg 8.3f1 is active+remapped+backfill_toofull, acting
[505,305,404,105,202]
pg 8.3f2 is active+remapped+backfill_wait+backfill_toofull, acting
[105,304,403,202,502]
pg 8.3f4 is active+remapped+backfill_toofull, acting
[205,505,102,405,303]
pg 8.3f5 is active+remapped+backfill_toofull, acting
[405,105,304,504,201]
pg 8.3f6 is active+remapped+backfill_wait+backfill_toofull, acting
[105,204,505,304,404]
pg 8.3f7 is active+undersized+degraded+remapped+backfill_toofull, acting
[205,505,404,305,2147483647]
pg 8.3fb is active+remapped+backfill_toofull, acting
[303,505,401,105,203]
pg 8.3fc is active+undersized+degraded+remapped+backfill_toofull, acting
[401,500,205,2147483647,302]
pg 8.3fd is active+undersized+degraded+remapped+backfill_toofull, acting
[501,302,2147483647,205,404]
pg 8.3fe is active+remapped+backfill_toofull, acting
[504,304,402,205,105]
pg 8.3ff is active+remapped+backfill_toofull, acting
[405,202,102,501,303]
About a day after enabling the pg_autoscaler module:
[admin@kvm7e ~]# ceph health detail
HEALTH_ERR noout flag(s) set; 18 nearfull osd(s); 4 pool(s) nearfull;
Degraded data redundancy: 4227162/93352306 objects degraded (4.528%), 250
pgs degraded, 253 pgs undersized; Degraded data redundancy (low space): 559
pgs backfill_toofull
OSDMAP_FLAGS noout flag(s) set
OSD_NEARFULL 18 nearfull osd(s)
osd.100 is near full
osd.101 is near full
osd.102 is near full
osd.103 is near full
osd.104 is near full
osd.105 is near full
osd.200 is near full
osd.201 is near full
osd.203 is near full
osd.300 is near full
osd.301 is near full
osd.303 is near full
osd.304 is near full
osd.401 is near full
osd.404 is near full
osd.501 is near full
osd.502 is near full
osd.504 is near full
POOL_NEARFULL 4 pool(s) nearfull
pool 'cephfs_data' is nearfull
pool 'cephfs_metadata' is nearfull
pool 'rbd_hdd' is nearfull
pool 'ec_hdd' is nearfull
PG_DEGRADED Degraded data redundancy: 4227162/93352306 objects degraded
(4.528%), 250 pgs degraded, 253 pgs undersized
pg 8.35a is active+undersized+degraded+remapped+backfill_toofull, acting
[2147483647,201,404,504,304]
pg 8.35b is stuck undersized for 13233.292244, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[203,503,2147483647,305,105]
pg 8.361 is stuck undersized for 67655.237839, current state
active+undersized+remapped+backfill_toofull, last acting
[505,404,304,2147483647,204]
pg 8.362 is stuck undersized for 67682.460453, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[304,2147483647,505,401,203]
pg 8.363 is stuck undersized for 67665.785277, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[404,203,301,504,2147483647]
pg 8.365 is stuck undersized for 13232.214476, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,303,104,205,504]
pg 8.366 is stuck undersized for 67665.803857, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[504,2147483647,204,404,305]
pg 8.368 is stuck undersized for 67655.243869, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,304,404,502,205]
pg 8.36d is stuck undersized for 67682.440479, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[404,2147483647,201,502,304]
pg 8.36f is stuck undersized for 13232.182277, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,503,104,2147483647,205]
pg 8.373 is stuck undersized for 67665.636152, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[403,303,2147483647,202,502]
pg 8.374 is stuck undersized for 13232.196886, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[205,505,102,2147483647,303]
pg 8.375 is stuck undersized for 13232.147722, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,105,304,504,201]
pg 8.379 is stuck undersized for 13232.052714, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[104,2147483647,205,302,504]
pg 8.37a is stuck undersized for 13233.378060, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,105,502,202,2147483647]
pg 8.37d is stuck undersized for 67665.810745, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[501,302,2147483647,205,404]
pg 8.381 is stuck undersized for 13233.351431, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[304,204,2147483647,101,503]
pg 8.382 is stuck undersized for 13232.226588, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[504,205,2147483647,300,105]
pg 8.387 is stuck undersized for 67665.791360, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,205,503,2147483647,401]
pg 8.391 is stuck undersized for 13233.342426, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,104,502,204,305]
pg 8.392 is stuck undersized for 13232.227756, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[504,305,2147483647,202,104]
pg 8.396 is stuck undersized for 13233.333363, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[303,102,2147483647,504,204]
pg 8.399 is stuck undersized for 67665.635750, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,403,204,505,305]
pg 8.39c is stuck undersized for 67655.266511, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[500,2147483647,402,305,204]
pg 8.39d is stuck undersized for 67655.215300, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[403,2147483647,502,305,203]
pg 8.3a0 is stuck undersized for 13233.339729, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,104,304,501,204]
pg 8.3a1 is stuck undersized for 13232.162171, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[105,505,205,303,2147483647]
pg 8.3a4 is stuck undersized for 13233.379809, current state
active+undersized+degraded+remapped+backfilling, last acting
[2147483647,305,102,502,203]
pg 8.3ad is stuck undersized for 13233.368722, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[301,202,501,2147483647,103]
pg 8.3ae is stuck undersized for 67665.744856, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[304,503,404,200,2147483647]
pg 8.3af is stuck undersized for 67665.769691, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,402,204,504,302]
pg 8.3b4 is stuck undersized for 67682.382958, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,205,404,302,503]
pg 8.3bc is stuck undersized for 13232.171438, current state
active+undersized+degraded+remapped+backfilling, last acting
[302,505,2147483647,104,201]
pg 8.3c3 is stuck undersized for 13232.151774, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,105,204,303,505]
pg 8.3c6 is stuck undersized for 67665.792220, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[404,304,205,504,2147483647]
pg 8.3ca is stuck undersized for 67665.817464, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[503,2147483647,205,305,400]
pg 8.3cc is stuck undersized for 67665.802241, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[504,204,2147483647,303,403]
pg 8.3d3 is stuck undersized for 13233.266745, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[204,101,2147483647,505,302]
pg 8.3de is stuck undersized for 67655.238851, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[402,2147483647,304,500,203]
pg 8.3df is stuck undersized for 67682.379100, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,200,303,500,404]
pg 8.3e1 is stuck undersized for 67655.229239, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[505,404,304,2147483647,204]
pg 8.3e3 is stuck undersized for 67665.788025, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[404,203,301,504,2147483647]
pg 8.3e5 is stuck undersized for 13233.333897, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,303,104,205,504]
pg 8.3e7 is stuck undersized for 13233.318293, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[103,202,505,2147483647,304]
pg 8.3ea is stuck undersized for 67682.450664, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,2147483647,203,403,504]
pg 8.3ec is stuck undersized for 13232.182236, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[502,302,2147483647,104,204]
pg 8.3f1 is stuck undersized for 67665.809335, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[505,305,404,2147483647,202]
pg 8.3fa is stuck undersized for 13232.182880, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[305,105,502,202,2147483647]
pg 8.3fc is stuck undersized for 67655.245817, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[401,500,205,2147483647,302]
pg 8.3fe is stuck undersized for 67665.800943, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[504,304,402,205,2147483647]
pg 8.3ff is stuck undersized for 13233.326383, current state
active+undersized+degraded+remapped+backfill_toofull, last acting
[2147483647,202,102,501,303]
PG_DEGRADED_FULL Degraded data redundancy (low space): 559 pgs
backfill_toofull
pg 8.3af is active+undersized+degraded+remapped+backfill_toofull, acting
[2147483647,402,204,504,302]
pg 8.3b0 is active+remapped+backfill_toofull, acting
[405,305,502,200,103]
pg 8.3b1 is active+remapped+backfill_wait+backfill_toofull, acting
[505,205,300,101,402]
pg 8.3b4 is active+undersized+degraded+remapped+backfill_toofull, acting
[2147483647,205,404,302,503]
pg 8.3b5 is active+remapped+backfill_toofull, acting
[204,505,100,403,304]
pg 8.3b6 is active+remapped+backfill_toofull, acting
[404,300,504,100,203]
pg 8.3b7 is active+remapped+backfill_toofull, acting
[204,105,504,303,401]
pg 8.3b8 is active+remapped+backfill_toofull, acting
[501,103,205,302,404]
pg 8.3b9 is active+remapped+backfill_toofull, acting
[502,301,402,103,204]
pg 8.3ba is active+remapped+backfill_toofull, acting
[303,102,203,403,505]
pg 8.3bb is active+remapped+backfill_toofull, acting
[505,204,102,304,404]
pg 8.3bd is active+remapped+backfill_toofull, acting
[202,402,105,503,304]
pg 8.3be is active+remapped+backfill_toofull, acting
[504,303,205,100,403]
pg 8.3c3 is active+undersized+degraded+remapped+backfill_toofull, acting
[2147483647,105,204,303,505]
pg 8.3c4 is active+remapped+backfill_wait+backfill_toofull, acting
[504,305,101,403,201]
pg 8.3c6 is active+undersized+degraded+remapped+backfill_toofull, acting
[404,304,205,504,2147483647]
pg 8.3ca is active+undersized+degraded+remapped+backfill_toofull, acting
[503,2147483647,205,305,400]
pg 8.3cb is active+remapped+backfill_toofull, acting
[105,200,402,505,304]
pg 8.3cc is active+undersized+degraded+remapped+backfill_toofull, acting
[504,204,2147483647,303,403]
pg 8.3cd is active+remapped+backfill_toofull, acting
[105,305,403,504,205]
pg 8.3cf is active+remapped+backfill_toofull, acting
[502,400,304,105,202]
pg 8.3d0 is active+remapped+backfill_toofull, acting
[405,502,305,104,205]
pg 8.3d1 is active+remapped+backfill_toofull, acting
[205,103,404,502,304]
pg 8.3d3 is active+undersized+degraded+remapped+backfill_toofull, acting
[204,101,2147483647,505,302]
pg 8.3d4 is active+remapped+backfill_toofull, acting
[503,305,404,100,205]
pg 8.3d7 is active+remapped+backfill_toofull, acting
[504,305,404,200,102]
pg 8.3d8 is active+remapped+backfill_toofull, acting
[403,202,102,303,500]
pg 8.3da is active+remapped+backfill_wait+backfill_toofull, acting
[104,201,404,504,304]
pg 8.3dc is active+remapped+backfill_wait+backfill_toofull, acting
[404,104,303,505,205]
pg 8.3de is active+undersized+degraded+remapped+backfill_toofull, acting
[402,2147483647,304,500,203]
pg 8.3df is active+undersized+degraded+remapped+backfill_toofull, acting
[2147483647,200,303,500,404]
pg 8.3e0 is active+remapped+backfill_wait+backfill_toofull, acting
[202,402,103,305,505]
pg 8.3e1 is active+undersized+degraded+remapped+backfill_toofull, acting
[505,404,304,2147483647,204]
pg 8.3e2 is active+remapped+backfill_toofull, acting
[304,102,505,401,203]
pg 8.3e3 is active+undersized+degraded+remapped+backfill_toofull, acting
[404,203,301,504,2147483647]
pg 8.3e5 is active+undersized+degraded+remapped+backfill_toofull, acting
[2147483647,303,104,205,504]
pg 8.3e6 is active+remapped+backfill_toofull, acting
[504,105,204,404,305]
pg 8.3e7 is active+undersized+degraded+remapped+backfill_toofull, acting
[103,202,505,2147483647,304]
pg 8.3e9 is active+remapped+backfill_toofull, acting
[504,103,305,403,203]
pg 8.3ea is active+undersized+degraded+remapped+backfill_toofull, acting
[305,2147483647,203,403,504]
pg 8.3ec is active+undersized+degraded+remapped+backfill_toofull, acting
[502,302,2147483647,104,204]
pg 8.3ed is active+remapped+backfill_wait+backfill_toofull, acting
[404,101,201,502,304]
pg 8.3ee is active+remapped+backfill_toofull, acting
[403,300,204,503,100]
pg 8.3f0 is active+remapped+backfill_toofull, acting
[102,203,500,304,403]
pg 8.3f1 is active+undersized+degraded+remapped+backfill_toofull, acting
[505,305,404,2147483647,202]
pg 8.3f2 is active+remapped+backfill_toofull, acting
[105,304,403,202,502]
pg 8.3f4 is active+remapped+backfill_toofull, acting
[205,505,102,405,303]
pg 8.3fa is active+undersized+degraded+remapped+backfill_toofull, acting
[305,105,502,202,2147483647]
pg 8.3fc is active+undersized+degraded+remapped+backfill_toofull, acting
[401,500,205,2147483647,302]
pg 8.3fe is active+undersized+degraded+remapped+backfill_toofull, acting
[504,304,402,205,2147483647]
pg 8.3ff is active+undersized+degraded+remapped+backfill_toofull, acting
[2147483647,202,102,501,303]
Ceph OSD utilisation breakdown:
[admin@kvm7e ~]# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS
100 hdd 1.81929 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 2.2 MiB 3.5 GiB 232
GiB 87.55 1.13 68 up
101 hdd 1.81929 1.00000 1.8 TiB 1.6 TiB 1.6 TiB 2.8 MiB 3.4 GiB 255
GiB 86.29 1.12 64 up
102 hdd 5.45789 1.00000 5.5 TiB 4.8 TiB 4.8 TiB 6.6 MiB 9.4 GiB 696
GiB 87.54 1.13 187 up
103 hdd 5.45789 1.00000 5.5 TiB 4.8 TiB 4.8 TiB 6.4 MiB 9.4 GiB 654
GiB 88.29 1.14 191 up
104 hdd 9.09560 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 11 MiB 12 GiB 3.2
TiB 64.67 0.84 192 up
105 hdd 9.09560 1.00000 9.1 TiB 7.1 TiB 7.1 TiB 10 MiB 14 GiB 1.9
TiB 78.58 1.02 256 up
200 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 2.0 MiB 3.2 GiB 317
GiB 82.99 1.08 64 up
201 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 2.2 MiB 3.4 GiB 292
GiB 84.34 1.09 63 up
202 hdd 5.45789 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 6.7 MiB 8.5 GiB 1.1
TiB 79.18 1.03 176 up
203 hdd 5.45789 1.00000 5.5 TiB 4.7 TiB 4.7 TiB 6.2 MiB 9.2 GiB 762
GiB 86.36 1.12 195 up
204 hdd 9.09560 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 12 MiB 14 GiB 2.1
TiB 77.12 1.00 290 up
205 hdd 9.09560 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 11 MiB 13 GiB 2.4
TiB 73.89 0.96 280 up
300 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 2.0 MiB 3.3 GiB 283
GiB 84.80 1.10 65 up
301 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 2.8 MiB 3.4 GiB 298
GiB 84.00 1.09 64 up
302 hdd 5.45789 1.00000 5.5 TiB 3.8 TiB 3.8 TiB 6.5 MiB 7.6 GiB 1.7
TiB 69.08 0.90 154 up
303 hdd 5.45789 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 6.9 MiB 8.7 GiB 1.0
TiB 80.97 1.05 182 up
304 hdd 9.09560 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 12 MiB 14 GiB 1.6
TiB 82.39 1.07 311 up
305 hdd 9.09560 1.00000 9.1 TiB 7.1 TiB 7.0 TiB 11 MiB 14 GiB 2.0
TiB 77.54 1.01 295 up
400 hdd 1.81929 1.00000 1.8 TiB 1.2 TiB 1.2 TiB 2.3 MiB 2.9 GiB 596
GiB 68.02 0.88 53 up
401 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 3.1 MiB 3.4 GiB 292
GiB 84.33 1.09 63 up
402 hdd 5.45789 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 7.6 MiB 8.3 GiB 1.2
TiB 77.33 1.00 171 up
403 hdd 5.45789 1.00000 5.5 TiB 4.1 TiB 4.1 TiB 6.3 MiB 8.1 GiB 1.4
TiB 74.97 0.97 175 up
404 hdd 9.09560 1.00000 9.1 TiB 7.9 TiB 7.8 TiB 11 MiB 15 GiB 1.2
TiB 86.38 1.12 321 up
405 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 11 MiB 14 GiB 2.2
TiB 76.26 0.99 144 up
500 hdd 1.81929 1.00000 1.8 TiB 1.3 TiB 1.3 TiB 2.3 MiB 3.0 GiB 516
GiB 72.32 0.94 53 up
501 hdd 1.81929 1.00000 1.8 TiB 1.5 TiB 1.5 TiB 2.7 MiB 3.4 GiB 308
GiB 83.46 1.08 63 up
502 hdd 5.45789 1.00000 5.5 TiB 4.7 TiB 4.6 TiB 5.3 MiB 9.2 GiB 820
GiB 85.32 1.11 192 up
503 hdd 5.45789 1.00000 5.5 TiB 3.9 TiB 3.9 TiB 6.4 MiB 7.8 GiB 1.5
TiB 71.61 0.93 165 up
504 hdd 9.09560 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 12 MiB 14 GiB 1.6
TiB 82.31 1.07 311 up
505 hdd 9.09560 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 11 MiB 14 GiB 2.2
TiB 75.78 0.98 284 up
10100 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 19 MiB 1005 MiB 376
GiB 9.70 0.13 35 up
10101 ssd 0.40630 1.00000 416 GiB 44 GiB 43 GiB 39 MiB 985 MiB 372
GiB 10.62 0.14 37 up
10200 ssd 0.40630 1.00000 416 GiB 33 GiB 32 GiB 17 MiB 1007 MiB 383
GiB 7.92 0.10 36 up
10201 ssd 0.40630 1.00000 416 GiB 32 GiB 31 GiB 33 MiB 991 MiB 384
GiB 7.67 0.10 37 up
10300 ssd 0.40630 1.00000 416 GiB 41 GiB 40 GiB 30 MiB 994 MiB 375
GiB 9.80 0.13 36 up
10301 ssd 0.40630 1.00000 416 GiB 43 GiB 42 GiB 37 MiB 987 MiB 373
GiB 10.37 0.13 37 up
10400 ssd 0.40630 1.00000 416 GiB 34 GiB 33 GiB 24 MiB 1000 MiB 382
GiB 8.27 0.11 35 up
10401 ssd 0.40630 1.00000 416 GiB 40 GiB 39 GiB 30 MiB 994 MiB 376
GiB 9.69 0.13 36 up
10500 ssd 0.40630 1.00000 416 GiB 38 GiB 37 GiB 30 MiB 994 MiB 378
GiB 9.15 0.12 37 up
10501 ssd 0.40630 1.00000 416 GiB 46 GiB 45 GiB 31 MiB 993 MiB 370
GiB 11.08 0.14 37 up
TOTAL 168 TiB 129 TiB 129 TiB 493 MiB 267 GiB 38
TiB 77.15
MIN/MAX VAR: 0.10/1.14 STDDEV: 34.38
Regards
David Herselman
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
