Hello all
Need some help why nothing is happening… version is 19.2.2.
Tried repairing the pgs, repair and force-backfill and force-recovery but
nothing initiating backfills.
# ceph -v
ceph version 19.2.2
# ceph -s
cluster:
id: 79d4fade-ffa9-11eb-a57d-f398f000cfc1
health: HEALTH_WARN
1 OSD(s) experiencing slow operations in BlueStore
nodeep-scrub flag(s) set
Degraded data redundancy: 6114281/137592495 objects degraded
(4.444%), 124 pgs degraded, 124 pgs undersized
270 pgs not deep-scrubbed in time
1 subtrees have overcommitted pool target_size_bytes
services:
mon: 3 daemons, quorum ld-converged03n,ld-converged01n,ld-converged02n (age
5d)
mgr: ld-converged01n.gnxoye(active, since 5d), standbys:
ld-converged02n.odvopy
mds: 4/4 daemons up, 2 standby, 2 hot standby
osd: 24 osds: 24 up (since 29h), 24 in (since 29h); 292 remapped pgs
flags nodeep-scrub
rgw: 2 daemons active (2 hosts, 1 zones)
data:
volumes: 4/4 healthy
pools: 24 pools, 1345 pgs
objects: 32.22M objects, 98 TiB
usage: 196 TiB used, 83 TiB / 279 TiB avail
pgs: 6114281/137592495 objects degraded (4.444%)
14349193/137592495 objects misplaced (10.429%)
864 active+clean
237 active+clean+remapped
124 active+undersized+degraded
55 active+clean+scrubbing
45 active+clean+remapped+scrubbing
10 active+clean+remapped+scrubbing+deep
10 active+clean+scrubbing+deep
io:
client: 50 KiB/s rd, 1.4 MiB/s wr, 8 op/s rd, 205 op/s wr
# ceph health detail
HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore; nodeep-scrub
flag(s) set; Degraded data redundancy: 6114283/137592537 objects degraded
(4.444%), 124 pgs degraded, 124 pgs undersized; 270 pgs not deep-scrubbed in
time; 1 subtrees have overcommitted pool target_size_bytes
[WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in
BlueStore
osd.21 observed slow operation indications in BlueStore
[WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set
[WRN] PG_DEGRADED: Degraded data redundancy: 6114283/137592537 objects degraded
(4.444%), 124 pgs degraded, 124 pgs undersized
pg 26.3 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [31,14,NONE,0,8]
pg 26.4 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [2,8,NONE,24,35]
pg 26.7 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [2,15,31,7,NONE]
pg 26.d is stuck undersized for 94m, current state
active+undersized+degraded, last acting [21,12,30,NONE,9]
pg 26.11 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [13,8,30,NONE,3]
pg 26.3c is stuck undersized for 94m, current state
active+undersized+degraded, last acting [0,7,35,5,NONE]
pg 26.3d is stuck undersized for 94m, current state
active+undersized+degraded, last acting [9,NONE,14,6,31]
pg 27.0 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [2,31,7,15,NONE]
pg 27.2 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [0,35,11,NONE,31]
pg 27.3 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [35,31,4,1,NONE]
pg 27.b is active+undersized+degraded, acting [2,8,NONE,5,12]
pg 27.c is stuck undersized for 94m, current state
active+undersized+degraded, last acting [2,8,12,31,NONE]
pg 27.10 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [23,8,14,NONE,0]
pg 27.14 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [30,16,9,NONE,6]
pg 27.17 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [23,4,14,NONE,1]
pg 27.1b is stuck undersized for 94m, current state
active+undersized+degraded, last acting [12,10,NONE,2,30]
pg 27.26 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [0,35,10,24,NONE]
pg 27.3c is stuck undersized for 94m, current state
active+undersized+degraded, last acting [32,9,2,13,NONE]
pg 33.1a is stuck undersized for 94m, current state
active+undersized+degraded, last acting [0,NONE,31,35,4]
pg 48.22 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [30,0,7,NONE,12]
pg 48.24 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [5,1,NONE,13,9]
pg 48.26 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [9,3,35,NONE,30]
pg 48.28 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [9,NONE,3,14,5]
pg 48.2b is stuck undersized for 94m, current state
active+undersized+degraded, last acting [15,4,5,NONE,0]
pg 48.2e is stuck undersized for 94m, current state
active+undersized+degraded, last acting [15,7,31,3,NONE]
pg 48.30 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [15,31,21,NONE,9]
pg 48.31 is stuck undersized for 9M, current state
active+undersized+degraded, last acting [4,1,13,30,NONE]
pg 48.32 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [4,0,5,13,NONE]
pg 48.41 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [8,32,NONE,0,16]
pg 48.43 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [30,0,12,7,NONE]
pg 48.4b is stuck undersized for 94m, current state
active+undersized+degraded, last acting [13,6,NONE,10,24]
pg 48.4f is stuck undersized for 94m, current state
active+undersized+degraded, last acting [30,15,2,7,NONE]
pg 48.50 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [2,9,NONE,13,24]
pg 48.53 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [4,30,14,3,NONE]
pg 48.54 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [9,6,14,NONE,30]
pg 48.56 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [35,9,5,1,NONE]
pg 48.59 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [30,14,7,NONE,3]
pg 48.5d is stuck undersized for 94m, current state
active+undersized+degraded, last acting [10,35,NONE,3,5]
pg 48.5f is stuck undersized for 94m, current state
active+undersized+degraded, last acting [5,13,8,0,NONE]
pg 48.62 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [4,2,32,35,NONE]
pg 48.70 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [3,10,5,15,NONE]
pg 48.72 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [1,31,11,15,NONE]
pg 48.75 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [3,24,NONE,9,35]
pg 48.7d is stuck undersized for 94m, current state
active+undersized+degraded, last acting [5,10,14,1,NONE]
pg 48.7e is stuck undersized for 94m, current state
active+undersized+degraded, last acting [32,NONE,15,9,0]
pg 48.82 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [24,13,3,9,NONE]
pg 48.86 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [23,9,12,2,NONE]
pg 48.8e is stuck undersized for 94m, current state
active+undersized+degraded, last acting [2,7,12,NONE,32]
pg 48.90 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [11,2,NONE,12,30]
pg 48.97 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [0,8,16,5,NONE]
pg 48.98 is stuck undersized for 94m, current state
active+undersized+degraded, last acting [23,9,12,2,NONE]
[WRN] PG_NOT_DEEP_SCRUBBED: 270 pgs not deep-scrubbed in time
pg 48.9b not deep-scrubbed since 2025-06-24T22:11:31.466631+0000
pg 48.97 not deep-scrubbed since 2025-06-28T14:03:27.678577+0000
pg 59.9d not deep-scrubbed since 2025-06-23T02:38:49.854500+0000
pg 48.90 not deep-scrubbed since 2025-06-27T07:20:46.510366+0000
pg 48.8e not deep-scrubbed since 2025-06-20T03:44:19.531953+0000
pg 48.8c not deep-scrubbed since 2025-06-27T04:38:11.342009+0000
pg 48.86 not deep-scrubbed since 2025-06-24T02:01:54.551428+0000
pg 59.8e not deep-scrubbed since 2025-06-22T17:48:29.668256+0000
pg 48.82 not deep-scrubbed since 2025-06-28T06:27:01.586778+0000
pg 59.76 not deep-scrubbed since 2025-06-26T04:36:17.368921+0000
pg 48.7d not deep-scrubbed since 2025-06-26T14:26:17.237509+0000
pg 48.7c not deep-scrubbed since 2025-06-23T18:57:42.886560+0000
pg 48.7b not deep-scrubbed since 2025-06-25T04:40:51.163596+0000
pg 59.71 not deep-scrubbed since 2025-06-23T22:57:42.807731+0000
pg 59.73 not deep-scrubbed since 2025-06-23T15:06:04.642059+0000
pg 58.72 not deep-scrubbed since 2025-06-26T08:28:44.995587+0000
pg 48.75 not deep-scrubbed since 2025-06-25T13:33:41.368882+0000
pg 48.74 not deep-scrubbed since 2025-06-22T18:15:30.731614+0000
pg 48.72 not deep-scrubbed since 2025-06-22T18:07:01.392496+0000
pg 48.71 not deep-scrubbed since 2025-06-23T15:40:23.134466+0000
pg 48.70 not deep-scrubbed since 2025-06-22T13:17:09.447156+0000
pg 59.66 not deep-scrubbed since 2025-06-24T21:58:12.342232+0000
pg 48.6d not deep-scrubbed since 2025-06-23T18:59:57.769702+0000
pg 58.66 not deep-scrubbed since 2025-06-22T18:23:49.451971+0000
pg 59.60 not deep-scrubbed since 2025-06-28T11:19:15.679226+0000
pg 58.60 not deep-scrubbed since 2025-06-27T14:13:20.771028+0000
pg 58.62 not deep-scrubbed since 2025-06-23T01:20:37.872422+0000
pg 58.6f not deep-scrubbed since 2025-06-27T13:40:17.031152+0000
pg 58.6b not deep-scrubbed since 2025-06-26T20:17:57.399397+0000
pg 59.6b not deep-scrubbed since 2025-06-25T07:55:45.671580+0000
pg 59.54 not deep-scrubbed since 2025-06-26T12:38:31.713680+0000
pg 48.5f not deep-scrubbed since 2025-06-27T07:22:23.643527+0000
pg 59.53 not deep-scrubbed since 2025-06-24T06:40:33.169587+0000
pg 48.58 not deep-scrubbed since 2025-06-27T17:31:04.072180+0000
pg 48.57 not deep-scrubbed since 2025-06-26T03:20:16.605485+0000
pg 48.56 not deep-scrubbed since 2025-06-28T05:41:54.098155+0000
pg 58.5f not deep-scrubbed since 2025-06-28T12:19:16.661402+0000
pg 48.52 not deep-scrubbed since 2025-06-24T14:49:29.295937+0000
pg 48.50 not deep-scrubbed since 2025-06-28T01:09:37.227387+0000
pg 48.4f not deep-scrubbed since 2025-06-20T01:16:09.369233+0000
pg 58.46 not deep-scrubbed since 2025-06-26T19:20:06.436318+0000
pg 59.40 not deep-scrubbed since 2025-06-28T13:21:24.493082+0000
pg 59.4d not deep-scrubbed since 2025-06-24T14:10:53.198832+0000
pg 58.4c not deep-scrubbed since 2025-06-22T18:24:20.137028+0000
pg 48.43 not deep-scrubbed since 2025-06-18T23:08:02.129787+0000
pg 59.49 not deep-scrubbed since 2025-06-26T03:05:00.546786+0000
pg 48.41 not deep-scrubbed since 2025-06-20T09:38:33.831563+0000
pg 48.40 not deep-scrubbed since 2025-06-22T21:41:26.468848+0000
pg 47.b not deep-scrubbed since 2025-06-28T11:03:41.301804+0000
pg 46.a not deep-scrubbed since 2025-06-26T14:59:32.312819+0000
220 more pgs...
[WRN] POOL_TARGET_SIZE_BYTES_OVERCOMMITTED: 1 subtrees have overcommitted pool
target_size_bytes
Regards
Dev
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
