Here is the status
# ceph orch upgrade status
{
"target_image":
"quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6",
"in_progress": true,
"which": "Upgrading all daemon types on all hosts",
"services_complete": [
"crash",
"mgr",
"mon"
],
"progress": "74/113 daemons upgraded",
"message": "",
"is_paused": false
}
Regards
Dev
> On Jun 19, 2025, at 8:06 PM, Devender Singh <deven...@netskrt.io> wrote:
>
>
> Hello all
>
> I have a cluster where my cluster is in hung state, Some back fills are there
> but I reduced it to 1 but still upgrade not progressing…
> Please help…
>
> ```# ceph health detail
> HEALTH_WARN 8 OSD(s) experiencing slow operations in BlueStore; Failed to
> apply 2 service(s): osd.all-available-devices,osd.iops_optimized; 1 failed
> cephadm daemon(s); failed to probe daemons or devices; noscrub,nodeep-scrub
> flag(s) set; Degraded data redundancy: 1600150/39365198 objects degraded
> (4.065%), 93 pgs degraded, 103 pgs undersized; 127 pgs not deep-scrubbed in
> time
> [WRN] BLUESTORE_SLOW_OP_ALERT: 8 OSD(s) experiencing slow operations in
> BlueStore
> osd.5 observed slow operation indications in BlueStore
> osd.9 observed slow operation indications in BlueStore
> osd.18 observed slow operation indications in BlueStore
> osd.36 observed slow operation indications in BlueStore
> osd.59 observed slow operation indications in BlueStore
> osd.66 observed slow operation indications in BlueStore
> osd.106 observed slow operation indications in BlueStore
> osd.110 observed slow operation indications in BlueStore
> [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 2 service(s):
> osd.all-available-devices,osd.iops_optimized
> osd.all-available-devices: Command timed out on host cephadm deploy (osd
> daemon) (default 900 second timeout)
> osd.iops_optimized: Command timed out on host cephadm deploy (osd daemon)
> (default 900 second timeout)
> [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
> daemon osd.19 on phl-prod-host04n.example.comis in error state
> [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
> Command "cephadm ceph-volume -- inventory" timed out on host
> phl-prod-converged03n.phl.netskrt.org (default 900 second timeout)
> [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set
> [WRN] PG_DEGRADED: Degraded data redundancy: 1600150/39365198 objects
> degraded (4.065%), 93 pgs degraded, 103 pgs undersized
> pg 25.0 is stuck undersized for 21h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [90,77,NONE,38,3]
> pg 25.1 is stuck undersized for 23h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [131,1,66,NONE,72]
> pg 25.c is stuck undersized for 23h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [1,135,110,28,NONE]
> pg 25.e is active+undersized+degraded+remapped+backfill_wait, acting
> [18,20,108,101,NONE]
> pg 25.14 is stuck undersized for 19h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [32,13,NONE,65,97]
> pg 25.17 is stuck undersized for 19h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [32,65,110,23,NONE]
> pg 25.1a is stuck undersized for 23h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [97,66,NONE,112,139]
> pg 25.1c is stuck undersized for 2h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [102,136,97,NONE,45]
> pg 25.1f is stuck undersized for 8h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [NONE,66,7,39,109]
> pg 26.45 is stuck undersized for 23h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [32,47]
> pg 26.48 is stuck undersized for 23h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [113,55]
> pg 26.4b is stuck undersized for 8h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [105,7]
> pg 26.59 is stuck undersized for 23h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [35,40]
> pg 26.6a is stuck undersized for 7h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [25,45]
> pg 26.74 is stuck undersized for 23h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [112,131]
> pg 26.77 is stuck undersized for 2h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [17,105]
> pg 26.9c is stuck undersized for 19h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [47,76]
> pg 26.bf is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [32,26]
> pg 26.c2 is stuck undersized for 2h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [97,135]
> pg 26.ec is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [26,80]
> pg 26.f7 is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting [49,105]
> pg 31.12 is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [47,110,131,37,NONE]
> pg 31.19 is stuck undersized for 2h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [NONE,87,3,137,39]
> pg 31.1b is stuck undersized for 2h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [108,NONE,42,102,97]
> pg 31.1c is stuck undersized for 19h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [113,66,NONE,45,72]
> pg 31.41 is stuck undersized for 2h, current state
> active+undersized+remapped+backfill_wait, last acting [14,101,NONE,67,18]
> pg 31.42 is stuck undersized for 7h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [37,134,NONE,82,8]
> pg 31.44 is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [49,NONE,101,3,62]
> pg 31.46 is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfilling, last acting
> [95,38,NONE,25,102]
> pg 31.47 is stuck undersized for 18h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [60,NONE,130,72,110]
> pg 31.4a is stuck undersized for 4h, current state
> active+undersized+degraded+remapped+backfilling, last acting
> [66,NONE,135,82,14]
> pg 31.4c is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [34,NONE,1,82,18]
> pg 31.4d is stuck undersized for 2h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [101,13,65,30,NONE]
> pg 31.52 is stuck undersized for 4m, current state
> active+undersized+remapped+backfill_wait, last acting [26,112,66,NONE,135]
> pg 31.53 is stuck undersized for 17h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [NONE,24,11,42,110]
> pg 31.55 is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [116,NONE,27,4,117]
> pg 31.57 is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [17,15,110,NONE,1]
> pg 31.5a is stuck undersized for 19h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [13,24,124,26,NONE]
> pg 31.5b is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [NONE,31,8,27,42]
> pg 31.5c is stuck undersized for 2h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [NONE,60,28,18,119]
> pg 31.5d is stuck undersized for 17h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [124,NONE,85,6,11]
> pg 31.5e is stuck undersized for 2h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [NONE,4,1,23,138]
> pg 31.60 is stuck undersized for 18h, current state
> active+undersized+degraded+remapped+backfilling, last acting
> [37,11,102,NONE,133]
> pg 31.64 is stuck undersized for 4m, current state
> active+undersized+remapped+backfill_wait, last acting [26,106,45,34,NONE]
> pg 31.65 is stuck undersized for 4h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [NONE,14,127,62,3]
> pg 31.66 is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [34,NONE,35,23,59]
> pg 31.67 is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [67,24,127,8,NONE]
> pg 31.68 is stuck undersized for 4m, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [32,41,18,17,NONE]
> pg 31.70 is stuck undersized for 19h, current state
> active+undersized+degraded+remapped+backfilling, last acting
> [106,38,1,NONE,97]
> pg 31.75 is stuck undersized for 2h, current state
> active+undersized+degraded+remapped+backfill_wait, last acting
> [26,31,34,8,NONE]
> pg 31.7f is stuck undersized for 2h, current state
> active+undersized+remapped+backfilling, last acting [NONE,101,110,10,45]
> [WRN] PG_NOT_DEEP_SCRUBBED: 127 pgs not deep-scrubbed in time
> pg 26.e5 not deep-scrubbed since 2025-06-07T05:59:22.612553+0000
> pg 26.ca not deep-scrubbed since 2025-06-06T18:21:07.435823+0000
> pg 26.bf not deep-scrubbed since 2025-06-03T07:29:02.791095+0000
> pg 26.be not deep-scrubbed since 2025-06-04T07:55:54.389714+0000
> pg 26.a5 not deep-scrubbed since 2025-06-07T07:28:56.878096+0000
> pg 26.85 not deep-scrubbed since 2025-06-03T15:18:59.395929+0000
> pg 26.7f not deep-scrubbed since 2025-06-04T21:29:28.412637+0000
> pg 26.7e not deep-scrubbed since 2025-06-03T09:14:19.585388+0000
> pg 26.7d not deep-scrubbed since 2025-06-03T18:37:30.931020+0000
> pg 26.7c not deep-scrubbed since 2025-06-04T13:38:00.061488+0000
> pg 26.73 not deep-scrubbed since 2025-06-03T06:20:15.111819+0000
> pg 26.6f not deep-scrubbed since 2025-06-03T13:45:24.880397+0000
> pg 26.6e not deep-scrubbed since 2025-05-26T23:15:32.099862+0000
> pg 26.6d not deep-scrubbed since 2025-06-04T14:04:10.449101+0000
> pg 31.62 not deep-scrubbed since 2025-06-03T13:34:49.518456+0000
> pg 26.65 not deep-scrubbed since 2025-06-04T07:56:25.353411+0000
> pg 31.66 not deep-scrubbed since 2025-06-03T10:32:05.364424+0000
> pg 26.62 not deep-scrubbed since 2025-06-04T09:35:58.267976+0000
> pg 31.65 not deep-scrubbed since 2025-06-03T16:04:40.003140+0000
> pg 31.5b not deep-scrubbed since 2025-06-03T14:18:18.835477+0000
> pg 26.5d not deep-scrubbed since 2025-06-04T15:14:30.870252+0000
> pg 31.58 not deep-scrubbed since 2025-06-03T03:09:27.568605+0000
> pg 26.5c not deep-scrubbed since 2025-06-03T01:57:27.644129+0000
> pg 31.5f not deep-scrubbed since 2025-06-03T05:53:20.860393+0000
> pg 31.52 not deep-scrubbed since 2025-05-27T00:01:27.040861+0000
> pg 31.53 not deep-scrubbed since 2025-05-24T09:37:58.964829+0000
> pg 26.55 not deep-scrubbed since 2025-06-04T21:25:34.135356+0000
> pg 26.54 not deep-scrubbed since 2025-06-04T06:07:12.978734+0000
> pg 31.56 not deep-scrubbed since 2025-06-04T12:58:17.599712+0000
> pg 31.57 not deep-scrubbed since 2025-06-03T07:02:16.859990+0000
> pg 26.51 not deep-scrubbed since 2025-06-03T05:42:22.435483+0000
> pg 26.4f not deep-scrubbed since 2025-06-03T09:10:22.617328+0000
> pg 31.4a not deep-scrubbed since 2025-05-28T00:54:55.246532+0000
> pg 26.4e not deep-scrubbed since 2025-06-03T11:16:49.278513+0000
> pg 31.4b not deep-scrubbed since 2025-06-03T10:24:24.123351+0000
> pg 26.4d not deep-scrubbed since 2025-06-04T19:01:44.614410+0000
> pg 31.49 not deep-scrubbed since 2025-05-28T04:56:29.368285+0000
> pg 31.42 not deep-scrubbed since 2025-05-28T08:38:57.151865+0000
> pg 26.41 not deep-scrubbed since 2025-06-03T05:47:35.443867+0000
> pg 26.40 not deep-scrubbed since 2025-06-03T05:43:13.283668+0000
> pg 25.17 not deep-scrubbed since 2025-06-03T09:26:52.253625+0000
> pg 32.2e not deep-scrubbed since 2025-06-05T13:01:06.175389+0000
> pg 22.1a not deep-scrubbed since 2025-06-07T01:40:45.063268+0000
> pg 31.13 not deep-scrubbed since 2025-06-03T12:21:17.965218+0000
> pg 22.1b not deep-scrubbed since 2025-06-04T12:22:44.947751+0000
> pg 25.14 not deep-scrubbed since 2025-05-28T06:26:32.552200+0000
> pg 26.17 not deep-scrubbed since 2025-06-03T18:37:26.617483+0000
> pg 31.12 not deep-scrubbed since 2025-05-28T08:39:23.271194+0000
> pg 31.1c not deep-scrubbed since 2025-06-03T08:17:51.230187+0000
> pg 25.1f not deep-scrubbed since 2025-05-28T00:19:23.653883+0000
> 77 more pgs…
>
> Regards
> Dev
>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
