Eugen,
Thanks for the hint. Here is the osd_remove_queue:
[root@ijc-mon1 ~]# ceph config-key get mgr/cephadm/osd_remove_queue|jq
[
{
"osd_id": 253,
"started": true,
"draining": false,
"stopped": false,
"replace": true,
"force": false,
"zap": true,
"hostname": "dig-osd4",
"drain_started_at": null,
"drain_stopped_at": null,
"drain_done_at": "2025-04-15T14:09:30.521534Z",
"process_started_at": "2025-04-15T14:09:14.091592Z"
},
{
"osd_id": 381,
"started": true,
"draining": false,
"stopped": false,
"replace": true,
"force": false,
"zap": false,
"hostname": "dig-osd6",
"drain_started_at": "2025-04-23T11:56:09.864724Z",
"drain_stopped_at": null,
"drain_done_at": "2025-04-25T06:53:03.678729Z",
"process_started_at": "2025-04-23T11:56:05.924923Z"
}
]
It is not empty the two stray daemons are listed. Not sure it these
entries are expected as I specified --replace... A similar issue was
reported in https://tracker.ceph.com/issues/67018 so before Reef but the
cause may be different. Still not clear for me how to get out of this,
except may be replacing the OSDs but this will take some time...
Best regards,
Michel
Le 27/04/2025 à 10:21, Eugen Block a écrit :
Hi,
what's the current ceph status? Wasn't there a bug in early Reef
versions preventing upgrades if there were removed OSDs in the queue?
But IIRC, the cephadm module would crash. Can you check
ceph config-key get mgr/cephadm/osd_remove_queue
And then I would check the mgr log, maybe set it to a higher debug
level to see what's blocking it.
Zitat von Michel Jouvin <michel.jou...@ijclab.in2p3.fr>:
Hi,
I tried to restart all the mgrs (we have 3, 1 active, 2 standby) by
executing 3 times the `ceph mgr fail`, no impact. I don't really
understand why I get these stray daemons after doing a 'ceph orch osd
rm --replace` but I think I have always seen this. I tried to mute
rather than disable the stray daemon check but it doesn't help
either. And I find strange this message every 10s about one of the
destroyed OSD and only one, reporting it is down and already
destroyed and saying it'll zap it (I think I didn't add --zap when I
removed it as the underlying disk is dead).
I'm completely stuck with this upgrade and I don't remember having
this kind of problems in previous upgrades with cephadm... Any idea
where to look for the cause and/or how to fix it?
Best regards,
Michel
Le 24/04/2025 à 23:34, Michel Jouvin a écrit :
Hi,
I'm trying to upgrade a (cephadm) cluster from 18.2.2 to 18.2.6,
using 'ceph orch upgrade'. When I enter the command 'ceph orch
upgrade start --ceph-version 18.2.6', I receive a message saying
that the upgrade has been initiated, with a similar message in the
logs but nothing happens after this. 'ceph orch upgrade status' says:
-------
[root@ijc-mon1 ~]# ceph orch upgrade status
{
"target_image": "quay.io/ceph/ceph:v18.2.6",
"in_progress": true,
"which": "Upgrading all daemon types on all hosts",
"services_complete": [],
"progress": "",
"message": "",
"is_paused": false
}
-------
The first time I entered the command, the cluster status was
HEALTH_WARN because of 2 stray daemons (caused by destroyed OSDs, rm
--replace). I set mgr/cephadm/warn_on_stray_daemons to false to
ignore these 2 daemons, the cluster is now HEALTH_OK but it doesn't
help. Following a Red Hat KB entry, I tried to failover the mgr,
stopped an restarted the upgrade but without any improvement. I have
not seen anything in the logs, except that there is an INF entry
every 10s about the destroyed OSD saying:
------
2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028)
14079 : cephadm [INF] osd.253 now down
2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028)
14080 : cephadm [INF] Daemon osd.253 on dig-osd4 was already removed
2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028)
14081 : cephadm [INF] Successfully destroyed old osd.253 on
dig-osd4; ready for replacement
2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028)
14082 : cephadm [INF] Zapping devices for osd.253 on dig-osd4
-----
The message seems to be only for one of the 2 destroyed OSDs since I
restarted the mgr. May this be the cause for the stucked upgrade?
What can I do for fixing this?
Thanks in advance for any hint. Best regards,
Michel
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
