Hi all,
Ceph 19.2.3 (Squid), cephadm-deployed, 528 OSDs.
After a drive failure on one host (sdc, ~10 min of SCSI retry stalls at
boot), ceph health detail now reports 13 CEPHADM_FAILED_DAEMON warnings
on that host:
daemon osd.266 on ceph-bh03 is in error state
daemon osd.267 on ceph-bh03 is in error state
... (13 total)
ceph orch ps confirms the same 13 as error. But checking each container
directly:
podman ps -a --filter name=osd.275
... Up 3 days
12 of the 13 containers are running fine, no restarts, clean RocksDB
stats in their logs. Only 1 (the OSD that actually sat on the dead
drive) has no container at all.
So it looks like cephadm's daemon-state cache went stale for the 12
healthy ones.
Has anyone seen this before? Is "ceph mgr fail" the right way to force a
clean orchestrator re-sync here, or is there a more targeted fix
(per-daemon redeploy, restarting the cephadm mgr module)?
Thanks,
Soroush
--
Soroush Pasandideh | Universität Osnabrück - Rechenzentrum | 0541 969 2336
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
