I'm not the best choice for a safe answer, but probably you should also
declare the pools replica/erasure configuration and what is the domain
failure choice: in my experience losing a node should not bring to pg
unknown.
Anyway I suggest to first understand why this happened and try to bring
all the pg to a clean state, before doing any other action on upper layers.
Bests.
Il 22/06/2026 12:46, Alessandro De Salvo via ceph-users ha scritto:
Dear ceph-users,
we are looking for an advice before taking any destructive action on a
production Ceph cluster (18.2.4) after a SAN failure.
We lost access to a Disk System hosting several OSDs. The bulk of the
recovery/backfills did complete, and most PGs are now `active+clean`,
but a small set of PGs remains in status `unavailable`. The Disk
System might still be recoverable, but not easily or quickly; it may
take days, or it may never come back. We are trying to understand the
risk of waiting versus marking the missing OSDs as lost.
We have not marked any OSDs as lost yet.
This is a Ceph Reef cluster, version 18.2.4.
Sanitized current status:
health: HEALTH_WARN
1 filesystem is degraded
1 MDS reports slow metadata IOs
noscrub,nodeep-scrub flags set
(temporarily enabled by us to reduce scrub load during
recovery/rebalancing)
Reduced data availability: 28 PGs inactive, 12 PGs down
slow ops still present
osd:
86 OSDs total
69 up/in
17 OSDs unavailable because of the SAN issue
pgs:
3320 active+clean
16 unknown
11 down
1 down+remapped
The problematic PGs are spread across several services/pools:
Main CephFS data pool: 4 PGs down
Secondary CephFS metadata/data: 1 metadata PG unknown, 4 data PGs unknown
RGW/S3 service A: 2 bucket-data PGs down + 1 metadata PG unknown
RGW/S3 service B: 5 mostly empty bucket-data PGs down + 1 metadata PG
unknown
RBD/Kubernetes: 1 PG unknown
The services we care about most are the filesystems, especially the
main CephFS and the secondary CephFS.
For the main CephFS, the metadata pool does not appear to have
inactive PGs. The remaining problematic PGs are in the data pool. Of
the main CephFS data PGs, only one appears to contain real data; the
others appear empty.
For the non-empty main CephFS data PG, `pg query` shows roughly:
state: down
acting/up: all current acting OSDs are up
last_update == last_complete
empty: 0
incomplete: 0
stat_sum:
num_bytes: ~80 GB
num_objects: ~131k
num_objects_missing: 0
num_objects_unfound: 0
num_objects_degraded: 0
num_objects_misplaced: 0
blocked:
peering is blocked due to down OSDs
down_osds_we_would_probe:
a few OSDs from previous intervals
peering_blocked_by:
"starting or marking this osd lost may let us proceed"
The recovery state for this PG includes past intervals with OSDs that
are currently unavailable. The current acting set is up, but Ceph
refuses to activate the PG because it still wants to probe down OSDs
from the PG history.
We tried to list objects from that PG:
rados -p <main_cephfs_data_pool> ls --pgid <pgid>
but it hangs/times out, so we cannot currently map RADOS objects back
to CephFS inode/path names.
For the secondary CephFS, the situation is more concerning because one
metadata PG is `unknown`. Most `pg query` attempts on unknown PGs return:
Error ENOENT: i don't have pgid <pgid>
For the RGW/S3 pools, a couple of bucket-data PGs are non-empty and
down, while some others appear empty. Some metadata PGs are unknown.
RGW can temporarily work after restarting daemons, but eventually gets
stuck again, presumably when requests hit unavailable RADOS objects or
metadata.
We also tested some RBD/Kubernetes volumes. For the Kubernetes RBD
pool, only one non-critical PVC seems to hang on `rbd info`; most
other images respond. So the main operational concern remains CephFS
and RGW.
Our questions are:
1. In a case like this, where a CephFS data PG is `down`, not
`incomplete`, and shows `num_objects_missing = 0` and
`num_objects_unfound = 0`, but peering is blocked by historical down
OSDs, what are the likely consequences of marking those OSDs as lost?
2. If we mark the unavailable OSDs as lost, should we expect the main
CephFS to:
- come back with only some files or file extents unreadable/corrupt,
- remain blocked/degraded,
- or risk a wider filesystem-level failure?
3. Since the main CephFS metadata pool does not appear to have
inactive PGs, is it reasonable to assume that the risk is mostly
limited to file data objects rather than the entire namespace? Or is
that assumption unsafe?
4. For the secondary CephFS, where a metadata PG is `unknown`, should
we treat declaring OSDs lost as much more dangerous? Could that make
the whole filesystem unusable rather than only losing some files?
5. Is there any recommended way to map the affected CephFS data PG to
file paths while the PG is still down? `rados ls --pgid` hangs, so we
currently cannot list objects in the PG.
6. Before declaring the unavailable OSDs lost, are there any
additional diagnostic or recovery steps you would recommend to better
assess the situation or reduce the risk of data loss?
Any advice would be greatly appreciated.
Best regards,
Nadir and Alessandro
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
--
ing. Sergio Rabellino
Università degli Studi di Torino
Dipartimento di Informatica
Tecnico di Ricerca
Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603
C.so Svizzera , 185 - 10149 - Torino
<http://www.di.unito.it>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
