Hi,
I don't really have simple answers here, but I'll comment inline.
Zitat von Alessandro De Salvo via ceph-users <[email protected]>:
Dear ceph-users,
we are looking for an advice before taking any destructive action on
a production Ceph cluster (18.2.4) after a SAN failure.
We lost access to a Disk System hosting several OSDs. The bulk of
the recovery/backfills did complete, and most PGs are now
`active+clean`, but a small set of PGs remains in status
`unavailable`. The Disk System might still be recoverable, but not
easily or quickly; it may take days, or it may never come back. We
are trying to understand the risk of waiting versus marking the
missing OSDs as lost.
We have not marked any OSDs as lost yet.
This is a Ceph Reef cluster, version 18.2.4.
Sanitized current status:
health: HEALTH_WARN
1 filesystem is degraded
1 MDS reports slow metadata IOs
noscrub,nodeep-scrub flags set
(temporarily enabled by us to reduce scrub load during recovery/rebalancing)
Reduced data availability: 28 PGs inactive, 12 PGs down
slow ops still present
osd:
86 OSDs total
69 up/in
17 OSDs unavailable because of the SAN issue
pgs:
3320 active+clean
16 unknown
11 down
1 down+remapped
The problematic PGs are spread across several services/pools:
Main CephFS data pool: 4 PGs down
Secondary CephFS metadata/data: 1 metadata PG unknown, 4 data PGs unknown
RGW/S3 service A: 2 bucket-data PGs down + 1 metadata PG unknown
RGW/S3 service B: 5 mostly empty bucket-data PGs down + 1 metadata PG unknown
RBD/Kubernetes: 1 PG unknown
Could you share the following output:
ceph osd tree
ceph osd pool ls detail
ceph health detail
The services we care about most are the filesystems, especially the
main CephFS and the secondary CephFS.
For the main CephFS, the metadata pool does not appear to have
inactive PGs. The remaining problematic PGs are in the data pool. Of
the main CephFS data PGs, only one appears to contain real data; the
others appear empty.
I find it very unlikely that only one of the PGs has data. If there
are down OSDs, they can't report to the mgr, so the stats are most
likely wrong.
For the non-empty main CephFS data PG, `pg query` shows roughly:
state: down
acting/up: all current acting OSDs are up
last_update == last_complete
empty: 0
incomplete: 0
stat_sum:
num_bytes: ~80 GB
num_objects: ~131k
num_objects_missing: 0
num_objects_unfound: 0
num_objects_degraded: 0
num_objects_misplaced: 0
blocked:
peering is blocked due to down OSDs
down_osds_we_would_probe:
a few OSDs from previous intervals
peering_blocked_by:
"starting or marking this osd lost may let us proceed"
The recovery state for this PG includes past intervals with OSDs
that are currently unavailable. The current acting set is up, but
Ceph refuses to activate the PG because it still wants to probe down
OSDs from the PG history.
We tried to list objects from that PG:
rados -p <main_cephfs_data_pool> ls --pgid <pgid>
but it hangs/times out, so we cannot currently map RADOS objects
back to CephFS inode/path names.
For the secondary CephFS, the situation is more concerning because
one metadata PG is `unknown`. Most `pg query` attempts on unknown
PGs return:
Error ENOENT: i don't have pgid <pgid>
Same applies here, down OSDs can't report anything.
For the RGW/S3 pools, a couple of bucket-data PGs are non-empty and
down, while some others appear empty. Some metadata PGs are unknown.
RGW can temporarily work after restarting daemons, but eventually
gets stuck again, presumably when requests hit unavailable RADOS
objects or metadata.
We also tested some RBD/Kubernetes volumes. For the Kubernetes RBD
pool, only one non-critical PVC seems to hang on `rbd info`; most
other images respond. So the main operational concern remains CephFS
and RGW.
Our questions are:
1. In a case like this, where a CephFS data PG is `down`, not
`incomplete`, and shows `num_objects_missing = 0` and
`num_objects_unfound = 0`, but peering is blocked by historical down
OSDs, what are the likely consequences of marking those OSDs as lost?
Data loss. IIRC, a down OSD means there are no OSDs to query at all
(for that PG) while incomplete means that there are not enough OSDs
"up" to determine which one has the most up-to-date state.
2. If we mark the unavailable OSDs as lost, should we expect the
main CephFS to:
- come back with only some files or file extents unreadable/corrupt,
- remain blocked/degraded,
- or risk a wider filesystem-level failure?
That's difficult to answer. Losing an entire OSD can mean several TB
of data lost, so it's unlikely that "only some files or file extents"
will be lost. I think it would allow Ceph to recover further, most
likely you'd have to scrub the FS afterwards, but at some point you
should have a working FS, I think.
3. Since the main CephFS metadata pool does not appear to have
inactive PGs, is it reasonable to assume that the risk is mostly
limited to file data objects rather than the entire namespace? Or is
that assumption unsafe?
I tend to concur, but what do you mean by "does not appear to have
inactive PGs"? The 'ceph health detail' output should have a clear
answer to that.
4. For the secondary CephFS, where a metadata PG is `unknown`,
should we treat declaring OSDs lost as much more dangerous? Could
that make the whole filesystem unusable rather than only losing some
files?
There are ways to recover the metadata, but be very careful with that!
https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/
5. Is there any recommended way to map the affected CephFS data PG
to file paths while the PG is still down? `rados ls --pgid` hangs,
so we currently cannot list objects in the PG.
I don't think so.
6. Before declaring the unavailable OSDs lost, are there any
additional diagnostic or recovery steps you would recommend to
better assess the situation or reduce the risk of data loss?
Is there no chance to access the missing OSDs? So one could at least
export PGs or something?
Any advice would be greatly appreciated.
Best regards,
Nadir and Alessandro
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
