Dear ceph-users,
we are looking for an advice before taking any destructive action on a
production Ceph cluster (18.2.4) after a SAN failure.
We lost access to a Disk System hosting several OSDs. The bulk of the
recovery/backfills did complete, and most PGs are now `active+clean`,
but a small set of PGs remains in status `unavailable`. The Disk System
might still be recoverable, but not easily or quickly; it may take days,
or it may never come back. We are trying to understand the risk of
waiting versus marking the missing OSDs as lost.
We have not marked any OSDs as lost yet.
This is a Ceph Reef cluster, version 18.2.4.
Sanitized current status:
health: HEALTH_WARN
1 filesystem is degraded
1 MDS reports slow metadata IOs
noscrub,nodeep-scrub flags set
(temporarily enabled by us to reduce scrub load during recovery/rebalancing)
Reduced data availability: 28 PGs inactive, 12 PGs down
slow ops still present
osd:
86 OSDs total
69 up/in
17 OSDs unavailable because of the SAN issue
pgs:
3320 active+clean
16 unknown
11 down
1 down+remapped
The problematic PGs are spread across several services/pools:
Main CephFS data pool: 4 PGs down
Secondary CephFS metadata/data: 1 metadata PG unknown, 4 data PGs unknown
RGW/S3 service A: 2 bucket-data PGs down + 1 metadata PG unknown
RGW/S3 service B: 5 mostly empty bucket-data PGs down + 1 metadata PG
unknown
RBD/Kubernetes: 1 PG unknown
The services we care about most are the filesystems, especially the main
CephFS and the secondary CephFS.
For the main CephFS, the metadata pool does not appear to have inactive
PGs. The remaining problematic PGs are in the data pool. Of the main
CephFS data PGs, only one appears to contain real data; the others
appear empty.
For the non-empty main CephFS data PG, `pg query` shows roughly:
state: down
acting/up: all current acting OSDs are up
last_update == last_complete
empty: 0
incomplete: 0
stat_sum:
num_bytes: ~80 GB
num_objects: ~131k
num_objects_missing: 0
num_objects_unfound: 0
num_objects_degraded: 0
num_objects_misplaced: 0
blocked:
peering is blocked due to down OSDs
down_osds_we_would_probe:
a few OSDs from previous intervals
peering_blocked_by:
"starting or marking this osd lost may let us proceed"
The recovery state for this PG includes past intervals with OSDs that
are currently unavailable. The current acting set is up, but Ceph
refuses to activate the PG because it still wants to probe down OSDs
from the PG history.
We tried to list objects from that PG:
rados -p <main_cephfs_data_pool> ls --pgid <pgid>
but it hangs/times out, so we cannot currently map RADOS objects back to
CephFS inode/path names.
For the secondary CephFS, the situation is more concerning because one
metadata PG is `unknown`. Most `pg query` attempts on unknown PGs return:
Error ENOENT: i don't have pgid <pgid>
For the RGW/S3 pools, a couple of bucket-data PGs are non-empty and
down, while some others appear empty. Some metadata PGs are unknown. RGW
can temporarily work after restarting daemons, but eventually gets stuck
again, presumably when requests hit unavailable RADOS objects or metadata.
We also tested some RBD/Kubernetes volumes. For the Kubernetes RBD pool,
only one non-critical PVC seems to hang on `rbd info`; most other images
respond. So the main operational concern remains CephFS and RGW.
Our questions are:
1. In a case like this, where a CephFS data PG is `down`, not
`incomplete`, and shows `num_objects_missing = 0` and
`num_objects_unfound = 0`, but peering is blocked by historical down
OSDs, what are the likely consequences of marking those OSDs as lost?
2. If we mark the unavailable OSDs as lost, should we expect the main
CephFS to:
- come back with only some files or file extents unreadable/corrupt,
- remain blocked/degraded,
- or risk a wider filesystem-level failure?
3. Since the main CephFS metadata pool does not appear to have inactive
PGs, is it reasonable to assume that the risk is mostly limited to file
data objects rather than the entire namespace? Or is that assumption unsafe?
4. For the secondary CephFS, where a metadata PG is `unknown`, should we
treat declaring OSDs lost as much more dangerous? Could that make the
whole filesystem unusable rather than only losing some files?
5. Is there any recommended way to map the affected CephFS data PG to
file paths while the PG is still down? `rados ls --pgid` hangs, so we
currently cannot list objects in the PG.
6. Before declaring the unavailable OSDs lost, are there any additional
diagnostic or recovery steps you would recommend to better assess the
situation or reduce the risk of data loss?
Any advice would be greatly appreciated.
Best regards,
Nadir and Alessandro
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
