test
Zitat von Eugen Block <[email protected]>:
Hi,
it's not really clear to me if you had the NVMe drives for the
CephFS metadata pool only or if you had DB/WAL for spinners as well
on those drives. I assume it was only the pool on them, regarding
your statement that there are active+clean PGs. More comments
inline...
Zitat von Enzo Van Thorre via ceph-users <[email protected]>:
Hello Ceph users and readers,
I am facing a really unknown situation for me (being kinda new)
with a CephFS cluster (Squid 19.2.2, Cephadm) where the data pool
is intact (7.2 TB used, 4.9TB stored), inode objects exist in the
recovered metadata pool, but when i try to mount the filesystem,
its empty.
What initially happened is that I had two groups of drives: nvme’s
and spinning. I had separate pools for both, and I eventually
deleted the nvme’s pool and then removed the nvmes from the
machine. I never thought that they had included metadata for
spinning rust’s pool, but here we are.
For the future: 'ceph osd ok-to-stop <ID>' can be helpful if you're
not sure if it's okay to stop an OSD. That could have prevented it
from happening.
Version: 19.2.2 Squid
Metadata pool ID 1 (cephfs.cephfs.meta), Size=1 (No Replicas :(,
yes I know I messed up).
yeah...
Data Pool ID 6 (ec_data), erasure coded with k=2 m=1.
that's not great either (k2, m1)
All OSDs are up. 256 PGs are active+clean. there’s then quite a bit
that are inconsistent, degraded, undersized, remapped, and other
stuff.
what is "other stuff"? A 'ceph health detail' could be helpful.
Here’s what I’ve done since losing the metadata pool due to OSD
purges. I tried to rebuild metadata from the data pool by doing:
1. cephfs-data-scan init --force-init
2. cephfs-data-scan scan_extents
3. cephfs-data-scan scan_inodes
ceph df shows 7.2 TiB stored in pool ec_data (which is the spinning
rust pool)
Inodes do exist. running rados -p cephfs.cephfs.meta ls | grep
"100." returns tons of objects (ex. 1000005f1f3.00000000).
If i try to mount via ceph-fuse, it shows an empty root. there is
no lost+found folder either.
A big problem is that when I try to link the orphaned inodes to
lost+found, the command finishes instantly with no output, as if it
believes the cluster is clean.
I run: cephfs-data-scan scan_links --filesystem cephfs
# and it returns code 0, no output, and runs in <1 second
So my final question is since scan_inodes has populated the
metadata pool with objects (100.xxxx), but scan_links ignores them,
how do I force a re-scan or just do SOMETHING so scan_links will
actually link these files into lost+found?
Unfortunately, I don't have a comment on that right now. But maybe
increasing debug level for MDS can help here?
All The Best,
Enzo
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
