Hi,
I have a problem with large amounts of dark data/orphans on cephfs data
pool. Data is stored mostly on a 8+3 EC pool (cephfs_data_echdd).
There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M
files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used
consistent with 8+3 EC.
The layout for most files:
ceph.dir.layout="stripe_unit=16777216 stripe_count=1
object_size=16777216 pool=cephfs_data_echdd"
But there is 1.4PiB discrepancy between the pool and the filesystem
which I cannot explain and I suspect there are a lot of orphan objects
there. I have run mds scrub on / and ~mdsdir as well. There is some mds
damage on some old small files (~400 files), which I do not think it's
relevant here.
ceph was created with 17.2.6 and runs 19.2.3.
Do you might have any suggestions how to fix this?
Thanks,
Andrej
spod01 ~# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 11 PiB 1.5 PiB 9.7 PiB 9.7 PiB 86.28
nvme 363 TiB 356 TiB 6.8 TiB 6.8 TiB 1.87
ssd 3.5 TiB 3.0 TiB 478 GiB 478 GiB 13.37
TOTAL 12 PiB 1.9 PiB 9.7 PiB 9.7 PiB 83.68
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED
MAX AVAIL
.mgr 1 1 850 MiB 213 2.5 GiB 0
178 TiB
cephfs_data 2 256 28 GiB 134.71M 86 GiB 2.86
972 GiB
cephfs_metadata 3 64 66 GiB 6.66M 199 GiB 6.38
972 GiB
cephfs_data_echdd 19 4096 7.0 PiB 594.70M 9.7 PiB 94.90
388 TiB
cephfs_data_ecnvme 26 64 1.2 TiB 2.84M 1.7 TiB 0.48
251 TiB
...
--
_____________________________________________________________
prof. dr. Andrej Filipcic, E-mail: [email protected]
Department of Experimental High Energy Physics - F9
Jozef Stefan Institute, Jamova 39, P.o.Box 3000
SI-1001 Ljubljana, Slovenia
Tel.: +386-1-477-3674 Fax: +386-1-477-3166
-------------------------------------------------------------
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
