On 3. 7. 2026 02:05, Anthony D'Atri via ceph-users wrote:
A couple of ideas:
Do you have osd_recovery_sleep_* set?
osd_recovery_sleep_hdd 0.100000
osd_delete_sleep_*?
not that one
Do you have any pools in the middle of pg_num adjustments, such that pg_num !=
pgp_num and a pg_num_target is reported?
no, that one is fine, we have autoscaling disabled.
Best,
Andrej
On Jul 2, 2026, at 2:52 PM, Andras Pataki via ceph-users <[email protected]>
wrote:
Hi,
I have also seen this on a recent large scale/parallel test I can on a test setup. After
creating about a hundred million files, then removing all of them - there were thousands
of objects left in the cephfs data pool corresponding to inodes that no longer existed.
I.e. it looks like the MDS is "leaking" data - does not correctly delete all
objects corresponding to files in the purge queue. We have never used any snapshots on
this file system, so it definitely isn't about snapshots holding data. Also, we
increased the purge queue settings (filer_max_purge_ops, mds_max_purge_* and friends) and
definitely waited long enough for all deletes to be processed (the objects were there
weeks after the file system was emptied). This was on a squid 19.2.3 cluster.
In the tests that lead to this, both the file creation and deletions were done
in parallel using a few dozen clients - stressing the MDS for sure. We had no
crashes/problems on the cluster, we never had to do any data recovery steps,
i.e. the MDS appeared to work fine through the tests. Also, this was a single
MDS cluster, i.e. the problem isn't related to subtrees moving across MDS's for
example. The problem became obvious since after removing all data from cephfs,
we were expecting the data pools to be empty, but they weren't. There were two
pools, a primary triple replicated one and an erasure coded one. Both of them
had stray objects.
I didn't pursue this further since I wasn't sure what useful information I
could gather for a bug report - but it is certainly a curious observation that
perhaps large, long living cephfs clusters might have significant space tied up
in these objects that should have been removed but weren't. Short of a full
scan of all objects and matching them to inodes - it is hard to tell how much
even.
Andras
On 7/2/26 12:17 PM, Andrej Filipčič via ceph-users wrote:
So, to follow up on this, I did some further investigation.
Checking for write amplification, I have copied 250TB of a mix of small and
large files (20M of them), and the stored space on EC pool matched what was
expected from actual data size to a few %. So EC overhead was not really a
factor. Also, after the removal of this data set, the stored space was
recovered as expected.
The I checked the full dump of cephfs and compared it to list of all objects in
the EC pool as follows:
rados -p cephfs_data_echdd ls > echdd.objectlist
find /ceph/ -printf "%i %p\n" > cephfs.inodes
this took several hours, while in the meantime, writing and removing of data to
cephfs was relatively low (few MB/s), so the impact of new objects and files
should have been minimal.
Then I selected all the objects of the form
1007b28abae.00000000
100cce97f6d.00000000
10067733861.00000000
100cc4646aa.00000000
200044d5c07.00000000
...
and checked if they match the files in the cephfs.inodes list
~3M of *.00000000 objects do not have the corresponding inode in the
cephfs.inodes
I did "rados stat" of ~70k of these objects (still running for all of them),
and almost all have timestamps from March and April these year. On 4th of May I remember
I have increased the mds purge queue values to speed up the removal of files since OSDs
were filling up too quickly, and since then it seems the objects are not left uncleaned
any more.
With "rados getxattr objectid parent" I have checked several of them, and they
all belong to two very active projects which typically write files with few GB/s all the
time (and remove as well so space usage is not increasing). The objects I have checked
belonged to files that were removed in cephfs, the projects have a separate file catalog
which is consistent with cephfs contents.
So, I do not understand why so many objects were left unremoved in EC pool in
the period of 2 months, but at least, 3M uncleaned files explains 1.5PB of
dark data by quick estimate.
We also scrubbed cephfs root and ~mdsdir several times and no leftovers to
remove were found.
I hope the problem is gone now, but I would still like a good advice on how to
proceed with the cleanup. I see these options:
1) remove the unmatched objects directly from EC pool with "rados rm". But
this might have undesired side effects or corruption.
2) create a new EC pool and migrate all the data there by copying files in
filesystem, and then destroy the old pool.
3) create a new filesystem with new EC pools and migrate the files.
4) run some advanced MDS disaster recovery procedure (cephfs-data-scan), but
this requires offline FS and I do not want to recover old files from existing
unmatched objects. Anyway, cephfs seems to be healthy now.
Any good ideas?
Best,
Andrej
On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote:
).
There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M files and
139M rentries. The pool shows 7PiB stored and 9.7PiB used consistent with 8+3
EC.
The layout for most files:
ceph.dir.layout="stripe_unit=16777216 stripe_count=1 object_size=16777216
pool=cephfs_data_echdd"
But there is 1.4PiB discrepancy between the pool and the filesystem
Do you have scrubs enabled? Which if any non-default config options do you have
set? Any undersized or degraded or backfilling PGs?
Which Ceph release? Do you have a sizable fraction of small files? If you’re
running Squid or earlier or don’t have EC optimizations enabled, even a tiny
file will allocate a multiple of 11*16=176 KB. An 129KB file will consume
352KB, etc. If I understand those layout options correctly.
If that’s what’s going on, going to Tentacle with EC optimizations would gain
you some efficiency for files newly [re]written. You could also migrate small
files to a replicated pool.
which I cannot explain and I suspect there are a lot of orphan objects there. I
have run mds scrub on / and ~mdsdir as well. There is some mds damage on some
old small files (~400 files), which I do not think it's relevant here.
Hello,
We had a similar issue last year with a group of users that created and removed
files at a very high rate.
Have you read
https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/cephfs/*cephfs-pool-data-usage-growth-without-explanation__;Iw!!DSb-azq1wVFtOg!XcB7cJMEzAXsAddGg1LH5ff1B33dit2O1vAhxnVlv2MMUrC85oUKgukdaytYJQqAfmmjiTWJhr7GLPIKSTbwQpbx18c$
?
We increased the purge rate parameters (very) aggressively to get back to a
comfortable situation (i.e. not a pool w/ near full warnings).
Loïc.
--
_____________________________________________________________
prof. dr. Andrej Filipcic, E-mail: [email protected]
Department of Experimental High Energy Physics - F9
Jozef Stefan Institute, Jamova 39, P.o.Box 3000
SI-1001 Ljubljana, Slovenia
Tel.: +386-1-477-3674 Fax: +386-1-477-3166
-------------------------------------------------------------
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
--
_____________________________________________________________
prof. dr. Andrej Filipcic, E-mail: [email protected]
Department of Experimental High Energy Physics - F9
Jozef Stefan Institute, Jamova 39, P.o.Box 3000
SI-1001 Ljubljana, Slovenia
Tel.: +386-1-477-3674 Fax: +386-1-477-3166
-------------------------------------------------------------
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
