I don't deal with CephFS much, but on RBD I had an issue like that, and it was because the filesystem needed fstrim.
-- Alex Gorbachev ISS/Storcium On Fri, Jul 3, 2026 at 3:48 AM Andrej Filipčič via ceph-users < [email protected]> wrote: > On 3. 7. 2026 02:05, Anthony D'Atri via ceph-users wrote: > > A couple of ideas: > > > > Do you have osd_recovery_sleep_* set? > > osd_recovery_sleep_hdd 0.100000 > > osd_delete_sleep_*? > not that one > > Do you have any pools in the middle of pg_num adjustments, such that > pg_num != pgp_num and a pg_num_target is reported? > no, that one is fine, we have autoscaling disabled. > > Best, > Andrej > > > >> On Jul 2, 2026, at 2:52 PM, Andras Pataki via ceph-users < > [email protected]> wrote: > >> > >> Hi, > >> > >> I have also seen this on a recent large scale/parallel test I can on a > test setup. After creating about a hundred million files, then removing > all of them - there were thousands of objects left in the cephfs data pool > corresponding to inodes that no longer existed. I.e. it looks like the MDS > is "leaking" data - does not correctly delete all objects corresponding to > files in the purge queue. We have never used any snapshots on this file > system, so it definitely isn't about snapshots holding data. Also, we > increased the purge queue settings (filer_max_purge_ops, mds_max_purge_* > and friends) and definitely waited long enough for all deletes to be > processed (the objects were there weeks after the file system was emptied). > This was on a squid 19.2.3 cluster. > >> In the tests that lead to this, both the file creation and deletions > were done in parallel using a few dozen clients - stressing the MDS for > sure. We had no crashes/problems on the cluster, we never had to do any > data recovery steps, i.e. the MDS appeared to work fine through the tests. > Also, this was a single MDS cluster, i.e. the problem isn't related to > subtrees moving across MDS's for example. The problem became obvious since > after removing all data from cephfs, we were expecting the data pools to be > empty, but they weren't. There were two pools, a primary triple replicated > one and an erasure coded one. Both of them had stray objects. > >> > >> I didn't pursue this further since I wasn't sure what useful > information I could gather for a bug report - but it is certainly a curious > observation that perhaps large, long living cephfs clusters might have > significant space tied up in these objects that should have been removed > but weren't. Short of a full scan of all objects and matching them to > inodes - it is hard to tell how much even. > >> > >> Andras > >> > >> > >>> On 7/2/26 12:17 PM, Andrej Filipčič via ceph-users wrote: > >>> > >>> So, to follow up on this, I did some further investigation. > >>> > >>> Checking for write amplification, I have copied 250TB of a mix of > small and large files (20M of them), and the stored space on EC pool > matched what was expected from actual data size to a few %. So EC overhead > was not really a factor. Also, after the removal of this data set, the > stored space was recovered as expected. > >>> > >>> The I checked the full dump of cephfs and compared it to list of all > objects in the EC pool as follows: > >>> > >>> rados -p cephfs_data_echdd ls > echdd.objectlist > >>> > >>> find /ceph/ -printf "%i %p\n" > cephfs.inodes > >>> > >>> this took several hours, while in the meantime, writing and removing > of data to cephfs was relatively low (few MB/s), so the impact of new > objects and files should have been minimal. > >>> > >>> Then I selected all the objects of the form > >>> 1007b28abae.00000000 > >>> 100cce97f6d.00000000 > >>> 10067733861.00000000 > >>> 100cc4646aa.00000000 > >>> 200044d5c07.00000000 > >>> ... > >>> > >>> and checked if they match the files in the cephfs.inodes list > >>> > >>> ~3M of *.00000000 objects do not have the corresponding inode in the > cephfs.inodes > >>> > >>> > >>> I did "rados stat" of ~70k of these objects (still running for all of > them), and almost all have timestamps from March and April these year. On > 4th of May I remember I have increased the mds purge queue values to speed > up the removal of files since OSDs were filling up too quickly, and since > then it seems the objects are not left uncleaned any more. > >>> > >>> With "rados getxattr objectid parent" I have checked several of them, > and they all belong to two very active projects which typically write files > with few GB/s all the time (and remove as well so space usage is not > increasing). The objects I have checked belonged to files that were > removed in cephfs, the projects have a separate file catalog which is > consistent with cephfs contents. > >>> > >>> So, I do not understand why so many objects were left unremoved in EC > pool in the period of 2 months, but at least, 3M uncleaned files explains > 1.5PB of dark data by quick estimate. > >>> > >>> We also scrubbed cephfs root and ~mdsdir several times and no > leftovers to remove were found. > >>> > >>> I hope the problem is gone now, but I would still like a good advice > on how to proceed with the cleanup. I see these options: > >>> > >>> 1) remove the unmatched objects directly from EC pool with "rados > rm". But this might have undesired side effects or corruption. > >>> > >>> 2) create a new EC pool and migrate all the data there by copying > files in filesystem, and then destroy the old pool. > >>> > >>> 3) create a new filesystem with new EC pools and migrate the files. > >>> > >>> 4) run some advanced MDS disaster recovery procedure > (cephfs-data-scan), but this requires offline FS and I do not want to > recover old files from existing unmatched objects. Anyway, cephfs seems to > be healthy now. > >>> > >>> Any good ideas? > >>> > >>> Best, > >>> Andrej > >>> > >>> > >>> On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote: > >>>>> ). > >>>>>>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with > 132M files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used > consistent with 8+3 EC. > >>>>>>> The layout for most files: > >>>>>>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 > object_size=16777216 pool=cephfs_data_echdd" > >>>>>>> > >>>>>>> But there is 1.4PiB discrepancy between the pool and the filesystem > >>>> Do you have scrubs enabled? Which if any non-default config options > do you have set? Any undersized or degraded or backfilling PGs? > >>>> > >>>> Which Ceph release? Do you have a sizable fraction of small files? > If you’re running Squid or earlier or don’t have EC optimizations enabled, > even a tiny file will allocate a multiple of 11*16=176 KB. An 129KB file > will consume 352KB, etc. If I understand those layout options correctly. > >>>> > >>>> If that’s what’s going on, going to Tentacle with EC optimizations > would gain you some efficiency for files newly [re]written. You could also > migrate small files to a replicated pool. > >>>> > >>>> > >>>>>>> which I cannot explain and I suspect there are a lot of orphan > objects there. I have run mds scrub on / and ~mdsdir as well. There is some > mds damage on some old small files (~400 files), which I do not think it's > relevant here. > >>>>>>> > >>>>>> Hello, > >>>>>> We had a similar issue last year with a group of users that created > and removed files at a very high rate. > >>>>>> > >>>>>> Have you read > https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/cephfs/*cephfs-pool-data-usage-growth-without-explanation__;Iw!!DSb-azq1wVFtOg!XcB7cJMEzAXsAddGg1LH5ff1B33dit2O1vAhxnVlv2MMUrC85oUKgukdaytYJQqAfmmjiTWJhr7GLPIKSTbwQpbx18c$ > ? > >>>>>> > >>>>>> We increased the purge rate parameters (very) aggressively to get > back to a comfortable situation (i.e. not a pool w/ near full warnings). > >>>>>> > >>>>>> > >>>>>> Loïc. > >>>>> -- > >>>>> _____________________________________________________________ > >>>>> prof. dr. Andrej Filipcic, E-mail: [email protected] > >>>>> Department of Experimental High Energy Physics - F9 > >>>>> Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > >>>>> SI-1001 Ljubljana, Slovenia > >>>>> Tel.: +386-1-477-3674 Fax: +386-1-477-3166 > >>>>> ------------------------------------------------------------- > >>>>> _______________________________________________ > >>>>> ceph-users mailing list -- [email protected] > >>>>> To unsubscribe send an email to [email protected] > >>>> _______________________________________________ > >>>> ceph-users mailing list -- [email protected] > >>>> To unsubscribe send an email to [email protected] > >>> > >> _______________________________________________ > >> ceph-users mailing list -- [email protected] > >> To unsubscribe send an email to [email protected] > > _______________________________________________ > > ceph-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > > -- > _____________________________________________________________ > prof. dr. Andrej Filipcic, E-mail: [email protected] > Department of Experimental High Energy Physics - F9 > Jozef Stefan Institute, Jamova 39, P.o.Box 3000 > SI-1001 Ljubljana, Slovenia > Tel.: +386-1-477-3674 Fax: +386-1-477-3166 > ------------------------------------------------------------- > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
