A couple of ideas: Do you have osd_recovery_sleep_* set? osd_delete_sleep_*? Do you have any pools in the middle of pg_num adjustments, such that pg_num != pgp_num and a pg_num_target is reported?
> On Jul 2, 2026, at 2:52 PM, Andras Pataki via ceph-users <[email protected]> > wrote: > > Hi, > > I have also seen this on a recent large scale/parallel test I can on a test > setup. After creating about a hundred million files, then removing all of > them - there were thousands of objects left in the cephfs data pool > corresponding to inodes that no longer existed. I.e. it looks like the MDS is > "leaking" data - does not correctly delete all objects corresponding to files > in the purge queue. We have never used any snapshots on this file system, so > it definitely isn't about snapshots holding data. Also, we increased the > purge queue settings (filer_max_purge_ops, mds_max_purge_* and friends) and > definitely waited long enough for all deletes to be processed (the objects > were there weeks after the file system was emptied). This was on a squid > 19.2.3 cluster. > In the tests that lead to this, both the file creation and deletions were > done in parallel using a few dozen clients - stressing the MDS for sure. We > had no crashes/problems on the cluster, we never had to do any data recovery > steps, i.e. the MDS appeared to work fine through the tests. Also, this was > a single MDS cluster, i.e. the problem isn't related to subtrees moving > across MDS's for example. The problem became obvious since after removing all > data from cephfs, we were expecting the data pools to be empty, but they > weren't. There were two pools, a primary triple replicated one and an > erasure coded one. Both of them had stray objects. > > I didn't pursue this further since I wasn't sure what useful information I > could gather for a bug report - but it is certainly a curious observation > that perhaps large, long living cephfs clusters might have significant space > tied up in these objects that should have been removed but weren't. Short of > a full scan of all objects and matching them to inodes - it is hard to tell > how much even. > > Andras > > >> On 7/2/26 12:17 PM, Andrej Filipčič via ceph-users wrote: >> >> So, to follow up on this, I did some further investigation. >> >> Checking for write amplification, I have copied 250TB of a mix of small and >> large files (20M of them), and the stored space on EC pool matched what was >> expected from actual data size to a few %. So EC overhead was not really a >> factor. Also, after the removal of this data set, the stored space was >> recovered as expected. >> >> The I checked the full dump of cephfs and compared it to list of all objects >> in the EC pool as follows: >> >> rados -p cephfs_data_echdd ls > echdd.objectlist >> >> find /ceph/ -printf "%i %p\n" > cephfs.inodes >> >> this took several hours, while in the meantime, writing and removing of data >> to cephfs was relatively low (few MB/s), so the impact of new objects and >> files should have been minimal. >> >> Then I selected all the objects of the form >> 1007b28abae.00000000 >> 100cce97f6d.00000000 >> 10067733861.00000000 >> 100cc4646aa.00000000 >> 200044d5c07.00000000 >> ... >> >> and checked if they match the files in the cephfs.inodes list >> >> ~3M of *.00000000 objects do not have the corresponding inode in the >> cephfs.inodes >> >> >> I did "rados stat" of ~70k of these objects (still running for all of them), >> and almost all have timestamps from March and April these year. On 4th of >> May I remember I have increased the mds purge queue values to speed up the >> removal of files since OSDs were filling up too quickly, and since then it >> seems the objects are not left uncleaned any more. >> >> With "rados getxattr objectid parent" I have checked several of them, and >> they all belong to two very active projects which typically write files with >> few GB/s all the time (and remove as well so space usage is not increasing). >> The objects I have checked belonged to files that were removed in cephfs, >> the projects have a separate file catalog which is consistent with cephfs >> contents. >> >> So, I do not understand why so many objects were left unremoved in EC pool >> in the period of 2 months, but at least, 3M uncleaned files explains 1.5PB >> of dark data by quick estimate. >> >> We also scrubbed cephfs root and ~mdsdir several times and no leftovers to >> remove were found. >> >> I hope the problem is gone now, but I would still like a good advice on how >> to proceed with the cleanup. I see these options: >> >> 1) remove the unmatched objects directly from EC pool with "rados rm". But >> this might have undesired side effects or corruption. >> >> 2) create a new EC pool and migrate all the data there by copying files in >> filesystem, and then destroy the old pool. >> >> 3) create a new filesystem with new EC pools and migrate the files. >> >> 4) run some advanced MDS disaster recovery procedure (cephfs-data-scan), but >> this requires offline FS and I do not want to recover old files from >> existing unmatched objects. Anyway, cephfs seems to be healthy now. >> >> Any good ideas? >> >> Best, >> Andrej >> >> >> On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote: >>>> ). >>>>>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with 132M >>>>>> files and 139M rentries. The pool shows 7PiB stored and 9.7PiB used >>>>>> consistent with 8+3 EC. >>>>>> The layout for most files: >>>>>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1 >>>>>> object_size=16777216 pool=cephfs_data_echdd" >>>>>> >>>>>> But there is 1.4PiB discrepancy between the pool and the filesystem >>> Do you have scrubs enabled? Which if any non-default config options do you >>> have set? Any undersized or degraded or backfilling PGs? >>> >>> Which Ceph release? Do you have a sizable fraction of small files? If >>> you’re running Squid or earlier or don’t have EC optimizations enabled, >>> even a tiny file will allocate a multiple of 11*16=176 KB. An 129KB file >>> will consume 352KB, etc. If I understand those layout options correctly. >>> >>> If that’s what’s going on, going to Tentacle with EC optimizations would >>> gain you some efficiency for files newly [re]written. You could also >>> migrate small files to a replicated pool. >>> >>> >>>>>> which I cannot explain and I suspect there are a lot of orphan objects >>>>>> there. I have run mds scrub on / and ~mdsdir as well. There is some mds >>>>>> damage on some old small files (~400 files), which I do not think it's >>>>>> relevant here. >>>>>> >>>>> Hello, >>>>> We had a similar issue last year with a group of users that created and >>>>> removed files at a very high rate. >>>>> >>>>> Have you read >>>>> https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/cephfs/*cephfs-pool-data-usage-growth-without-explanation__;Iw!!DSb-azq1wVFtOg!XcB7cJMEzAXsAddGg1LH5ff1B33dit2O1vAhxnVlv2MMUrC85oUKgukdaytYJQqAfmmjiTWJhr7GLPIKSTbwQpbx18c$ >>>>> ? >>>>> >>>>> We increased the purge rate parameters (very) aggressively to get back to >>>>> a comfortable situation (i.e. not a pool w/ near full warnings). >>>>> >>>>> >>>>> Loïc. >>>> >>>> -- >>>> _____________________________________________________________ >>>> prof. dr. Andrej Filipcic, E-mail: [email protected] >>>> Department of Experimental High Energy Physics - F9 >>>> Jozef Stefan Institute, Jamova 39, P.o.Box 3000 >>>> SI-1001 Ljubljana, Slovenia >>>> Tel.: +386-1-477-3674 Fax: +386-1-477-3166 >>>> ------------------------------------------------------------- >>>> _______________________________________________ >>>> ceph-users mailing list -- [email protected] >>>> To unsubscribe send an email to [email protected] >>> _______________________________________________ >>> ceph-users mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >> >> > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
