On 2. 7. 2026 21:25, Gregory Farnum wrote:
Hmm, this is unfortunate news. I don’t think I’ve heard about anything
like this before.
Andrej, what Ceph version are you running, and has it changed during
this time frame?
we did run 19.2.3 during that time frame and upgraded to 20.2.1 on 11th
of June, so the version does not seem to match the time frames.
I would imagine the most likely thing to be our “purge queue” is
somehow some times not deleting objects that it marks as removed. But
I’ve no idea how that could go wrong. As much information as we can
get about stuff like mds restarts during the deletion, single or
mult-mds, precise versions, and any pool or file layouts may help
narrow things down. (For instance, it may be that the leak only
happens if using multiple data pools, or if it was written on a
particular version and a later buggy version parses something wrong.)
well, we had a bit complicated situation, since quite some OSDs died and
were replaced in the meantime, so recovery was running for a significant
portion of time. Also, we had multiple mdses and switched to 1 when we
started to have a problem with nearfull OSDs. Also we restarted mds many
times since the memory usage went high. We also use multiple data pools,
though the one in question is the most active.
I can dig through the logs and provide more detailed information.
Best,
Andrej
-Greg
On Thu, Jul 2, 2026 at 8:53 AM Andras Pataki via ceph-users
<[email protected]> wrote:
Hi,
I have also seen this on a recent large scale/parallel test I can
on a
test setup. After creating about a hundred million files, then
removing
all of them - there were thousands of objects left in the cephfs data
pool corresponding to inodes that no longer existed. I.e. it looks
like
the MDS is "leaking" data - does not correctly delete all objects
corresponding to files in the purge queue. We have never used any
snapshots on this file system, so it definitely isn't about snapshots
holding data. Also, we increased the purge queue settings
(filer_max_purge_ops, mds_max_purge_* and friends) and definitely
waited
long enough for all deletes to be processed (the objects were there
weeks after the file system was emptied). This was on a squid 19.2.3
cluster.
In the tests that lead to this, both the file creation and deletions
were done in parallel using a few dozen clients - stressing the
MDS for
sure. We had no crashes/problems on the cluster, we never had to
do any
data recovery steps, i.e. the MDS appeared to work fine through the
tests. Also, this was a single MDS cluster, i.e. the problem isn't
related to subtrees moving across MDS's for example. The problem
became
obvious since after removing all data from cephfs, we were
expecting the
data pools to be empty, but they weren't. There were two pools, a
primary triple replicated one and an erasure coded one. Both of them
had stray objects.
I didn't pursue this further since I wasn't sure what useful
information
I could gather for a bug report - but it is certainly a curious
observation that perhaps large, long living cephfs clusters might
have
significant space tied up in these objects that should have been
removed
but weren't. Short of a full scan of all objects and matching
them to
inodes - it is hard to tell how much even.
Andras
On 7/2/26 12:17 PM, Andrej Filipčič via ceph-users wrote:
>
> So, to follow up on this, I did some further investigation.
>
> Checking for write amplification, I have copied 250TB of a mix of
> small and large files (20M of them), and the stored space on EC
pool
> matched what was expected from actual data size to a few %. So EC
> overhead was not really a factor. Also, after the removal of
this data
> set, the stored space was recovered as expected.
>
> The I checked the full dump of cephfs and compared it to list of
all
> objects in the EC pool as follows:
>
> rados -p cephfs_data_echdd ls > echdd.objectlist
>
> find /ceph/ -printf "%i %p\n" > cephfs.inodes
>
> this took several hours, while in the meantime, writing and
removing
> of data to cephfs was relatively low (few MB/s), so the impact
of new
> objects and files should have been minimal.
>
> Then I selected all the objects of the form
> 1007b28abae.00000000
> 100cce97f6d.00000000
> 10067733861.00000000
> 100cc4646aa.00000000
> 200044d5c07.00000000
> ...
>
> and checked if they match the files in the cephfs.inodes list
>
> ~3M of *.00000000 objects do not have the corresponding inode
in the
> cephfs.inodes
>
>
> I did "rados stat" of ~70k of these objects (still running for
all of
> them), and almost all have timestamps from March and April these
year.
> On 4th of May I remember I have increased the mds purge queue
values
> to speed up the removal of files since OSDs were filling up too
> quickly, and since then it seems the objects are not left uncleaned
> any more.
>
> With "rados getxattr objectid parent" I have checked several of
them,
> and they all belong to two very active projects which typically
write
> files with few GB/s all the time (and remove as well so space
usage is
> not increasing). The objects I have checked belonged to files that
> were removed in cephfs, the projects have a separate file catalog
> which is consistent with cephfs contents.
>
> So, I do not understand why so many objects were left unremoved
in EC
> pool in the period of 2 months, but at least, 3M uncleaned files
> explains 1.5PB of dark data by quick estimate.
>
> We also scrubbed cephfs root and ~mdsdir several times and no
> leftovers to remove were found.
>
> I hope the problem is gone now, but I would still like a good
advice
> on how to proceed with the cleanup. I see these options:
>
> 1) remove the unmatched objects directly from EC pool with "rados
> rm". But this might have undesired side effects or corruption.
>
> 2) create a new EC pool and migrate all the data there by copying
> files in filesystem, and then destroy the old pool.
>
> 3) create a new filesystem with new EC pools and migrate the files.
>
> 4) run some advanced MDS disaster recovery procedure
> (cephfs-data-scan), but this requires offline FS and I do not
want to
> recover old files from existing unmatched objects. Anyway, cephfs
> seems to be healthy now.
>
> Any good ideas?
>
> Best,
> Andrej
>
>
> On 10. 6. 2026 01:36, Anthony D'Atri via ceph-users wrote:
>>> ).
>>>>> There is ~5.6PiB stored on /ceph, shown by ceph.dir.rbytes with
>>>>> 132M files and 139M rentries. The pool shows 7PiB stored and
>>>>> 9.7PiB used consistent with 8+3 EC.
>>>>> The layout for most files:
>>>>> ceph.dir.layout="stripe_unit=16777216 stripe_count=1
>>>>> object_size=16777216 pool=cephfs_data_echdd"
>>>>>
>>>>> But there is 1.4PiB discrepancy between the pool and the
filesystem
>> Do you have scrubs enabled? Which if any non-default config
options
>> do you have set? Any undersized or degraded or backfilling PGs?
>>
>> Which Ceph release? Do you have a sizable fraction of small files?
>> If you’re running Squid or earlier or don’t have EC optimizations
>> enabled, even a tiny file will allocate a multiple of 11*16=176
KB.
>> An 129KB file will consume 352KB, etc. If I understand those
layout
>> options correctly.
>>
>> If that’s what’s going on, going to Tentacle with EC optimizations
>> would gain you some efficiency for files newly [re]written. You
>> could also migrate small files to a replicated pool.
>>
>>
>>>>> which I cannot explain and I suspect there are a lot of orphan
>>>>> objects there. I have run mds scrub on / and ~mdsdir as well.
>>>>> There is some mds damage on some old small files (~400 files),
>>>>> which I do not think it's relevant here.
>>>>>
>>>> Hello,
>>>> We had a similar issue last year with a group of users that
created
>>>> and removed files at a very high rate.
>>>>
>>>> Have you read
>>>>
https://urldefense.com/v3/__https://docs.clyso.com/docs/kb/cephfs/*cephfs-pool-data-usage-growth-without-explanation__;Iw!!DSb-azq1wVFtOg!XcB7cJMEzAXsAddGg1LH5ff1B33dit2O1vAhxnVlv2MMUrC85oUKgukdaytYJQqAfmmjiTWJhr7GLPIKSTbwQpbx18c$
>>>> ?
>>>>
>>>> We increased the purge rate parameters (very) aggressively to
get
>>>> back to a comfortable situation (i.e. not a pool w/ near full
>>>> warnings).
>>>>
>>>>
>>>> Loïc.
>>>
>>> --
>>> _____________________________________________________________
>>> prof. dr. Andrej Filipcic, E-mail: [email protected]
>>> Department of Experimental High Energy Physics - F9
>>> Jozef Stefan Institute, Jamova 39, P.o.Box 3000
>>> SI-1001 Ljubljana, Slovenia
>>> Tel.: +386-1-477-3674 Fax: +386-1-477-3166
>>> -------------------------------------------------------------
>>> _______________________________________________
>>> ceph-users mailing list -- [email protected]
>>> To unsubscribe send an email to [email protected]
>> _______________________________________________
>> ceph-users mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>
>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
--
_____________________________________________________________
prof. dr. Andrej Filipcic, E-mail:[email protected]
Department of Experimental High Energy Physics - F9
Jozef Stefan Institute, Jamova 39, P.o.Box 3000
SI-1001 Ljubljana, Slovenia
Tel.: +386-1-477-3674 Fax: +386-1-477-3166
-------------------------------------------------------------
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
