Hi Greg, This one? https://tracker.ceph.com/issues/64646
Symptoms: - CLONES are reported on 'rados df' while pool has no snapshots. - 'rados lssnap -p <pool_name>' command shows no snapshots but some clones are listed by 'rados listsnaps -p <pool_name> <object_name>' even sometimes with no 'head' object. @Trey, if this is the one --- make sure it is before running the command --- running a 'ceph osd pool force-remove-snap <pool_name>' should put all leaked clone objects back in the trim queue and the OSDs should get rid of them. Regards, Frédéric. Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> frederic.n...@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Le ven. 3 oct. 2025 à 05:19, Gregory Farnum <gfar...@redhat.com> a écrit : > There was a RADOS bug with trimming deleted snapshots, but I don’t have the > tracker ticket for more details — it involved some subtleties with one of > the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, > I suspect it was that. > -Greg > > On Thu, Oct 2, 2025 at 5:08 PM Anthony D'Atri <anthony.da...@gmail.com> > wrote: > > > FWIW, my understanding is that this RGW issue was fixed several releases > > ago. The OP’s cluster IIRC is mostly CephFS, so I suspect something else > > is going on. > > > > > > > On Oct 2, 2025, at 7:29 PM, Manuel Rios - EDH < > mrios...@easydatahost.com> > > wrote: > > > > > > Hi, > > > > > > Here user that suffer years ago a problem with orphans. > > > > > > Years ago, after much research, we discovered that for some reason the > > WALLDB/Metadata entries were being deleted and corrupted, but the data on > > the disks weren't physically erased. > > > Sometimes the garbage collector (deferred delete) would fail and skip > > the deletion, leaving hundreds of TB behind. > > > Speaking with other heavy CEPH users, they were aware of this and > > couldn't find a great solution either, just they instead use replica 3 , > > used replica 4. (big customers with big budget) > > > At the time, we were presented with two options: wipe each disk, and > > CEPH would only rebuild the data it knows is valid, but that would take > > time, maybe in your case where your full NVME will take not too much. Or > > create a new cluster and move the valid data. > > > > > > In our case ceph orphan tool start looping due bugs and didn’t provide > a > > real solution, our case 1PB ceph, with aprox 300TB orphaned. > > > > > > I remember ceph orphan tool running for weeks ☹ bad ass time. > > > > > > Our ceph use case : S3 and version 12 to 14 nautilus... > > > > > > Sometimes , we as administrators doesn’t care about this issues until > > you need to wipe a lot of data. And you use simple calc and don’t match. > > > > > > Regards, > > > > > > -----Mensaje original----- > > > De: Alexander Patrakov <patra...@gmail.com> > > > Enviado el: jueves, 2 de octubre de 2025 22:56 > > > Para: Anthony D'Atri <anthony.da...@gmail.com> > > > CC: ceph-users@ceph.io > > > Asunto: [ceph-users] Re: Orphaned CephFS objects > > > > > > On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri <anthony.da...@gmail.com > > > > wrote: > > > > > >> There is design work for a future ability to migrate a pool > > transparently, for example to effect a new EC profile, but that won't be > > available anytime soon. > > > > > > This is, unfortunately, irrelevant in this case. Migrating a pool will > > > migrate all the objects and their snapshots, even the unwanted ones. > > > What Trey has (as far as I understood) is that there are some > > > RADOS-level snapshots that do not correspond to any CephFS-level > > > snapshots and are thus garbage, not to be migrated. > > > > > > That's why the talk about file migration and not pool-level operations. > > > > > > Now to the original question: > > > > > >> will I be able to do 'ceph fs rm_data_pool' once there are no longer > > any > > >> objects associated with the CephFS instance on the pool, or will the > MDS > > >> have ghost object records that cause the command to balk? > > > > > > Just tested in a test cluster - it won't balk and won't demand force > > > even if you remove a pool that is actually used by files. So beware. > > > > > > $ ceph osd pool create badfs_evilpool 32 ssd-only > > > pool 'badfs_evilpool' created > > > $ ceph fs add_data_pool badfs badfs_evilpool > > > added data pool 38 to fsmap > > > $ ceph fs ls > > > name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data > > > cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] > > > name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data > > > badfs_evilpool ] > > > $ cephfs-shell -f badfs > > > CephFS:~/>>> ls > > > dir1/ dir2/ > > > CephFS:~/>>> mkdir evil > > > CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool > > > ceph.dir.layout.pool is successfully set to badfs_evilpool > > > CephFS:~/>>> put /usr/bin/ls /evil/ls > > > $ ceph fs rm_data_pool badfs badfs_evilpool > > > removed data pool 38 from fsmap > > > > > > -- > > > Alexander Patrakov > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
