Hello, I run a cluster in jewel 10.2.2, I have deleted the last Bucket of a radosGW pool to delete this pool and recreate it in EC (was replicate)
Detail of the pool : > pool 36 'erasure.rgw.buckets.data' replicated size 3 min_size 2 crush_ruleset > 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 31459 flags > hashpspool stripe_width 0 > POOLS: > NAME ID USED %USED MAX AVAIL > OBJECTS > erasure.rgw.buckets.data 36 11838M 0 75013G > 4735 After the GC, I found lots of orphan objects still remain in the pool : > $ rados ls -p erasure.rgw.buckets.data | egrep -c "(multipart|shadow)" > 4735 > $ rados ls -p erasure.rgw.buckets.data | grep -c multipart > 2368 > $ rados ls -p erasure.rgw.buckets.data | grep -c shadow > 2367 example : > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_CC-MAIN-2016-40/segments/1474738660158.61/warc/CC-MAIN-20160924173740-00147-ip-10-143-35-109.ec2.internal.warc.gz.2~WezpbEQW1C9nskvtnyAteCVoO3D255Q.29 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_CC-MAIN-2016-40/segments/1474738660158.61/warc/CC-MAIN-20160924173740-00147-ip-10-143-35-109.ec2.internal.warc.gz.2~WezpbEQW1C9nskvtnyAteCVoO3D255Q.61 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_segments/1466783398869.97/wet/CC-MAIN-20160624154958-00194-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~7ru9WPCLMf9Lpi__TP1NXuYwjSU7KQK.11_1 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_crawl-data/CC-MAIN-2016-26/segments/1466783396147.66/wet/CC-MAIN-20160624154956-00071-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~7bKg6WEmNo23IQ6rd8oWF_vbaG0QAFR.6_1 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_segments/1466783398516.82/wet/CC-MAIN-20160624154958-00172-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~ap5QynCJTco_L7yK6bn4M_bnHBbBe64.14_1 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_CC-MAIN-2016-40/segments/1474738662400.75/warc/CC-MAIN-20160924173742-00076-ip-10-143-35-109.ec2.internal.warc.gz.2~LEM4bpbbdiTu86rs3Ew_LFNN_oHg_m7.13 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_CC-MAIN-2016-40/segments/1474738662400.75/warc/CC-MAIN-20160924173742-00033-ip-10-143-35-109.ec2.internal.warc.gz.2~FrN02NmencyDwXavvuzwqR8M8WnWNbH.8_1 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_segments/1466783395560.14/wet/CC-MAIN-20160624154955-00118-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~GqyEUdSepIxGwPOXfKLSxtS8miWGASe.3 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_segments/1466783395346.6/wet/CC-MAIN-20160624154955-00083-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~cTQ86ZEmOvxYD4BUI7zW37X-JcJeMgW.19 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_CC-MAIN-2016-40/segments/1474738660158.61/warc/CC-MAIN-20160924173740-00147-ip-10-143-35-109.ec2.internal.warc.gz.2~WezpbEQW1C9nskvtnyAteCVoO3D255Q.62 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_CC-MAIN-2016-40/segments/1474738662400.75/warc/CC-MAIN-20160924173742-00259-ip-10-143-35-109.ec2.internal.warc.gz.2~1b-olF9koids0gqT9DsO0y1vAsTOasf.12_1 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_CC-MAIN-2016-40/segments/1474738660338.16/warc/CC-MAIN-20160924173740-00067-ip-10-143-35-109.ec2.internal.warc.gz.2~JxuX8v0DmsSgAr3iprPBoHx6PoTKRi6.19_1 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_segments/1466783397864.87/wet/CC-MAIN-20160624154957-00110-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~q2_hY5oSoBWaSZgxh0NdK8JvxmEySPB.29 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__shadow_segments/1466783396949.33/wet/CC-MAIN-20160624154956-00000-ip-10-164-35-72.ec2.internal.warc.wet.gz.2~kUInFVpsWy23JFm9eWNPiFNKlXrjDQU.18_1 > c9724aff-5fa0-4dd9-b494-57bdb48fab4e.1371134.1__multipart_CC-MAIN-2016-40/segments/1474738662400.75/warc/CC-MAIN-20160924173742-00076-ip-10-143-35-109.ec2.internal.warc.gz.2~LEM4bpbbdiTu86rs3Ew_LFNN_oHg_m7.36 firstly, Can I delete the pool even if there is orphan object in ? Should I delete other metadata (index, data_extra) pools related to this pool defined in the zone ? is there other data I should clean to be sure to no have side effect by removing those objects by deleting the pool instead of deleting them with radosgw-admin orphan ? for now, I have followed this doc to find and delete them : https://access.redhat.com/documentation/en/red-hat-ceph-storage/1.3/single/object-gateway-guide-for-ubuntu/#finding_orphan_objects I have ran this command : > radosgw-admin --cluster cephprod orphans find --pool=erasure.rgw.buckets.data > --job-id=erasure but it stuck on a loop, is this a normal behavior ? example of the output I have for at least 2h : > storing 1 entries at orphan.scan.erasure.linked.2 > storing 1 entries at orphan.scan.erasure.linked.5 > storing 1 entries at orphan.scan.erasure.linked.9 > storing 1 entries at orphan.scan.erasure.linked.19 > storing 1 entries at orphan.scan.erasure.linked.25 > storing 1 entries at orphan.scan.erasure.linked.40 > storing 1 entries at orphan.scan.erasure.linked.43 > storing 1 entries at orphan.scan.erasure.linked.47 > storing 1 entries at orphan.scan.erasure.linked.56 > storing 1 entries at orphan.scan.erasure.linked.63 > storing 1 entries at orphan.scan.erasure.linked.9 > storing 1 entries at orphan.scan.erasure.linked.25 > storing 1 entries at orphan.scan.erasure.linked.40 > storing 1 entries at orphan.scan.erasure.linked.56 > storing 1 entries at orphan.scan.erasure.linked.2 > storing 1 entries at orphan.scan.erasure.linked.5 > storing 1 entries at orphan.scan.erasure.linked.9 > storing 1 entries at orphan.scan.erasure.linked.19 > storing 1 entries at orphan.scan.erasure.linked.25 > storing 1 entries at orphan.scan.erasure.linked.40 > storing 1 entries at orphan.scan.erasure.linked.43 > storing 1 entries at orphan.scan.erasure.linked.47 > storing 1 entries at orphan.scan.erasure.linked.56 > storing 1 entries at orphan.scan.erasure.linked.63 > storing 1 entries at orphan.scan.erasure.linked.2 > storing 1 entries at orphan.scan.erasure.linked.5 > storing 1 entries at orphan.scan.erasure.linked.9 > storing 1 entries at orphan.scan.erasure.linked.19 > storing 1 entries at orphan.s can.erasure.linked.25 > storing 1 entries at orphan.scan.erasure.linked.40 > storing 1 entries at orphan.scan.erasure.linked.43 > storing 1 entries at orphan.scan.erasure.linked.47 > storing 1 entries at orphan.scan.erasure.linked.56 > storing 1 entries at orphan.scan.erasure.linked.63 > storing 1 entries at orphan.scan.erasure.linked.9 > storing 1 entries at orphan.scan.erasure.linked.25 > storing 1 entries at orphan.scan.erasure.linked.40 > storing 1 entries at orphan.scan.erasure.linked.56 > storing 1 entries at orphan.scan.erasure.linked.2 > storing 1 entries at orphan.scan.erasure.linked.5 > storing 1 entries at orphan.scan.erasure.linked.9 > storing 1 entries at orphan.scan.erasure.linked.19 > storing 1 entries at orphan.scan.erasure.linked.25 > storing 1 entries at orphan.scan.erasure.linked.40 > storing 1 entries at orphan.scan.erasure.linked.43 > storing 1 entries at orphan.scan.erasure.linked.47 > storing 1 entries at orphan.scan.erasure.linked.56 > storing 1 entries at orphan.scan.erasure.linked.63 > storing 1 entries at orphan.scan.erasure.linked.2 > storing 1 entries at orphan.scan.erasure.linked.5 > storing 1 entries at orphan.scan.erasure.linked.9 > storing 1 entries at orphan.scan.erasure.linked.19 > storing 1 entries at orphan.scan.erasure.linked.25 > storing 1 entries at orphan.scan.erasure.linked.40 > storing 1 entries at orphan.scan.erasure.linked.43 > storing 1 entries at orphan.scan.erasure.linked.47 > storing 1 entries at orphan.scan.erasure.linked.56 > storing 1 entries at orphan.scan.erasure.linked.63 > storing 1 entries at orphan.scan.erasure.linked.9 > storing 1 entries at orphan.scan.erasure.linked.25 > storing 1 entries at orphan.scan.erasure.linked.40 > storing 1 entries at orphan.scan.erasure.linked.56 Thanks for your help -- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
