On 5/29/19 11:22 PM, J. Eric Ivancich wrote:
> Hi Wido,
>
> When you run `radosgw-admin gc list`, I assume you are *not* using the
> "--include-all" flag, right? If you're not using that flag, then
> everything listed should be expired and be ready for clean-up. If after
> running `radosgw-admin gc process` the same entries appear in
> `radosgw-admin gc list` then gc apparently stalled.
>
Not using the --include-all in both cases.
GC seems to stall and doesn't do anything when looking at it with
--debug-rados=10
> There were a few bugs within gc processing that could prevent it from
> making forward progress. They were resolved with a PR (master:
> https://github.com/ceph/ceph/pull/26601 ; mimic backport:
> https://github.com/ceph/ceph/pull/27796). Unfortunately that code was
> backported after the 13.2.5 release, but it is in place for the 13.2.6
> release of mimic.
>
Thanks! I'll might grab some packages from Shaman to give GC a try.
Wido
> Eric
>
>
> On 5/29/19 3:19 AM, Wido den Hollander wrote:
>> Hi,
>>
>> I've got a Ceph cluster with this status:
>>
>> health: HEALTH_WARN
>> 3 large omap objects
>>
>> After looking into it I see that the issue comes from objects in the
>> '.rgw.gc' pool.
>>
>> Investigating it I found that the gc.* objects have a lot of OMAP keys:
>>
>> for OBJ in $(rados -p .rgw.gc ls); do
>> echo $OBJ
>> rados -p .rgw.gc listomapkeys $OBJ|wc -l
>> done
>>
>> I then found out that on average these objects have about 100k of OMAP
>> keys each, but two stand out and have about 3M OMAP keys.
>>
>> I can list the GC with 'radosgw-admin gc list' and this yields a JSON
>> which is a couple of MB in size.
>>
>> I ran:
>>
>> $ radosgw-admin gc process
>>
>> That runs for hours and then finishes, but the large list of OMAP keys
>> stays.
>>
>> Running Mimic 13.3.5 on this cluster.
>>
>> Has anybody seen this before?
>>
>> Wido
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
