Re: [ceph-users] Speeding up garbage collection in RGW

2017-10-27 Thread David Turner
I had the exact same error when using --bypass-gc. We too decided to destroy this realm and start it fresh. For us, 95% of the data in this realm is backups for other systems and they're find rebuilding it. So our plan is to migrate the 5% of the data to a temporary s3 location and then rebuild

Re: [ceph-users] Speeding up garbage collection in RGW

2017-10-27 Thread Bryan Stillwell
On Wed, Oct 25, 2017 at 4:02 PM, Yehuda Sadeh-Weinraub wrote: > > On Wed, Oct 25, 2017 at 2:32 PM, Bryan Stillwell > wrote: > > That helps a little bit, but overall the process would take years at this > > rate: > > > > # for i in {1..3600}; do ceph

Re: [ceph-users] Speeding up garbage collection in RGW

2017-10-25 Thread Yehuda Sadeh-Weinraub
From: Yehuda Sadeh-Weinraub <yeh...@redhat.com> > Date: Wednesday, October 25, 2017 at 11:32 AM > To: Bryan Stillwell <bstillw...@godaddy.com> > Cc: David Turner <drakonst...@gmail.com>, Ben Hines <bhi...@gmail.com>, > "ceph-users@lists.ceph.com" <ceph-use

Re: [ceph-users] Speeding up garbage collection in RGW

2017-10-25 Thread Bryan Stillwell
Yehuda Sadeh-Weinraub <yeh...@redhat.com> Date: Wednesday, October 25, 2017 at 11:32 AM To: Bryan Stillwell <bstillw...@godaddy.com> Cc: David Turner <drakonst...@gmail.com>, Ben Hines <bhi...@gmail.com>, "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.co

Re: [ceph-users] Speeding up garbage collection in RGW

2017-10-25 Thread Yehuda Sadeh-Weinraub
that. > > Bryan > > From: ceph-users <ceph-users-boun...@lists.ceph.com> on behalf of David > Turner <drakonst...@gmail.com> > Date: Tuesday, October 24, 2017 at 4:07 PM > To: Ben Hines <bhi...@gmail.com> > Cc: "ceph-users@lists.ceph.com" <ceph-us

Re: [ceph-users] Speeding up garbage collection in RGW

2017-10-25 Thread Bryan Stillwell
com> on behalf of David Turner <drakonst...@gmail.com> Date: Tuesday, October 24, 2017 at 4:07 PM To: Ben Hines <bhi...@gmail.com> Cc: "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Speeding up garbage collection in RGW Thank you so much for

Re: [ceph-users] Speeding up garbage collection in RGW

2017-10-24 Thread David Turner
Jul 25, 2017 at 4:01 PM Bryan Stillwell <bstillw...@godaddy.com> >>> wrote: >>> >>>> Excellent, thank you! It does exist in 0.94.10! :) >>>> >>>> >>>> >>>> Bryan >>>> >>>> &g

Re: [ceph-users] Speeding up garbage collection in RGW

2017-10-23 Thread David Turner
@lists.ceph.com" > <ceph-users@lists.ceph.com> > *Subject: *Re: [ceph-users] Speeding up garbage collection in RGW > > > > I’ve just realized that the option is present in Hammer (0.94.10) as well, > you should try that. > > > > *From: *Bryan Stillwel

Re: [ceph-users] Speeding up garbage collection in RGW

2017-07-25 Thread Bryan Stillwell
>, "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com> Subject: EXT: Re: [ceph-users] Speeding up garbage collection in RGW Unfortunately, we're on hammer still (0.94.10). That option looks like it would work better, so maybe it's time to move the upgrade up in the schedu

Re: [ceph-users] Speeding up garbage collection in RGW

2017-07-25 Thread Pavan Rallabhandi
ceph-users@lists.ceph.com> Subject: EXT: Re: [ceph-users] Speeding up garbage collection in RGW Unfortunately, we're on hammer still (0.94.10). That option looks like it would work better, so maybe it's time to move the upgrade up in the schedule. I've been playing with the various gc options and

Re: [ceph-users] Speeding up garbage collection in RGW

2017-07-25 Thread Bryan Stillwell
. Thanks, Bryan From: Pavan Rallabhandi <prallabha...@walmartlabs.com> Date: Tuesday, July 25, 2017 at 3:00 AM To: Bryan Stillwell <bstillw...@godaddy.com>, "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Speeding up garbage col

Re: [ceph-users] Speeding up garbage collection in RGW

2017-07-25 Thread Pavan Rallabhandi
If your Ceph version is >=Jewel, you can try the `--bypass-gc` option in radosgw-admin, which would remove the tails objects as well without marking them to be GCed. Thanks, On 25/07/17, 1:34 AM, "ceph-users on behalf of Bryan Stillwell"

Re: [ceph-users] Speeding up garbage collection in RGW

2017-07-24 Thread Z Will
om> > Date: Monday, July 24, 2017 at 2:43 PM > To: Bryan Stillwell <bstillw...@godaddy.com>, "ceph-users@lists.ceph.com" > <ceph-users@lists.ceph.com> > Subject: Re: [ceph-users] Speeding up garbage collection in RGW > > I hope someone else can answer your

Re: [ceph-users] Speeding up garbage collection in RGW

2017-07-24 Thread Bryan Stillwell
.ceph.com" <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Speeding up garbage collection in RGW I hope someone else can answer your question better, but in my case I found something like this helpful to delete objects faster than I could through the gateway: rados -p defaul

Re: [ceph-users] Speeding up garbage collection in RGW

2017-07-24 Thread Roger Brown
I hope someone else can answer your question better, but in my case I found something like this helpful to delete objects faster than I could through the gateway: rados -p default.rgw.buckets.data ls | grep 'replace this with pattern matching files you want to delete' | xargs -d '\n' -n 200 rados

[ceph-users] Speeding up garbage collection in RGW

2017-07-24 Thread Bryan Stillwell
I'm in the process of cleaning up a test that an internal customer did on our production cluster that produced over a billion objects spread across 6000 buckets. So far I've been removing the buckets like this: printf %s\\n bucket{1..6000} | xargs -I{} -n 1 -P 32 radosgw-admin bucket rm