Re: [ceph-users] Fixing bad radosgw index

2018-04-23 Thread Matt Benjamin 
Mimic (and higher) contain a new async gc mechanism, which should
handle this workload internally.

Matt

On Mon, Apr 23, 2018 at 2:55 PM, David Turner  wrote:
> When figuring out why space is not freeing up after deleting buckets and
> objects in RGW, look towards the RGW Garbage Collection.  This has come up
> on the ML several times in the past.  I am almost finished catching up on a
> GC of 200 Million objects that was taking up a substantial amount of space
> in my cluster.  I did this by running about 30 screens with the command
> `while true; do radosgw-admin gc process; sleep 10; done` in each of them.
> It appears that there are 32 available sockets for the gc to be processed
> and this helped us catch up on 200M objects in under 2 months.
>
> On Mon, Apr 16, 2018 at 12:01 PM Robert Stanford 
> wrote:
>>
>>
>>  This doesn't work for me:
>>
>> for i in `radosgw-admin bucket list`; do radosgw-admin bucket unlink
>> --bucket=$i --uid=myuser; done   (tried with and without '=')
>>
>>  Errors for each bucket:
>>
>> failure: (2) No such file or directory2018-04-16 15:37:54.022423
>> 7f7c250fbc80  0 could not get bucket info for bucket="bucket5",
>>
>> On Mon, Apr 16, 2018 at 8:30 AM, Casey Bodley  wrote:
>>>
>>>
>>>
>>> On 04/14/2018 12:54 PM, Robert Stanford wrote:
>>>
>>>
>>>  I deleted my default.rgw.buckets.data and default.rgw.buckets.index
>>> pools in an attempt to clean them out.  I brought this up on the list and
>>> received replies telling me essentially, "You shouldn't do that."  There was
>>> however no helpful advice on recovering.
>>>
>>>  When I run 'radosgw-admin bucket list' I get a list of all my old
>>> buckets (I thought they'd be cleaned out when I deleted and recreated
>>> default.rgw.buckets.index, but I was wrong.)  Deleting them with s3cmd and
>>> radosgw-admin does nothing; they still appear (though s3cmd will give a
>>> '404' error.)  Running radosgw-admin with 'bucket check' and '--fix' does
>>> nothing as well.  So, how do I get myself out of this mess.
>>>
>>>  On another, semi-related note, I've been deleting (existing) buckets and
>>> their contents with s3cmd (and --recursive); the space is never freed from
>>> ceph and the bucket still appears in s3cmd ls.  Looks like my radosgw has
>>> several issues, maybe all related to deleting and recreating the pools.
>>>
>>>  Thanks
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>> The 'bucket list' command takes a user and prints the list of buckets
>>> they own - this list is read from the user object itself. You can remove
>>> these entries with the 'bucket unlink' command.
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fixing bad radosgw index

2018-04-23 Thread David Turner 
When figuring out why space is not freeing up after deleting buckets and
objects in RGW, look towards the RGW Garbage Collection.  This has come up
on the ML several times in the past.  I am almost finished catching up on a
GC of 200 Million objects that was taking up a substantial amount of space
in my cluster.  I did this by running about 30 screens with the command
`while true; do radosgw-admin gc process; sleep 10; done` in each of them.
It appears that there are 32 available sockets for the gc to be processed
and this helped us catch up on 200M objects in under 2 months.

On Mon, Apr 16, 2018 at 12:01 PM Robert Stanford 
wrote:

>
>  This doesn't work for me:
>
> for i in `radosgw-admin bucket list`; do radosgw-admin bucket unlink
> --bucket=$i --uid=myuser; done   (tried with and without '=')
>
>  Errors for each bucket:
>
> failure: (2) No such file or directory2018-04-16 15:37:54.022423
> 7f7c250fbc80  0 could not get bucket info for bucket="bucket5",
>
> On Mon, Apr 16, 2018 at 8:30 AM, Casey Bodley  wrote:
>
>>
>>
>> On 04/14/2018 12:54 PM, Robert Stanford wrote:
>>
>>
>>  I deleted my default.rgw.buckets.data and default.rgw.buckets.index
>> pools in an attempt to clean them out.  I brought this up on the list and
>> received replies telling me essentially, "You shouldn't do that."  There
>> was however no helpful advice on recovering.
>>
>>  When I run 'radosgw-admin bucket list' I get a list of all my old
>> buckets (I thought they'd be cleaned out when I deleted and recreated
>> default.rgw.buckets.index, but I was wrong.)  Deleting them with s3cmd and
>> radosgw-admin does nothing; they still appear (though s3cmd will give a
>> '404' error.)  Running radosgw-admin with 'bucket check' and '--fix' does
>> nothing as well.  So, how do I get myself out of this mess.
>>
>>  On another, semi-related note, I've been deleting (existing) buckets and
>> their contents with s3cmd (and --recursive); the space is never freed from
>> ceph and the bucket still appears in s3cmd ls.  Looks like my radosgw has
>> several issues, maybe all related to deleting and recreating the pools.
>>
>>  Thanks
>>
>>
>> ___
>> ceph-users mailing 
>> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>> The 'bucket list' command takes a user and prints the list of buckets
>> they own - this list is read from the user object itself. You can remove
>> these entries with the 'bucket unlink' command.
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fixing bad radosgw index

2018-04-16 Thread Robert Stanford 
 This doesn't work for me:

for i in `radosgw-admin bucket list`; do radosgw-admin bucket unlink
--bucket=$i --uid=myuser; done   (tried with and without '=')

 Errors for each bucket:

failure: (2) No such file or directory2018-04-16 15:37:54.022423
7f7c250fbc80  0 could not get bucket info for bucket="bucket5",

On Mon, Apr 16, 2018 at 8:30 AM, Casey Bodley  wrote:

>
>
> On 04/14/2018 12:54 PM, Robert Stanford wrote:
>
>
>  I deleted my default.rgw.buckets.data and default.rgw.buckets.index pools
> in an attempt to clean them out.  I brought this up on the list and
> received replies telling me essentially, "You shouldn't do that."  There
> was however no helpful advice on recovering.
>
>  When I run 'radosgw-admin bucket list' I get a list of all my old buckets
> (I thought they'd be cleaned out when I deleted and recreated
> default.rgw.buckets.index, but I was wrong.)  Deleting them with s3cmd and
> radosgw-admin does nothing; they still appear (though s3cmd will give a
> '404' error.)  Running radosgw-admin with 'bucket check' and '--fix' does
> nothing as well.  So, how do I get myself out of this mess.
>
>  On another, semi-related note, I've been deleting (existing) buckets and
> their contents with s3cmd (and --recursive); the space is never freed from
> ceph and the bucket still appears in s3cmd ls.  Looks like my radosgw has
> several issues, maybe all related to deleting and recreating the pools.
>
>  Thanks
>
>
> ___
> ceph-users mailing 
> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> The 'bucket list' command takes a user and prints the list of buckets they
> own - this list is read from the user object itself. You can remove these
> entries with the 'bucket unlink' command.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fixing bad radosgw index

2018-04-16 Thread Casey Bodley 



On 04/14/2018 12:54 PM, Robert Stanford wrote:


 I deleted my default.rgw.buckets.data and default.rgw.buckets.index 
pools in an attempt to clean them out.  I brought this up on the list 
and received replies telling me essentially, "You shouldn't do that." 
There was however no helpful advice on recovering.


 When I run 'radosgw-admin bucket list' I get a list of all my old 
buckets (I thought they'd be cleaned out when I deleted and recreated 
default.rgw.buckets.index, but I was wrong.)  Deleting them with s3cmd 
and radosgw-admin does nothing; they still appear (though s3cmd will 
give a '404' error.)  Running radosgw-admin with 'bucket check' and 
'--fix' does nothing as well.  So, how do I get myself out of this mess.


 On another, semi-related note, I've been deleting (existing) buckets 
and their contents with s3cmd (and --recursive); the space is never 
freed from ceph and the bucket still appears in s3cmd ls.  Looks like 
my radosgw has several issues, maybe all related to deleting and 
recreating the pools.


 Thanks


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


The 'bucket list' command takes a user and prints the list of buckets 
they own - this list is read from the user object itself. You can remove 
these entries with the 'bucket unlink' command.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Fixing bad radosgw index

2018-04-14 Thread Robert Stanford 
 I deleted my default.rgw.buckets.data and default.rgw.buckets.index pools
in an attempt to clean them out.  I brought this up on the list and
received replies telling me essentially, "You shouldn't do that."  There
was however no helpful advice on recovering.

 When I run 'radosgw-admin bucket list' I get a list of all my old buckets
(I thought they'd be cleaned out when I deleted and recreated
default.rgw.buckets.index, but I was wrong.)  Deleting them with s3cmd and
radosgw-admin does nothing; they still appear (though s3cmd will give a
'404' error.)  Running radosgw-admin with 'bucket check' and '--fix' does
nothing as well.  So, how do I get myself out of this mess.

 On another, semi-related note, I've been deleting (existing) buckets and
their contents with s3cmd (and --recursive); the space is never freed from
ceph and the bucket still appears in s3cmd ls.  Looks like my radosgw has
several issues, maybe all related to deleting and recreating the pools.

 Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com