Re: [ceph-users] Large OMAP Object
On Wed, Nov 20, 2019 at 5:16 PM wrote: > > All; > > Since I haven't heard otherwise, I have to assume that the only way to get > this to go away is to dump the contents of the RGW bucket(s), and recreate > it (them)? Things to try: * check the bucket sharding status: radosgw-admin bucket limit check * reshard the bucket if you aren't running multi-site and the shards are just too big (did you disable resharding?) * resharding is working/index is actually smaller: check if the index shard is actually in use, compare the id to the actual current id (see bucket stats); it could be a leftover/leaked object (that sometimes happened during resharding in older versions) Paul > > How did this get past release approval? A change which makes a valid cluster > state in-valid, with no mitigation other than downtime, in a minor release. > > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > dhils...@performair.com > Sent: Friday, November 15, 2019 9:13 AM > To: ceph-users@lists.ceph.com > Cc: Stephen Self > Subject: Re: [ceph-users] Large OMAP Object > > Wido; > > Ok, yes, I have tracked it down to the index for one of our buckets. I > missed the ID in the ceph df output previously. Next time I'll wait to read > replies until I've finished my morning coffee. > > How would I go about correcting this? > > The content for this bucket is basically just junk, as we're still doing > production qualification, and workflow planning. Moving from Windows file > shares to self-hosted cloud storage is a significant undertaking. > > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > > > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido > den Hollander > Sent: Friday, November 15, 2019 8:40 AM > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Large OMAP Object > > > > On 11/15/19 4:35 PM, dhils...@performair.com wrote: > > All; > > > > Thank you for your help so far. I have found the log entries from when the > > object was found, but don't see a reference to the pool. > > > > Here the logs: > > 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub > > starts > > 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap > > object found. Object: > > 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key > > count: 380425 Size (bytes): 82896978 > > > > In this case it's in pool 56, check 'ceph df' to see which pool that is. > > To me this seems like a RGW bucket which index grew too big. > > Use: > > $ radosgw-admin bucket list > $ radosgw-admin metadata get bucket: > > And match that UUID back to the bucket. > > Wido > > > Thank you, > > > > Dominic L. Hilsbos, MBA > > Director – Information Technology > > Perform Air International Inc. > > dhils...@performair.com > > www.PerformAir.com > > > > > > > > -Original Message- > > From: Wido den Hollander [mailto:w...@42on.com] > > Sent: Friday, November 15, 2019 1:56 AM > > To: Dominic Hilsbos; ceph-users@lists.ceph.com > > Cc: Stephen Self > > Subject: Re: [ceph-users] Large OMAP Object > > > > Did you check /var/log/ceph/ceph.log on one of the Monitors to see which > > pool and Object the large Object is in? > > > > Wido > > > > On 11/15/19 12:23 AM, dhils...@performair.com wrote: > >> All; > >> > >> We had a warning about a large OMAP object pop up in one of our clusters > >> overnight. The cluster is configured for CephFS, but nothing mounts a > >> CephFS, at this time. > >> > >> The cluster mostly uses RGW. I've checked the cluster log, the MON log, > >> and the MGR log on one of the mons, with no useful references to the pool > >> / pg where the large OMAP objects resides. > >> > >> Is my only option to find this large OMAP object to go through the OSD > >> logs for the individual OSDs in the cluster? > >> > >> Thank you, > >> > >> Dominic L. Hilsbos, MBA > >> Director - Information Technology > >> Perform Air International Inc. > >> dhils...@performair.com > >> www.PerformAir.com > &g
Re: [ceph-users] Large OMAP Object
It's a warning, not an error, and if you consider it to not be a problem, I believe you can change osd_deep_scrub_large_omap_object_value_sum_threshold back to 2M. On Wed, Nov 20, 2019 at 11:37 AM wrote: > > All; > > Since I haven't heard otherwise, I have to assume that the only way to get > this to go away is to dump the contents of the RGW bucket(s), and recreate > it (them)? > > How did this get past release approval? A change which makes a valid cluster > state in-valid, with no mitigation other than downtime, in a minor release. > > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > dhils...@performair.com > Sent: Friday, November 15, 2019 9:13 AM > To: ceph-users@lists.ceph.com > Cc: Stephen Self > Subject: Re: [ceph-users] Large OMAP Object > > Wido; > > Ok, yes, I have tracked it down to the index for one of our buckets. I > missed the ID in the ceph df output previously. Next time I'll wait to read > replies until I've finished my morning coffee. > > How would I go about correcting this? > > The content for this bucket is basically just junk, as we're still doing > production qualification, and workflow planning. Moving from Windows file > shares to self-hosted cloud storage is a significant undertaking. > > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > > > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido > den Hollander > Sent: Friday, November 15, 2019 8:40 AM > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Large OMAP Object > > > > On 11/15/19 4:35 PM, dhils...@performair.com wrote: > > All; > > > > Thank you for your help so far. I have found the log entries from when the > > object was found, but don't see a reference to the pool. > > > > Here the logs: > > 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub > > starts > > 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap > > object found. Object: > > 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key > > count: 380425 Size (bytes): 82896978 > > > > In this case it's in pool 56, check 'ceph df' to see which pool that is. > > To me this seems like a RGW bucket which index grew too big. > > Use: > > $ radosgw-admin bucket list > $ radosgw-admin metadata get bucket: > > And match that UUID back to the bucket. > > Wido > > > Thank you, > > > > Dominic L. Hilsbos, MBA > > Director – Information Technology > > Perform Air International Inc. > > dhils...@performair.com > > www.PerformAir.com > > > > > > > > -Original Message- > > From: Wido den Hollander [mailto:w...@42on.com] > > Sent: Friday, November 15, 2019 1:56 AM > > To: Dominic Hilsbos; ceph-users@lists.ceph.com > > Cc: Stephen Self > > Subject: Re: [ceph-users] Large OMAP Object > > > > Did you check /var/log/ceph/ceph.log on one of the Monitors to see which > > pool and Object the large Object is in? > > > > Wido > > > > On 11/15/19 12:23 AM, dhils...@performair.com wrote: > >> All; > >> > >> We had a warning about a large OMAP object pop up in one of our clusters > >> overnight. The cluster is configured for CephFS, but nothing mounts a > >> CephFS, at this time. > >> > >> The cluster mostly uses RGW. I've checked the cluster log, the MON log, > >> and the MGR log on one of the mons, with no useful references to the pool > >> / pg where the large OMAP objects resides. > >> > >> Is my only option to find this large OMAP object to go through the OSD > >> logs for the individual OSDs in the cluster? > >> > >> Thank you, > >> > >> Dominic L. Hilsbos, MBA > >> Director - Information Technology > >> Perform Air International Inc. > >> dhils...@performair.com > >> www.PerformAir.com > >> ___ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > ___ > > ceph-users mailing list > &
Re: [ceph-users] Large OMAP Object
All; Since I haven't heard otherwise, I have to assume that the only way to get this to go away is to dump the contents of the RGW bucket(s), and recreate it (them)? How did this get past release approval? A change which makes a valid cluster state in-valid, with no mitigation other than downtime, in a minor release. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of dhils...@performair.com Sent: Friday, November 15, 2019 9:13 AM To: ceph-users@lists.ceph.com Cc: Stephen Self Subject: Re: [ceph-users] Large OMAP Object Wido; Ok, yes, I have tracked it down to the index for one of our buckets. I missed the ID in the ceph df output previously. Next time I'll wait to read replies until I've finished my morning coffee. How would I go about correcting this? The content for this bucket is basically just junk, as we're still doing production qualification, and workflow planning. Moving from Windows file shares to self-hosted cloud storage is a significant undertaking. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido den Hollander Sent: Friday, November 15, 2019 8:40 AM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Large OMAP Object On 11/15/19 4:35 PM, dhils...@performair.com wrote: > All; > > Thank you for your help so far. I have found the log entries from when the > object was found, but don't see a reference to the pool. > > Here the logs: > 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub > starts > 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap object > found. Object: > 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key > count: 380425 Size (bytes): 82896978 > In this case it's in pool 56, check 'ceph df' to see which pool that is. To me this seems like a RGW bucket which index grew too big. Use: $ radosgw-admin bucket list $ radosgw-admin metadata get bucket: And match that UUID back to the bucket. Wido > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > > > -Original Message- > From: Wido den Hollander [mailto:w...@42on.com] > Sent: Friday, November 15, 2019 1:56 AM > To: Dominic Hilsbos; ceph-users@lists.ceph.com > Cc: Stephen Self > Subject: Re: [ceph-users] Large OMAP Object > > Did you check /var/log/ceph/ceph.log on one of the Monitors to see which > pool and Object the large Object is in? > > Wido > > On 11/15/19 12:23 AM, dhils...@performair.com wrote: >> All; >> >> We had a warning about a large OMAP object pop up in one of our clusters >> overnight. The cluster is configured for CephFS, but nothing mounts a >> CephFS, at this time. >> >> The cluster mostly uses RGW. I've checked the cluster log, the MON log, and >> the MGR log on one of the mons, with no useful references to the pool / pg >> where the large OMAP objects resides. >> >> Is my only option to find this large OMAP object to go through the OSD logs >> for the individual OSDs in the cluster? >> >> Thank you, >> >> Dominic L. Hilsbos, MBA >> Director - Information Technology >> Perform Air International Inc. >> dhils...@performair.com >> www.PerformAir.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP Object
Wido; Ok, yes, I have tracked it down to the index for one of our buckets. I missed the ID in the ceph df output previously. Next time I'll wait to read replies until I've finished my morning coffee. How would I go about correcting this? The content for this bucket is basically just junk, as we're still doing production qualification, and workflow planning. Moving from Windows file shares to self-hosted cloud storage is a significant undertaking. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido den Hollander Sent: Friday, November 15, 2019 8:40 AM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Large OMAP Object On 11/15/19 4:35 PM, dhils...@performair.com wrote: > All; > > Thank you for your help so far. I have found the log entries from when the > object was found, but don't see a reference to the pool. > > Here the logs: > 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub > starts > 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap object > found. Object: > 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key > count: 380425 Size (bytes): 82896978 > In this case it's in pool 56, check 'ceph df' to see which pool that is. To me this seems like a RGW bucket which index grew too big. Use: $ radosgw-admin bucket list $ radosgw-admin metadata get bucket: And match that UUID back to the bucket. Wido > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > > > -Original Message- > From: Wido den Hollander [mailto:w...@42on.com] > Sent: Friday, November 15, 2019 1:56 AM > To: Dominic Hilsbos; ceph-users@lists.ceph.com > Cc: Stephen Self > Subject: Re: [ceph-users] Large OMAP Object > > Did you check /var/log/ceph/ceph.log on one of the Monitors to see which > pool and Object the large Object is in? > > Wido > > On 11/15/19 12:23 AM, dhils...@performair.com wrote: >> All; >> >> We had a warning about a large OMAP object pop up in one of our clusters >> overnight. The cluster is configured for CephFS, but nothing mounts a >> CephFS, at this time. >> >> The cluster mostly uses RGW. I've checked the cluster log, the MON log, and >> the MGR log on one of the mons, with no useful references to the pool / pg >> where the large OMAP objects resides. >> >> Is my only option to find this large OMAP object to go through the OSD logs >> for the individual OSDs in the cluster? >> >> Thank you, >> >> Dominic L. Hilsbos, MBA >> Director - Information Technology >> Perform Air International Inc. >> dhils...@performair.com >> www.PerformAir.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP Object
Paul; I upgraded the cluster in question from 14.2.2 to 14.2.4 just before this came up, so that makes sense. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Paul Emmerich Sent: Friday, November 15, 2019 8:48 AM To: Wido den Hollander Cc: Ceph Users Subject: Re: [ceph-users] Large OMAP Object Note that the size limit changed from 2M keys to 200k keys recently (14.2.3 or 14.2.2 or something), so that object is probably older and that's just the first deep scrub with the reduced limit that triggered the warning. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Nov 15, 2019 at 4:40 PM Wido den Hollander wrote: > > > > On 11/15/19 4:35 PM, dhils...@performair.com wrote: > > All; > > > > Thank you for your help so far. I have found the log entries from when the > > object was found, but don't see a reference to the pool. > > > > Here the logs: > > 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub > > starts > > 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap > > object found. Object: > > 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key > > count: 380425 Size (bytes): 82896978 > > > > In this case it's in pool 56, check 'ceph df' to see which pool that is. > > To me this seems like a RGW bucket which index grew too big. > > Use: > > $ radosgw-admin bucket list > $ radosgw-admin metadata get bucket: > > And match that UUID back to the bucket. > > Wido > > > Thank you, > > > > Dominic L. Hilsbos, MBA > > Director – Information Technology > > Perform Air International Inc. > > dhils...@performair.com > > www.PerformAir.com > > > > > > > > -Original Message- > > From: Wido den Hollander [mailto:w...@42on.com] > > Sent: Friday, November 15, 2019 1:56 AM > > To: Dominic Hilsbos; ceph-users@lists.ceph.com > > Cc: Stephen Self > > Subject: Re: [ceph-users] Large OMAP Object > > > > Did you check /var/log/ceph/ceph.log on one of the Monitors to see which > > pool and Object the large Object is in? > > > > Wido > > > > On 11/15/19 12:23 AM, dhils...@performair.com wrote: > >> All; > >> > >> We had a warning about a large OMAP object pop up in one of our clusters > >> overnight. The cluster is configured for CephFS, but nothing mounts a > >> CephFS, at this time. > >> > >> The cluster mostly uses RGW. I've checked the cluster log, the MON log, > >> and the MGR log on one of the mons, with no useful references to the pool > >> / pg where the large OMAP objects resides. > >> > >> Is my only option to find this large OMAP object to go through the OSD > >> logs for the individual OSDs in the cluster? > >> > >> Thank you, > >> > >> Dominic L. Hilsbos, MBA > >> Director - Information Technology > >> Perform Air International Inc. > >> dhils...@performair.com > >> www.PerformAir.com > >> ___ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP Object
Note that the size limit changed from 2M keys to 200k keys recently (14.2.3 or 14.2.2 or something), so that object is probably older and that's just the first deep scrub with the reduced limit that triggered the warning. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Nov 15, 2019 at 4:40 PM Wido den Hollander wrote: > > > > On 11/15/19 4:35 PM, dhils...@performair.com wrote: > > All; > > > > Thank you for your help so far. I have found the log entries from when the > > object was found, but don't see a reference to the pool. > > > > Here the logs: > > 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub > > starts > > 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap > > object found. Object: > > 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key > > count: 380425 Size (bytes): 82896978 > > > > In this case it's in pool 56, check 'ceph df' to see which pool that is. > > To me this seems like a RGW bucket which index grew too big. > > Use: > > $ radosgw-admin bucket list > $ radosgw-admin metadata get bucket: > > And match that UUID back to the bucket. > > Wido > > > Thank you, > > > > Dominic L. Hilsbos, MBA > > Director – Information Technology > > Perform Air International Inc. > > dhils...@performair.com > > www.PerformAir.com > > > > > > > > -Original Message- > > From: Wido den Hollander [mailto:w...@42on.com] > > Sent: Friday, November 15, 2019 1:56 AM > > To: Dominic Hilsbos; ceph-users@lists.ceph.com > > Cc: Stephen Self > > Subject: Re: [ceph-users] Large OMAP Object > > > > Did you check /var/log/ceph/ceph.log on one of the Monitors to see which > > pool and Object the large Object is in? > > > > Wido > > > > On 11/15/19 12:23 AM, dhils...@performair.com wrote: > >> All; > >> > >> We had a warning about a large OMAP object pop up in one of our clusters > >> overnight. The cluster is configured for CephFS, but nothing mounts a > >> CephFS, at this time. > >> > >> The cluster mostly uses RGW. I've checked the cluster log, the MON log, > >> and the MGR log on one of the mons, with no useful references to the pool > >> / pg where the large OMAP objects resides. > >> > >> Is my only option to find this large OMAP object to go through the OSD > >> logs for the individual OSDs in the cluster? > >> > >> Thank you, > >> > >> Dominic L. Hilsbos, MBA > >> Director - Information Technology > >> Perform Air International Inc. > >> dhils...@performair.com > >> www.PerformAir.com > >> ___ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP Object
On 11/15/19 4:35 PM, dhils...@performair.com wrote: > All; > > Thank you for your help so far. I have found the log entries from when the > object was found, but don't see a reference to the pool. > > Here the logs: > 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub > starts > 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap object > found. Object: > 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key > count: 380425 Size (bytes): 82896978 > In this case it's in pool 56, check 'ceph df' to see which pool that is. To me this seems like a RGW bucket which index grew too big. Use: $ radosgw-admin bucket list $ radosgw-admin metadata get bucket: And match that UUID back to the bucket. Wido > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > > > > -Original Message- > From: Wido den Hollander [mailto:w...@42on.com] > Sent: Friday, November 15, 2019 1:56 AM > To: Dominic Hilsbos; ceph-users@lists.ceph.com > Cc: Stephen Self > Subject: Re: [ceph-users] Large OMAP Object > > Did you check /var/log/ceph/ceph.log on one of the Monitors to see which > pool and Object the large Object is in? > > Wido > > On 11/15/19 12:23 AM, dhils...@performair.com wrote: >> All; >> >> We had a warning about a large OMAP object pop up in one of our clusters >> overnight. The cluster is configured for CephFS, but nothing mounts a >> CephFS, at this time. >> >> The cluster mostly uses RGW. I've checked the cluster log, the MON log, and >> the MGR log on one of the mons, with no useful references to the pool / pg >> where the large OMAP objects resides. >> >> Is my only option to find this large OMAP object to go through the OSD logs >> for the individual OSDs in the cluster? >> >> Thank you, >> >> Dominic L. Hilsbos, MBA >> Director - Information Technology >> Perform Air International Inc. >> dhils...@performair.com >> www.PerformAir.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP Object
All; Thank you for your help so far. I have found the log entries from when the object was found, but don't see a reference to the pool. Here the logs: 2019-11-14 03:10:16.508601 osd.1 (osd.1) 21 : cluster [DBG] 56.7 deep-scrub starts 2019-11-14 03:10:18.325881 osd.1 (osd.1) 22 : cluster [WRN] Large omap object found. Object: 56:f7d15b13:::.dir.f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1:head Key count: 380425 Size (bytes): 82896978 Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com -Original Message- From: Wido den Hollander [mailto:w...@42on.com] Sent: Friday, November 15, 2019 1:56 AM To: Dominic Hilsbos; ceph-users@lists.ceph.com Cc: Stephen Self Subject: Re: [ceph-users] Large OMAP Object Did you check /var/log/ceph/ceph.log on one of the Monitors to see which pool and Object the large Object is in? Wido On 11/15/19 12:23 AM, dhils...@performair.com wrote: > All; > > We had a warning about a large OMAP object pop up in one of our clusters > overnight. The cluster is configured for CephFS, but nothing mounts a > CephFS, at this time. > > The cluster mostly uses RGW. I've checked the cluster log, the MON log, and > the MGR log on one of the mons, with no useful references to the pool / pg > where the large OMAP objects resides. > > Is my only option to find this large OMAP object to go through the OSD logs > for the individual OSDs in the cluster? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP Object
Did you check /var/log/ceph/ceph.log on one of the Monitors to see which pool and Object the large Object is in? Wido On 11/15/19 12:23 AM, dhils...@performair.com wrote: > All; > > We had a warning about a large OMAP object pop up in one of our clusters > overnight. The cluster is configured for CephFS, but nothing mounts a > CephFS, at this time. > > The cluster mostly uses RGW. I've checked the cluster log, the MON log, and > the MGR log on one of the mons, with no useful references to the pool / pg > where the large OMAP objects resides. > > Is my only option to find this large OMAP object to go through the OSD logs > for the individual OSDs in the cluster? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP Object
Hi this probably comes from your RGW which is a big consumer/producer of OMAP for bucket indexes. Have a look at this previous post and just adapt the pool name to match the one where it’s detected: https://www.spinics.net/lists/ceph-users/msg51681.html Regards JC > On Nov 14, 2019, at 15:23, dhils...@performair.com wrote: > > All; > > We had a warning about a large OMAP object pop up in one of our clusters > overnight. The cluster is configured for CephFS, but nothing mounts a > CephFS, at this time. > > The cluster mostly uses RGW. I've checked the cluster log, the MON log, and > the MGR log on one of the mons, with no useful references to the pool / pg > where the large OMAP objects resides. > > Is my only option to find this large OMAP object to go through the OSD logs > for the individual OSDs in the cluster? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information Technology > Perform Air International Inc. > dhils...@performair.com > www.PerformAir.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Large OMAP Object
All; We had a warning about a large OMAP object pop up in one of our clusters overnight. The cluster is configured for CephFS, but nothing mounts a CephFS, at this time. The cluster mostly uses RGW. I've checked the cluster log, the MON log, and the MGR log on one of the mons, with no useful references to the pool / pg where the large OMAP objects resides. Is my only option to find this large OMAP object to go through the OSD logs for the individual OSDs in the cluster? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP object in RGW GC pool
On 6/11/19 9:48 PM, J. Eric Ivancich wrote: > Hi Wido, > > Interleaving below > > On 6/11/19 3:10 AM, Wido den Hollander wrote: >> >> I thought it was resolved, but it isn't. >> >> I counted all the OMAP values for the GC objects and I got back: >> >> gc.0: 0 >> gc.11: 0 >> gc.14: 0 >> gc.15: 0 >> gc.16: 0 >> gc.18: 0 >> gc.19: 0 >> gc.1: 0 >> gc.20: 0 >> gc.21: 0 >> gc.22: 0 >> gc.23: 0 >> gc.24: 0 >> gc.25: 0 >> gc.27: 0 >> gc.29: 0 >> gc.2: 0 >> gc.30: 0 >> gc.3: 0 >> gc.4: 0 >> gc.5: 0 >> gc.6: 0 >> gc.7: 0 >> gc.8: 0 >> gc.9: 0 >> gc.13: 110996 >> gc.10: 04 >> gc.26: 42 >> gc.28: 111292 >> gc.17: 111314 >> gc.12: 111534 >> gc.31: 111956 > > Casey Bodley mentioned to me that he's seen similar behavior to what > you're describing when RGWs are upgraded but not all OSDs are upgraded > as well. Is it possible that the OSDs hosting gc.13, gc.10, and so forth > are running a different version of ceph? > Yes, the OSDs are still on 13.2.5. As this is a big (2500 OSD) production environment we only created a temporary machine with 13.2.6 (just a few hours before it's release) to run the GC. We did not upgrade the cluster itself as we will have to wait with that before we have validated the release on the testing cluster before. Wido > Eric > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP object in RGW GC pool
Hi Wido, Interleaving below On 6/11/19 3:10 AM, Wido den Hollander wrote: > > I thought it was resolved, but it isn't. > > I counted all the OMAP values for the GC objects and I got back: > > gc.0: 0 > gc.11: 0 > gc.14: 0 > gc.15: 0 > gc.16: 0 > gc.18: 0 > gc.19: 0 > gc.1: 0 > gc.20: 0 > gc.21: 0 > gc.22: 0 > gc.23: 0 > gc.24: 0 > gc.25: 0 > gc.27: 0 > gc.29: 0 > gc.2: 0 > gc.30: 0 > gc.3: 0 > gc.4: 0 > gc.5: 0 > gc.6: 0 > gc.7: 0 > gc.8: 0 > gc.9: 0 > gc.13: 110996 > gc.10: 04 > gc.26: 42 > gc.28: 111292 > gc.17: 111314 > gc.12: 111534 > gc.31: 111956 Casey Bodley mentioned to me that he's seen similar behavior to what you're describing when RGWs are upgraded but not all OSDs are upgraded as well. Is it possible that the OSDs hosting gc.13, gc.10, and so forth are running a different version of ceph? Eric -- J. Eric Ivancich he/him/his Red Hat Storage Ann Arbor, Michigan, USA ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP object in RGW GC pool
On 6/4/19 8:00 PM, J. Eric Ivancich wrote: > On 6/4/19 7:37 AM, Wido den Hollander wrote: >> I've set up a temporary machine next to the 13.2.5 cluster with the >> 13.2.6 packages from Shaman. >> >> On that machine I'm running: >> >> $ radosgw-admin gc process >> >> That seems to work as intended! So the PR seems to have fixed it. >> >> Should be fixed permanently when 13.2.6 is officially released. >> >> Wido > > Thank you, Wido, for sharing the results of your experiment. I'm happy > to learn that it was successful. And v13.2.6 was just released about 2 > hours ago. > I thought it was resolved, but it isn't. I counted all the OMAP values for the GC objects and I got back: gc.0: 0 gc.11: 0 gc.14: 0 gc.15: 0 gc.16: 0 gc.18: 0 gc.19: 0 gc.1: 0 gc.20: 0 gc.21: 0 gc.22: 0 gc.23: 0 gc.24: 0 gc.25: 0 gc.27: 0 gc.29: 0 gc.2: 0 gc.30: 0 gc.3: 0 gc.4: 0 gc.5: 0 gc.6: 0 gc.7: 0 gc.8: 0 gc.9: 0 gc.13: 110996 gc.10: 04 gc.26: 42 gc.28: 111292 gc.17: 111314 gc.12: 111534 gc.31: 111956 So as you can see a few remain. I ran: $ radosgw-admin gc process --debug-rados=10 That finishes within 10 seconds. Then I tried: $ radosgw-admin gc process --debug-rados=10 --include-all That also finishes within 10 seconds. What I noticed in the logs was this: 2019-06-11 09:06:58.711 7f8ffb876240 10 librados: call oid=gc.17 nspace= 2019-06-11 09:06:58.717 7f8ffb876240 10 librados: Objecter returned from call r=-16 The return value is '-16' for gc.17 where for gc.18 or any other object with 0 OMAP values it is: 2019-06-11 09:06:58.717 7f8ffb876240 10 librados: call oid=gc.18 nspace= 2019-06-11 09:06:58.720 7f8ffb876240 10 librados: Objecter returned from call r=0 So I set --debug-rgw=10 RGWGC::process failed to acquire lock on gc.17 I haven't tried stopping all the RGWs yet as that will impact the services, but might that be the root-cause here? Wido > Eric > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP object in RGW GC pool
On 6/4/19 7:37 AM, Wido den Hollander wrote: > I've set up a temporary machine next to the 13.2.5 cluster with the > 13.2.6 packages from Shaman. > > On that machine I'm running: > > $ radosgw-admin gc process > > That seems to work as intended! So the PR seems to have fixed it. > > Should be fixed permanently when 13.2.6 is officially released. > > Wido Thank you, Wido, for sharing the results of your experiment. I'm happy to learn that it was successful. And v13.2.6 was just released about 2 hours ago. Eric ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP object in RGW GC pool
On 5/30/19 2:45 PM, Wido den Hollander wrote: > > > On 5/29/19 11:22 PM, J. Eric Ivancich wrote: >> Hi Wido, >> >> When you run `radosgw-admin gc list`, I assume you are *not* using the >> "--include-all" flag, right? If you're not using that flag, then >> everything listed should be expired and be ready for clean-up. If after >> running `radosgw-admin gc process` the same entries appear in >> `radosgw-admin gc list` then gc apparently stalled. >> > > Not using the --include-all in both cases. > > GC seems to stall and doesn't do anything when looking at it with > --debug-rados=10 > >> There were a few bugs within gc processing that could prevent it from >> making forward progress. They were resolved with a PR (master: >> https://github.com/ceph/ceph/pull/26601 ; mimic backport: >> https://github.com/ceph/ceph/pull/27796). Unfortunately that code was >> backported after the 13.2.5 release, but it is in place for the 13.2.6 >> release of mimic. >> > > Thanks! I'll might grab some packages from Shaman to give GC a try. > I've set up a temporary machine next to the 13.2.5 cluster with the 13.2.6 packages from Shaman. On that machine I'm running: $ radosgw-admin gc process That seems to work as intended! So the PR seems to have fixed it. Should be fixed permanently when 13.2.6 is officially released. Wido > Wido > >> Eric >> >> >> On 5/29/19 3:19 AM, Wido den Hollander wrote: >>> Hi, >>> >>> I've got a Ceph cluster with this status: >>> >>> health: HEALTH_WARN >>> 3 large omap objects >>> >>> After looking into it I see that the issue comes from objects in the >>> '.rgw.gc' pool. >>> >>> Investigating it I found that the gc.* objects have a lot of OMAP keys: >>> >>> for OBJ in $(rados -p .rgw.gc ls); do >>> echo $OBJ >>> rados -p .rgw.gc listomapkeys $OBJ|wc -l >>> done >>> >>> I then found out that on average these objects have about 100k of OMAP >>> keys each, but two stand out and have about 3M OMAP keys. >>> >>> I can list the GC with 'radosgw-admin gc list' and this yields a JSON >>> which is a couple of MB in size. >>> >>> I ran: >>> >>> $ radosgw-admin gc process >>> >>> That runs for hours and then finishes, but the large list of OMAP keys >>> stays. >>> >>> Running Mimic 13.3.5 on this cluster. >>> >>> Has anybody seen this before? >>> >>> Wido >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP object in RGW GC pool
On 5/29/19 11:22 PM, J. Eric Ivancich wrote: > Hi Wido, > > When you run `radosgw-admin gc list`, I assume you are *not* using the > "--include-all" flag, right? If you're not using that flag, then > everything listed should be expired and be ready for clean-up. If after > running `radosgw-admin gc process` the same entries appear in > `radosgw-admin gc list` then gc apparently stalled. > Not using the --include-all in both cases. GC seems to stall and doesn't do anything when looking at it with --debug-rados=10 > There were a few bugs within gc processing that could prevent it from > making forward progress. They were resolved with a PR (master: > https://github.com/ceph/ceph/pull/26601 ; mimic backport: > https://github.com/ceph/ceph/pull/27796). Unfortunately that code was > backported after the 13.2.5 release, but it is in place for the 13.2.6 > release of mimic. > Thanks! I'll might grab some packages from Shaman to give GC a try. Wido > Eric > > > On 5/29/19 3:19 AM, Wido den Hollander wrote: >> Hi, >> >> I've got a Ceph cluster with this status: >> >> health: HEALTH_WARN >> 3 large omap objects >> >> After looking into it I see that the issue comes from objects in the >> '.rgw.gc' pool. >> >> Investigating it I found that the gc.* objects have a lot of OMAP keys: >> >> for OBJ in $(rados -p .rgw.gc ls); do >> echo $OBJ >> rados -p .rgw.gc listomapkeys $OBJ|wc -l >> done >> >> I then found out that on average these objects have about 100k of OMAP >> keys each, but two stand out and have about 3M OMAP keys. >> >> I can list the GC with 'radosgw-admin gc list' and this yields a JSON >> which is a couple of MB in size. >> >> I ran: >> >> $ radosgw-admin gc process >> >> That runs for hours and then finishes, but the large list of OMAP keys >> stays. >> >> Running Mimic 13.3.5 on this cluster. >> >> Has anybody seen this before? >> >> Wido >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OMAP object in RGW GC pool
Hi Wido, When you run `radosgw-admin gc list`, I assume you are *not* using the "--include-all" flag, right? If you're not using that flag, then everything listed should be expired and be ready for clean-up. If after running `radosgw-admin gc process` the same entries appear in `radosgw-admin gc list` then gc apparently stalled. There were a few bugs within gc processing that could prevent it from making forward progress. They were resolved with a PR (master: https://github.com/ceph/ceph/pull/26601 ; mimic backport: https://github.com/ceph/ceph/pull/27796). Unfortunately that code was backported after the 13.2.5 release, but it is in place for the 13.2.6 release of mimic. Eric On 5/29/19 3:19 AM, Wido den Hollander wrote: > Hi, > > I've got a Ceph cluster with this status: > > health: HEALTH_WARN > 3 large omap objects > > After looking into it I see that the issue comes from objects in the > '.rgw.gc' pool. > > Investigating it I found that the gc.* objects have a lot of OMAP keys: > > for OBJ in $(rados -p .rgw.gc ls); do > echo $OBJ > rados -p .rgw.gc listomapkeys $OBJ|wc -l > done > > I then found out that on average these objects have about 100k of OMAP > keys each, but two stand out and have about 3M OMAP keys. > > I can list the GC with 'radosgw-admin gc list' and this yields a JSON > which is a couple of MB in size. > > I ran: > > $ radosgw-admin gc process > > That runs for hours and then finishes, but the large list of OMAP keys > stays. > > Running Mimic 13.3.5 on this cluster. > > Has anybody seen this before? > > Wido > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Large OMAP object in RGW GC pool
Hi, I've got a Ceph cluster with this status: health: HEALTH_WARN 3 large omap objects After looking into it I see that the issue comes from objects in the '.rgw.gc' pool. Investigating it I found that the gc.* objects have a lot of OMAP keys: for OBJ in $(rados -p .rgw.gc ls); do echo $OBJ rados -p .rgw.gc listomapkeys $OBJ|wc -l done I then found out that on average these objects have about 100k of OMAP keys each, but two stand out and have about 3M OMAP keys. I can list the GC with 'radosgw-admin gc list' and this yields a JSON which is a couple of MB in size. I ran: $ radosgw-admin gc process That runs for hours and then finishes, but the large list of OMAP keys stays. Running Mimic 13.3.5 on this cluster. Has anybody seen this before? Wido ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] large omap object in usage_log_pool
Thanks Casey. This helped me understand the purpose of this pool. I
trimmed the usage logs which reduced the number of keys stored in that
index significantly and I may even disable the usage log entirely as I
don't believe we use it for anything.
On Fri, May 24, 2019 at 3:51 PM Casey Bodley wrote:
>
>
> On 5/24/19 1:15 PM, shubjero wrote:
> > Thanks for chiming in Konstantin!
> >
> > Wouldn't setting this value to 0 disable the sharding?
> >
> > Reference: http://docs.ceph.com/docs/mimic/radosgw/config-ref/
> >
> > rgw override bucket index max shards
> > Description:Represents the number of shards for the bucket index
> > object, a value of zero indicates there is no sharding. It is not
> > recommended to set a value too large (e.g. thousand) as it increases
> > the cost for bucket listing. This variable should be set in the client
> > or global sections so that it is automatically applied to
> > radosgw-admin commands.
> > Type:Integer
> > Default:0
> >
> > rgw dynamic resharding is enabled:
> > ceph daemon mon.controller1 config show | grep rgw_dynamic_resharding
> > "rgw_dynamic_resharding": "true",
> >
> > I'd like to know more about the purpose of our .usage pool and the
> > 'usage_log_pool' in general as I cant find much about this component
> > of ceph.
>
> You can find docs for the usage log at
> http://docs.ceph.com/docs/master/radosgw/admin/#usage
>
> Unless trimmed, the usage log will continue to grow. If you aren't using
> it, I'd recommend turning it off and trimming it all.
>
> >
> > On Thu, May 23, 2019 at 11:24 PM Konstantin Shalygin wrote:
> >> in the config.
> >> ```"rgw_override_bucket_index_max_shards": "8",```. Should this be
> >> increased?
> >>
> >> Should be decreased to default `0`, I think.
> >>
> >> Modern Ceph releases resolve large omaps automatically via bucket dynamic
> >> resharding:
> >>
> >> ```
> >>
> >> {
> >> "option": {
> >> "name": "rgw_dynamic_resharding",
> >> "type": "bool",
> >> "level": "basic",
> >> "desc": "Enable dynamic resharding",
> >> "long_desc": "If true, RGW will dynamicall increase the number of
> >> shards in buckets that have a high number of objects per shard.",
> >> "default": true,
> >> "daemon_default": "",
> >> "tags": [],
> >> "services": [
> >> "rgw"
> >> ],
> >> "see_also": [
> >> "rgw_max_objs_per_shard"
> >> ],
> >> "min": "",
> >> "max": ""
> >> }
> >> }
> >> ```
> >>
> >> ```
> >>
> >> {
> >> "option": {
> >> "name": "rgw_max_objs_per_shard",
> >> "type": "int64_t",
> >> "level": "basic",
> >> "desc": "Max objects per shard for dynamic resharding",
> >> "long_desc": "This is the max number of objects per bucket index
> >> shard that RGW will allow with dynamic resharding. RGW will trigger an
> >> automatic reshard operation on the bucket if it exceeds this number.",
> >> "default": 10,
> >> "daemon_default": "",
> >> "tags": [],
> >> "services": [
> >> "rgw"
> >> ],
> >> "see_also": [
> >> "rgw_dynamic_resharding"
> >> ],
> >> "min": "",
> >> "max": ""
> >> }
> >> }
> >> ```
> >>
> >>
> >> So when your bucket reached new 100k objects rgw will shard this bucket
> >> automatically.
> >>
> >> Some old buckets may be not sharded, like your ancients from Giant. You
> >> can check fill status like this: `radosgw-admin bucket limit check | jq
> >> '.[]'`. If some buckets is not reshared you can shart it by hand via
> >> `radosgw-admin reshard add ...`. Also, there may be some stale reshard
> >> instances (fixed ~ in 12.2.11), you can check it via `radosgw-admin
> >> reshard stale-instances list` and then remove via `reshard stale-instances
> >> rm`.
> >>
> >>
> >>
> >> k
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] large omap object in usage_log_pool
On 5/24/19 1:15 PM, shubjero wrote:
Thanks for chiming in Konstantin!
Wouldn't setting this value to 0 disable the sharding?
Reference: http://docs.ceph.com/docs/mimic/radosgw/config-ref/
rgw override bucket index max shards
Description:Represents the number of shards for the bucket index
object, a value of zero indicates there is no sharding. It is not
recommended to set a value too large (e.g. thousand) as it increases
the cost for bucket listing. This variable should be set in the client
or global sections so that it is automatically applied to
radosgw-admin commands.
Type:Integer
Default:0
rgw dynamic resharding is enabled:
ceph daemon mon.controller1 config show | grep rgw_dynamic_resharding
"rgw_dynamic_resharding": "true",
I'd like to know more about the purpose of our .usage pool and the
'usage_log_pool' in general as I cant find much about this component
of ceph.
You can find docs for the usage log at
http://docs.ceph.com/docs/master/radosgw/admin/#usage
Unless trimmed, the usage log will continue to grow. If you aren't using
it, I'd recommend turning it off and trimming it all.
On Thu, May 23, 2019 at 11:24 PM Konstantin Shalygin wrote:
in the config.
```"rgw_override_bucket_index_max_shards": "8",```. Should this be
increased?
Should be decreased to default `0`, I think.
Modern Ceph releases resolve large omaps automatically via bucket dynamic
resharding:
```
{
"option": {
"name": "rgw_dynamic_resharding",
"type": "bool",
"level": "basic",
"desc": "Enable dynamic resharding",
"long_desc": "If true, RGW will dynamicall increase the number of shards in
buckets that have a high number of objects per shard.",
"default": true,
"daemon_default": "",
"tags": [],
"services": [
"rgw"
],
"see_also": [
"rgw_max_objs_per_shard"
],
"min": "",
"max": ""
}
}
```
```
{
"option": {
"name": "rgw_max_objs_per_shard",
"type": "int64_t",
"level": "basic",
"desc": "Max objects per shard for dynamic resharding",
"long_desc": "This is the max number of objects per bucket index shard that
RGW will allow with dynamic resharding. RGW will trigger an automatic reshard operation on the
bucket if it exceeds this number.",
"default": 10,
"daemon_default": "",
"tags": [],
"services": [
"rgw"
],
"see_also": [
"rgw_dynamic_resharding"
],
"min": "",
"max": ""
}
}
```
So when your bucket reached new 100k objects rgw will shard this bucket
automatically.
Some old buckets may be not sharded, like your ancients from Giant. You can
check fill status like this: `radosgw-admin bucket limit check | jq '.[]'`. If
some buckets is not reshared you can shart it by hand via `radosgw-admin
reshard add ...`. Also, there may be some stale reshard instances (fixed ~ in
12.2.11), you can check it via `radosgw-admin reshard stale-instances list` and
then remove via `reshard stale-instances rm`.
k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] large omap object in usage_log_pool
Thanks for chiming in Konstantin!
Wouldn't setting this value to 0 disable the sharding?
Reference: http://docs.ceph.com/docs/mimic/radosgw/config-ref/
rgw override bucket index max shards
Description:Represents the number of shards for the bucket index
object, a value of zero indicates there is no sharding. It is not
recommended to set a value too large (e.g. thousand) as it increases
the cost for bucket listing. This variable should be set in the client
or global sections so that it is automatically applied to
radosgw-admin commands.
Type:Integer
Default:0
rgw dynamic resharding is enabled:
ceph daemon mon.controller1 config show | grep rgw_dynamic_resharding
"rgw_dynamic_resharding": "true",
I'd like to know more about the purpose of our .usage pool and the
'usage_log_pool' in general as I cant find much about this component
of ceph.
On Thu, May 23, 2019 at 11:24 PM Konstantin Shalygin wrote:
>
> in the config.
> ```"rgw_override_bucket_index_max_shards": "8",```. Should this be
> increased?
>
> Should be decreased to default `0`, I think.
>
> Modern Ceph releases resolve large omaps automatically via bucket dynamic
> resharding:
>
> ```
>
> {
> "option": {
> "name": "rgw_dynamic_resharding",
> "type": "bool",
> "level": "basic",
> "desc": "Enable dynamic resharding",
> "long_desc": "If true, RGW will dynamicall increase the number of
> shards in buckets that have a high number of objects per shard.",
> "default": true,
> "daemon_default": "",
> "tags": [],
> "services": [
> "rgw"
> ],
> "see_also": [
> "rgw_max_objs_per_shard"
> ],
> "min": "",
> "max": ""
> }
> }
> ```
>
> ```
>
> {
> "option": {
> "name": "rgw_max_objs_per_shard",
> "type": "int64_t",
> "level": "basic",
> "desc": "Max objects per shard for dynamic resharding",
> "long_desc": "This is the max number of objects per bucket index
> shard that RGW will allow with dynamic resharding. RGW will trigger an
> automatic reshard operation on the bucket if it exceeds this number.",
> "default": 10,
> "daemon_default": "",
> "tags": [],
> "services": [
> "rgw"
> ],
> "see_also": [
> "rgw_dynamic_resharding"
> ],
> "min": "",
> "max": ""
> }
> }
> ```
>
>
> So when your bucket reached new 100k objects rgw will shard this bucket
> automatically.
>
> Some old buckets may be not sharded, like your ancients from Giant. You can
> check fill status like this: `radosgw-admin bucket limit check | jq '.[]'`.
> If some buckets is not reshared you can shart it by hand via `radosgw-admin
> reshard add ...`. Also, there may be some stale reshard instances (fixed ~ in
> 12.2.11), you can check it via `radosgw-admin reshard stale-instances list`
> and then remove via `reshard stale-instances rm`.
>
>
>
> k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] large omap object in usage_log_pool
in the config.
```"rgw_override_bucket_index_max_shards": "8",```. Should this be
increased?
Should be decreased to default `0`, I think.
Modern Ceph releases resolve large omaps automatically via bucket
dynamic resharding:
```
{
"option": {
"name": "rgw_dynamic_resharding",
"type": "bool",
"level": "basic",
"desc": "Enable dynamic resharding",
"long_desc": "If true, RGW will dynamicall increase the number
of shards in buckets that have a high number of objects per shard.",
"default": true,
"daemon_default": "",
"tags": [],
"services": [
"rgw"
],
"see_also": [
"rgw_max_objs_per_shard"
],
"min": "",
"max": ""
}
}
```
```
{
"option": {
"name": "rgw_max_objs_per_shard",
"type": "int64_t",
"level": "basic",
"desc": "Max objects per shard for dynamic resharding",
"long_desc": "This is the max number of objects per bucket
index shard that RGW will allow with dynamic resharding. RGW will
trigger an automatic reshard operation on the bucket if it exceeds this
number.",
"default": 10,
"daemon_default": "",
"tags": [],
"services": [
"rgw"
],
"see_also": [
"rgw_dynamic_resharding"
],
"min": "",
"max": ""
}
}
```
So when your bucket reached new 100k objects rgw will shard this bucket
automatically.
Some old buckets may be not sharded, like your ancients from Giant. You
can check fill status like this: `radosgw-admin bucket limit check | jq
'.[]'`. If some buckets is not reshared you can shart it by hand via
`radosgw-admin reshard add ...`. Also, there may be some stale reshard
instances (fixed ~ in 12.2.11), you can check it via `radosgw-admin
reshard stale-instances list` and then remove via `reshard
stale-instances rm`.
k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] large omap object in usage_log_pool
Hi there, We have an old cluster that was built on Giant that we have maintained and upgraded over time and are now running Mimic 13.2.5. The other day we received a HEALTH_WARN about 1 large omap object in the pool '.usage' which is our usage_log_pool defined in our radosgw zone. I am trying to understand the purpose of the usage_log_pool and whether or not we have appropriate settings (shards, replicas, etc) in place. We were able to identify the 1 large omap object as 'usage.22' in the .usage pool. This particular "bucket" had over 2 million "omapkeys" ```for i in `rados -p .usage ls`; do echo $i; rados -p .usage listomapkeys $i | wc -l; done``` -snip- usage.13 20 usage.22 2023790 usage.25 14 -snip- These keys all seem to be metadata/pointers of valid data from our OpenStack's object storage where we hold about 1PB of unique data. To resolve the HEALTH_WARN we changeg the 'osd_deep_scrub_large_omap_object_key_threshold' from '200' to '250' using 'ceph config set osd ...' on our Mon's. I'd like to know the importance of this pool as I also noticed that this pool's replication is only set to 2, instead of 3 like all our other pools with the exception of .users.email (also 2). If important, I'd like to set the replication to 3 and curious to know if there would be any negative impact to the cluster. The .usage pool says 0 bytes used in 'ceph df' but it contains 30 objects for which there are many omapkeys. I am also wondering about bucket index max shards for which we have '8' set in the config. ```"rgw_override_bucket_index_max_shards": "8",```. Should this be increased? Thanks in advance for any responses, I have found this mailing list to be an excellent source of information! Jared Baker Ontario Institute for Cancer Research ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] large omap object
There may be a mismatch between be auto-restarting and the omap warning
code. Looks like you already have 349 shards, with 13 of them warning on
size!
You can increase a config value to shut that error up, but you may want to
get somebody from RGW to look at how you’ve managed to exceed those default
limits.
On Wed, Jun 13, 2018 at 4:36 AM stephan schultchen <
stephan.schultc...@gmail.com> wrote:
> Hello,
>
> i am running a ceph 13.2.0 cluster exclusively for radosrw / s3.
>
> i only have one big bucket. and the cluster is currently in warning state:
>
> cluster:
> id: d605c463-9f1c-4d91-a390-a28eedb21650
> health: HEALTH_WARN
> 13 large omap objects
>
> i tried to google it, but i was not able to find what to do about the
> "large omap objects".
>
> as far as i understand ceph should automatically re shard an s3 bucket
> when an omap is getting to big. or is this something i have to do?
>
> "radosgw-admin reshard list" tells that no resharding is ongoing right now.
>
>
> radosgw-admin metadata get
> bucket.instance:nuxeo_live:6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.4854.4
> {
> "key":
> "bucket.instance:nuxeo_live:6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.4854.4",
> "ver": {
> "tag": "Y2epzPoujRDfxM5CNMZgKPaA",
> "ver": 6
> },
> "mtime": "2018-06-08 14:48:15.515840Z",
> "data": {
> "bucket_info": {
> "bucket": {
> "name": "nuxeo_live",
> "marker": "6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.4848.1",
> "bucket_id": "6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.4854.4",
> "tenant": "",
> "explicit_placement": {
> "data_pool": "",
> "data_extra_pool": "",
> "index_pool": ""
> }
> },
> "creation_time": "2018-05-23 13:31:57.664398Z",
> "owner": "nuxeo_live",
> "flags": 0,
> "zonegroup": "506cc27c-fef5-4b89-a9f3-4c928a74b955",
> "placement_rule": "default-placement",
> "has_instance_obj": "true",
> "quota": {
> "enabled": false,
> "check_on_raw": false,
> "max_size": -1,
> "max_size_kb": 0,
> "max_objects": -1
> },
> "num_shards": 349,
> "bi_shard_hash_type": 0,
> "requester_pays": "false",
> "has_website": "false",
> "swift_versioning": "false",
> "swift_ver_location": "",
> "index_type": 0,
> "mdsearch_config": [],
> "reshard_status": 2,
> "new_bucket_instance_id":
> "6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.176143.1"
> },
> "attrs": [
> {
> "key": "user.rgw.acl",
> "val":
> "AgKpAwIhCgAAAG51eGVvX2xpdmUPbnV4ZW8gbGl2ZSB1c2VyBAN8AQEKbnV4ZW9fbGl2ZQ8BCgAAAG51eGVvX2xpdmUFA0UCAgQACgAAAG51eGVvX2xpdmUAAAICBA8PbnV4ZW8gbGl2ZSB1c2VyAA=="
> },
> {
> "key": "user.rgw.idtag",
> "val": ""
> }
> ]
>
> i also tried to manually trigger a resharding. but it failed with:
>
>
>- NOTICE: operation will not remove old bucket index objects ***
>- these will need to be removed manually ***
>tenant:
>bucket name: nuxeo_live
>old bucket instance id: 6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.184670.1
>new bucket instance id: 6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.176197.1
>WARNING: RGWReshard::add failed to drop lock on
>bucket_name:6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.184670.1 ret=-2
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] large omap object
Hello,
i am running a ceph 13.2.0 cluster exclusively for radosrw / s3.
i only have one big bucket. and the cluster is currently in warning state:
cluster:
id: d605c463-9f1c-4d91-a390-a28eedb21650
health: HEALTH_WARN
13 large omap objects
i tried to google it, but i was not able to find what to do about the
"large omap objects".
as far as i understand ceph should automatically re shard an s3 bucket when
an omap is getting to big. or is this something i have to do?
"radosgw-admin reshard list" tells that no resharding is ongoing right now.
radosgw-admin metadata get
bucket.instance:nuxeo_live:6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.4854.4
{
"key":
"bucket.instance:nuxeo_live:6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.4854.4",
"ver": {
"tag": "Y2epzPoujRDfxM5CNMZgKPaA",
"ver": 6
},
"mtime": "2018-06-08 14:48:15.515840Z",
"data": {
"bucket_info": {
"bucket": {
"name": "nuxeo_live",
"marker": "6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.4848.1",
"bucket_id": "6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.4854.4",
"tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"creation_time": "2018-05-23 13:31:57.664398Z",
"owner": "nuxeo_live",
"flags": 0,
"zonegroup": "506cc27c-fef5-4b89-a9f3-4c928a74b955",
"placement_rule": "default-placement",
"has_instance_obj": "true",
"quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"num_shards": 349,
"bi_shard_hash_type": 0,
"requester_pays": "false",
"has_website": "false",
"swift_versioning": "false",
"swift_ver_location": "",
"index_type": 0,
"mdsearch_config": [],
"reshard_status": 2,
"new_bucket_instance_id":
"6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.176143.1"
},
"attrs": [
{
"key": "user.rgw.acl",
"val":
"AgKpAwIhCgAAAG51eGVvX2xpdmUPbnV4ZW8gbGl2ZSB1c2VyBAN8AQEKbnV4ZW9fbGl2ZQ8BCgAAAG51eGVvX2xpdmUFA0UCAgQACgAAAG51eGVvX2xpdmUAAAICBA8PbnV4ZW8gbGl2ZSB1c2VyAA=="
},
{
"key": "user.rgw.idtag",
"val": ""
}
]
i also tried to manually trigger a resharding. but it failed with:
- NOTICE: operation will not remove old bucket index objects ***
- these will need to be removed manually ***
tenant:
bucket name: nuxeo_live
old bucket instance id: 6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.184670.1
new bucket instance id: 6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.176197.1
WARNING: RGWReshard::add failed to drop lock on
bucket_name:6f85d718-fd2e-4c1b-a21d-bafb04a8cfcc.184670.1 ret=-2
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
