Re: [ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Troy Ablan
Yep, that's on me. I did enable it in the config originally, and I think that I thought at the time that it might be useful, but I wasn't aware of a sharding caveat owing to most of our traffic is happening on one rgw user. I think I know what I need to do to fix it now though. Thanks

Re: [ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Paul Emmerich
Yeah, the number of shards is configurable ("rgw usage num shards"? or something). Are you sure you aren't using it? This feature is not enabled by default, someone had to explicitly set "rgw enable usage log" for you to run into this problem. Paul -- Paul Emmerich Looking for help with your

Re: [ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Troy Ablan
Paul, Apparently never. Appears to (potentially) have every request from the beginning of time (late last year, in my case). In our use case, we don't really need this data (not multi-tenant), so I might simply clear it. But in the case where this were an extremely high transaction cluster

Re: [ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Paul Emmerich
Looks like the usage log (radosgw-admin usage show), how often do you trim it? -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Oct 14, 2019 at 11:55 PM Troy Ablan

[ceph-users] default.rgw.log contains large omap object

2019-10-14 Thread Troy Ablan
Hi folks, Mimic cluster here, RGW pool with only default zone. I have a persistent error here LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details. I think I've narrowed it