I was able to remove the meta objects, but the cluster is still in WARN state HEALTH_WARN 1 large omap objects LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'us-prd-1.rgw.log' Search the cluster log for 'Large omap object found' for more details.
How do I go about clearing it out? I don't see any other references to large omap in any of the logs. I've tried restarted the mgr's, the monitors, and even the osd that reported the issue. -Brett On Thu, Jul 25, 2019 at 2:55 PM Brett Chancellor <bchancel...@salesforce.com> wrote: > 14.2.1 > Thanks, I'll try that. > > On Thu, Jul 25, 2019 at 2:54 PM Casey Bodley <cbod...@redhat.com> wrote: > >> What ceph version is this cluster running? Luminous or later should not >> be writing any new meta.log entries when it detects a single-zone >> configuration. >> >> I'd recommend editing your zonegroup configuration (via 'radosgw-admin >> zonegroup get' and 'put') to set both log_meta and log_data to false, >> then commit the change with 'radosgw-admin period update --commit'. >> >> You can then delete any meta.log.* and data_log.* objects from your log >> pool using the rados tool. >> >> On 7/25/19 2:30 PM, Brett Chancellor wrote: >> > Casey, >> > These clusters were setup with the intention of one day doing multi >> > site replication. That has never happened. The cluster has a single >> > realm, which contains a single zonegroup, and that zonegroup contains >> > a single zone. >> > >> > -Brett >> > >> > On Thu, Jul 25, 2019 at 2:16 PM Casey Bodley <cbod...@redhat.com >> > <mailto:cbod...@redhat.com>> wrote: >> > >> > Hi Brett, >> > >> > These meta.log objects store the replication logs for metadata >> > sync in >> > multisite. Log entries are trimmed automatically once all other >> zones >> > have processed them. Can you verify that all zones in the multisite >> > configuration are reachable and syncing? Does 'radosgw-admin sync >> > status' on any zone show that it's stuck behind on metadata sync? >> > That >> > would prevent these logs from being trimmed and result in these >> large >> > omap warnings. >> > >> > On 7/25/19 1:59 PM, Brett Chancellor wrote: >> > > I'm having an issue similar to >> > > >> > >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033611.html >> . >> > >> > > I don't see where any solution was proposed. >> > > >> > > $ ceph health detail >> > > HEALTH_WARN 1 large omap objects >> > > LARGE_OMAP_OBJECTS 1 large omap objects >> > > 1 large objects found in pool 'us-prd-1.rgw.log' >> > > Search the cluster log for 'Large omap object found' for >> > more details. >> > > >> > > $ grep "Large omap object" /var/log/ceph/ceph.log >> > > 2019-07-25 14:58:21.758321 osd.3 (osd.3) 15 : cluster [WRN] >> > Large omap >> > > object found. Object: >> > > >> 51:61eb35fe:::meta.log.e557cf47-46df-4b45-988e-9a94c5004a2e.19:head >> > > Key count: 3382154 Size (bytes): 611384043 >> > > >> > > $ rados -p us-prd-1.rgw.log listomapkeys >> > > meta.log.e557cf47-46df-4b45-988e-9a94c5004a2e.19 |wc -l >> > > 3382154 >> > > >> > > $ rados -p us-prd-1.rgw.log listomapvals >> > > meta.log.e557cf47-46df-4b45-988e-9a94c5004a2e.19 >> > > This returns entries from almost every bucket, across multiple >> > > tenants. Several of the entries are from buckets that no longer >> > exist >> > > on the system. >> > > >> > > $ ceph df |egrep 'OBJECTS|.rgw.log' >> > > POOL ID STORED OBJECTS USED %USED MAX >> > > AVAIL >> > > us-prd-1.rgw.log 51 758 MiB 228 758 MiB >> > > 0 102 TiB >> > > >> > > Thanks, >> > > >> > > -Brett >> > > >> > > _______________________________________________ >> > > ceph-users mailing list >> > > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> >> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > >> >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com