Hi,
Only 2 of the buckets are really used:
"buckets": [
{
"bucket": "test",
"tenant": "",
"num_objects": 968107,
"num_shards": 16,
"objects_per_shard": 60506,
"fill_status": "OK"
}
]
"buckets": [
{
"bucket": "prototype",
"tenant": "",
"num_objects": 4633533,
"num_shards": 98,
"objects_per_shard": 47280,
"fill_status": "OK"
}
]
In the log it shows this:
cluster [WRN] Large omap object found. Object:
7:3019f91b:::.dir.2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.213.0:head PG:
7.d89f980c (7.c) Key count: 206795 Size (bytes): 46499042
pool id 7 is default.rgw.buckets.index
The entry in the log doesn't match any of the bucket ids but this one exists:
"id": "2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637",
Thanks to a RedHat KB article that mentioned checking for stale instances I ran
this:
# radosgw-admin reshard stale-instances list | wc -l
713
This command:
```
for obj in $(rados -p default.rgw.buckets.index ls | grep
2b67ef7c-2015-4ca0-bf50-b7595d01e46e.74194.637); do
printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys
$obj | wc -l) done ```
returns this:
-bash: command substitution: line 4: syntax error: unexpected end of file
I figured perhaps you were using ``` to denote code so I tried running it
without that and also on one line and neither of those did the trick.
Is that just bash?
Thanks so much for all of your help thus far.
-Drew
-----Original Message-----
From: Benoît Knecht <bkne...@protonmail.ch>
Sent: Friday, March 5, 2021 12:00 PM
To: Drew Weaver <drew.wea...@thenap.com>
Cc: 'ceph-users@ceph.io' <ceph-users@ceph.io>
Subject: RE: [ceph-users] Resolving LARGE_OMAP_OBJECTS
On Friday, March 5th, 2021 at 15:20, Drew Weaver <drew.wea...@thenap.com> wrote:
> Sorry to sound clueless but no matter what I search for on El Goog I can't
> figure out how to answer the question as to whether dynamic sharding is
> enabled in our environment.
>
> It's not configured as true in the config files, but it is the default.
>
> Is there a radosgw-admin command to determine whether or not it's enabled in
> the running environment?
If `rgw_dynamic_resharding` is not explicitly set to `false` in your
environment, I think we can assume dynamic resharding is enabled. And if any of
your buckets have more than one shard and you didn't reshard them manually,
you'll know for sure dynamic resharding is working; you can check the number of
shards on a bucket with `radosgw-admin bucket stats --bucket=<name>`, there's a
`num_shards` field. You can also check with `radosgw-admin bucket limit check`
if any of your buckets are about to be resharded.
Assuming dynamic resharding is enabled and none of your buckets are about to be
resharded, I would then find out which object has too many OMAP keys by
grepping the logs. The name of the object will contain the bucket ID (also
found in the output of `radosgw-admin bucket stats`), so you'll know which
bucket is causing the issue. And you can check how many OMAP keys are in each
shard of that bucket index using
```
for obj in $(rados -p default.rgw.buckets.index ls | grep
eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4); do
printf "%-60s %7d\n" $obj $(rados -p default.rgw.buckets.index listomapkeys
$obj | wc -l) done ```
(where `eaf0ece5-9f4a-4aa8-9d67-8c6698f7919b.88726492.4` is your bucket ID). If
the number of keys are very uneven amongst the shards, there's probably an
issue that needs to be addressed. If you they are relatively even but slightly
above the warning threshold, it's probably a versioned bucket, and it should be
safe to simply increase the threshold.
Cheers,
--
Ben
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
