Hi,
A few weeks ago I found radosgw index has been inconsistent with
reality. Some object I can not list, but I can get them by key. Please see
the details below:
*BACKGROUND：*
Ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous
(stable)
Index pool is on ssd.
There is a very big bucket with more than 10 million object and 500TB
data.
Ceph health is OK.
I use s3 api on radosgw.
*DESCRIBE:*
When use s3 list_object() to list, some uploaded object can not be
listed and some uploaded object have an old lastModified time.
But at the same time, we can get this object by an exact key. And if I
put a new object into this bucket, it can be listed.
It seems that some indexes during a period of time have been lost.
I try to run "*radosgw-admin bucket check --bucket <bucketname> --fix
--check-objects*" and I get nothing at all.
*SOME ELSE：*
I found that one bucket will have many indexes, and we can use
"*radosgw-admin
metadata list bucket.instance | grep "{bucket name}*" to show them. But I
can not found a doc to describe this feature. And we can use "*radosgw-admin
bucket stats --bucket {bucket_name}*" to get id as the active instance id.
I use "*rados listomapkeys*" at active(or latest) index to get all
object in a index, it is really lost. But when I use "*rados listomapkeys*"
at another index which is not active as mentioned above, I found the lost
object index.
* Resharding is within my consideration. Listomapkeys means do this
action on all shards(more than 300). In my understanding, a big bucket
has one latest index and many old indexes. Every index has many shards. So
listomapkeys on a index means listomapkeys on many shards.*
*QUESTION:*
Why my index lost?
How to recover？
Why radosgw has many index instances, how do radosgw use them and how
to change active index?
Thanks,
Inksink
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
