Thanks Robin, Of the two issues, this seems to me like it must be #22928.
Since the majority of index entries for this bucket are in the .rgw.buckets pool, but newer entries have been created in .rgw.buckets.index, it's clearly failing to use the explicit placement pool - and with the index data split across two pools I don't see how resharding could correct this.
I can get the object names from the "new" (incorrect) indexes with something like:
for i in `rados -p .rgw.buckets.index ls - | grep "default.2049236.2"`; do rados -p .rgw.buckets.index listomapkeys $i|grep "^[a-zA-Z0-9]"; done
Fortunately there are only ~20, in this bucket (my grep is just a stupid way to skip what I assume are the multipart parts, which have a non-ascii first char).
... and these files are downloadable, at least using s3cmd (minio client fails, it seems to try and check the index first).
Once I have these newer files downloaded, then to restore access to the older index I like the suggestion in the issue tracker to create a new placement target in the zone, and modify the bucket's placement rule to match. It seems like it might be safer than copying objects from one index pool to the other (though the latter certainly sounds faster and easier!)
From a quick check, I suspect I probably have 40 or so other buckets with this problem... will need to check them more closely.
Actually it looks like a lot of the affected buckets were created around 10/2016 - I suspect the placement policies were incorrect for a short time due to confusion over the hammer->jewel upgrade (the realm/period/zonegroup/zone conversion didn't really go smoothly!)
On 02/16/2018 11:39 PM, Robin H. Johnson wrote:
On Fri, Feb 16, 2018 at 07:06:21PM -0600, Graham Allan wrote: [snip great debugging] This seems similar to two open issues, could be either of them depending on how old that bucket is. http://tracker.ceph.com/issues/22756 http://tracker.ceph.com/issues/22928 - I have a mitigation posted to 22756. - There's a PR posted for 22928, but it'll probably only be in v12.2.4.
-- Graham Allan Minnesota Supercomputing Institute - g...@umn.edu _______________________________________________ ceph-users mailing list email@example.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com