Hi all,
After a major failure, and getting our cluster health back OK (with some
help from inktank folks, thanks), we found out that we have managed to
corrupt one of our bucket indices. As far as i can track it, we are missing
the omapheader on that specific index, so we're unable to use radosgw-admin
tools to fix it.
While a healthy (smaller) bucket answers
# radosgw-admin bucket check -b imgdoviz
{ "existing_header": { "usage": { "rgw.main": { "size_kb": 4140,
"size_kb_actual": 4484,
"num_objects": 157}}},
"calculated_header": { "usage": { "rgw.main": { "size_kb": 4140,
"size_kb_actual": 4484,
"num_objects": 157}}}}
The faulty one fails with
# radosgw-admin bucket check -b imgiz
failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
Invalid argument
failed to check index err=(22) Invalid argument
When i push further
# radosgw-admin bucket check -b imgiz --check-objects --fix
failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
Invalid argument
Checking objects, decreasing bucket 2-phase commit timeout.
** Note that timeout will reset only when operation completes successfully
**
ERROR: failed operation r=-22
ERROR: failed operation r=-22
..
last line keeps repeating without any progress.
I checked the file omapheaders and while the healty bucket has:
# rados -p .rgw.buckets getomapheader .dir.6912.3
header (49 bytes) :
0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 : ..+.............
0010 : 00 a8 af 40 00 00 00 00 00 00 10 46 00 00 00 00 : [email protected]....
0020 : 00 9d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
0030 : 00 : .
the faulty one is missing it
# rados -p .rgw.buckets getomapheader .dir.4470.1
header (0 bytes) :
I'm currently in the process of understanding how to create a readable
header. My hopes are even while its wrong, radosgw-admin will be able to
read through objects and fix the necessary parts. But i'm not sure how to
set the new-header. It seems there is a setomapheader counterpart for
getomapheader but it only accepts values from commandline so i don't know
how to push a rgw-readable binary with it.
Is this somewhat possible?
Thanks in advance.
--
erdem agaoglu
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
