On Mon, Mar 13, 2017 at 1:35 PM, John Spray <jsp...@redhat.com> wrote:
> On Mon, Mar 13, 2017 at 10:28 AM, Dan van der Ster <d...@vanderster.com>
> wrote:
>> Hi John,
>>
>> Last week we updated our prod CephFS cluster to 10.2.6 (clients and
>> server side), and for the first time today we've got an object info
>> size mismatch:
>>
>> I found this ticket you created in the tracker, which is why I've
>> emailed you: http://tracker.ceph.com/issues/18240
>>
>> Here's the detail of our error:
>>
>> 2017-03-13 07:17:49.989297 osd.67 <snip>:6819/3441125 262 : cluster
>> [ERR] deep-scrub 1.3da 1:5bc0e9dc:::10000260f4b.00000003:head on disk
>> size (4187974) does not match object info size (4193094) adjusted for
>> ondisk to (4193094)
>
>
> Hmm, never investigated that particular ticket but did notice that the
> issue never re-occurred on master, or on jewel since the start of
> 2017.
>
> The last revision where it cropped up in testing was
> 5b402f8a7b5a763852e93cd0a5decd34572f4518, looking at the history
> between that commit and the tip of the jewel branch I don't see
> anything that I recognise as a corruption fix (apart from the omap fix
> which is clearly unrelated as we're looking at data objects in these
> failures).
>
> John
Thanks for checking, John. In the end we deleted the inconsistent
file, rescrubbed, and will just keep a look out for similar cases.
-- Dan
>
>
>> All three replica's have the same object size/md5sum:
>>
>> # ls -l 10000260f4b.00000003__head_3B9703DA__1
>> -rw-r--r--. 1 ceph ceph 4187974 Mar 12 18:50
>> 10000260f4b.00000003__head_3B9703DA__1
>> # md5sum 10000260f4b.00000003__head_3B9703DA__1
>> db1e1bab199b33fce3ad9195832626ef 10000260f4b.00000003__head_3B9703DA__1
>>
>> And indeed the object info does not agree with the files on disk:
>>
>> # ceph-dencoder type object_info_t import /tmp/attr1 decode dump_json
>> {
>> "oid": {
>> "oid": "10000260f4b.00000003",
>> "key": "",
>> "snapid": -2,
>> "hash": 999752666,
>> "max": 0,
>> "pool": 1,
>> "namespace": ""
>> },
>> "version": "5262'221037",
>> "prior_version": "5262'221031",
>> "last_reqid": "osd.67.0:1180241",
>> "user_version": 221031,
>> "size": 4193094,
>> "mtime": "0.000000",
>> "local_mtime": "0.000000",
>> "lost": 0,
>> "flags": 52,
>> "snaps": [],
>> "truncate_seq": 80,
>> "truncate_size": 0,
>> "data_digest": 2779145704,
>> "omap_digest": 4294967295,
>> "watchers": {}
>> }
>>
>>
>> PG repair doesn't handle these kind of corruptions, but I found a
>> recipe in an old thread to fix the object info with hexedit. Before
>> doing this I wanted to see if we can understand exactly how this is
>> possible.
>>
>> I managed to find the exact cephfs file, and asked the user how they
>> created it. They said the file was the output of a make test on some
>> program. The make test was taking awhile, so they left their laptop,
>> and when they returned to the computer, the ssh connection to their
>> cephfs workstation had broken. I assume this means that the process
>> writing the file had been killed while writing to cephfs. But I don't
>> understand how a killed client process could result in inconsistent
>> object info.
>>
>> Is there anything else needed to help debug this inconsistency?
>>
>> Cheers, Dan
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
