Any news on this? I you give me some hints, I can try to reproduce
it here.
-Arne
On 10.07.2012 08:57, Arne Jansen wrote:
> On 10.07.2012 06:16, Sami Liedes wrote:
>> On Mon, Jul 09, 2012 at 11:05:47AM +0200, Arne Jansen wrote:
>>>> * Just before the crash:
>>>> btrfs: invalid parameters for read_extent_buffer: start (32771) >
>>>> eb->len (32768). eb start is 2261163409408, level 100, generation
>>>> 4412718571037421157, nritems 538968254. len param 17. debug
>>>> 2/989/538968254/4412718571037421157/0x0/0/0x0/0x0
>>>>
>>>
>>> At a first glance: the generation converted to ascii is: "ent() ==",
>>> so someone is patching the memory with ascii text, possibly C source.
>>> It might be interesting to dump the full contents of the eb, to get
>>> a clue on the source of the data.
>>
>> I changed the code to dump the contents of the eb struct at the point
>> where that error (btrfs: invalid parameters...) is printed, at the
>> "checksum mismatch 4" site and at the "node seems invalid now" site.
>> Now I have a big log of 1795 corrupted ebs. So far nothing that looks
>> remotely like ascii text, though. But I have two different versions of
>> the eb that caused that warning, a less corrupted one and a more
>> corrupted one:
>>
>> ------------------------------------------------------------
>> btrfs: --- start eb contents at ffff8801b13cc4c8 ---
>> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00
>> ...f............
>> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> .......0.......
>> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff
>> X.<.....X.<.....
>> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff
>> ........p.<.....
>> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00
>> p.<.............
>> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff
>> ._..............
>> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff
>> @L.......f......
>> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff
>> ........@.......
>> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff
>> @...............
>> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00
>> ..<.............
>> btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00
>> ........
>
> this one looks good so far.
>
>> btrfs: --- end eb contents at ffff8801b13cc4c8 ---
>> btrfs: dm-6 checksum verify failed on 2239404212224 wanted B5F632BC found
>> 3579FB59 level 160
>> btrfs: node seems invalid now. checksum ok = 1
>> btrfs: --- start eb contents at ffff8801b13cc4c8 ---
>> [... identical dump to above ...]
>> btrfs: --- end eb contents at ffff8801b13cc4c8 ---
>> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len
>> (32768). eb start is 2239404212224, level 160, generation
>> 4716553384049587249, nritems 295705211. len param 17. debug
>> 2/989/295705211/4716553384049587249/0x0/0/0x0/0x0
>> btrfs: --- start eb contents at ffff8801b13cc4c8 ---
>> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00
>> ...f............
>> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> .......0.......
>> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff
>> X.<.....X.<.....
>> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff
>> ........p.<.....
>> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00
>> p.<.............
>> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff
>> ._..............
>> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff
>> @L.......f......
>> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff
>> ........@.......
>> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff
>> @...............
>> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00
>> ..<.............
>> btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00
>> ........{.......
>> btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00
>> 14q<P.tA........
>> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00
>> ........
>> btrfs: --- end eb contents at ffff8801b13cc4c8 ---
>> ------------[ cut here ]------------
>> WARNING: at fs/btrfs/extent_io.c:4533 read_extent_buffer+0x1ef/0x220
>> [btrfs]()
>> Hardware name: System Product Name
>> Modules linked in: <omitted> [last unloaded: scsi_wait_scan]
>> Pid: 25829, comm: btrfs-endio-met Tainted: G W 3.4.4+btrfsdebug3+
>> #4
>> [...]
>> ------------------------------------------------------------
>>
>> Here's the difference (lines reordered from diff to make comparing
>> easier):
>>
>> ------------------------------------------------------------
>> -btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00
>> ..<.............
>> +btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00
>> ..<.............
>> -btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00
>> ........{.......
>> -btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00
>> 14q<P.tA........
>> ------------------------------------------------------------
>
> The diff is just the eb->debug[] Jan added.
>
>>
>> If there's one pattern that catches eye in the dumps, it's that in
>> many places where the same eb is dumped multiple times due to multiple
>> "checksum mismatch 4"s, there are bytes at offsets 0x30 and 0x31 that
>> always seem to have the same value and both separately increase
>> between the dumps, usually(?) by two. Then that might be normal, I
>> haven't looked at what should be at a struct eb :) There are some
>> instances where they wrap, for example
>>
>> f8 f8 -> fa fa -> fc fc -> fe fe -> 00 00 -> 02 02 -> ...,
>
> this is the spinlock to protect the refcnt, so some reference taking/releasing
> has been going on.
>
>>
>> with no other changes visible in the corrupted ebs. But that is by no
>> means the only change in the struct eb contents that can be observed,
>> only the most obvious pattern.
>>
>> For example, here's a 11-long string of eb contents. First the dmesg:
>>
>> ------------------------------------------------------------
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 1]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 2]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 3]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 4]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 5]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 6]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 7]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 8]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 9]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 10]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found
>> C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 11]
>> ------------------------------------------------------------
>>
>> Where DUMP 1 contents (file cc1.txt if you'd want to apply the diffs
>> below) is
>>
>> ------------------------------------------------------------
>> btrfs: --- start eb contents at ffff8801cafbacc0 ---
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> btrfs: ffff8801cafbad30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad40: 00 00 10 00 00 00 00 00 04 04 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad50: 50 ad fb ca 01 88 ff ff 50 ad fb ca 01 88 ff ff
>> P.......P.......
>> btrfs: ffff8801cafbad60: 02 02 00 00 00 00 00 00 68 ad fb ca 01 88 ff ff
>> ........h.......
>> btrfs: ffff8801cafbad70: 68 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00
>> h...............
>> btrfs: ffff8801cafbad80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad90: 00 a2 20 08 00 ea ff ff c0 2a f4 07 00 ea ff ff ..
>> ......*......
>> btrfs: ffff8801cafbada0: 00 ac 89 07 00 ea ff ff c0 42 79 07 00 ea ff ff
>> .........By.....
>> btrfs: ffff8801cafbadb0: 80 af 77 07 00 ea ff ff 40 37 49 08 00 ea ff ff
>> ..w.....@7I.....
>> btrfs: ffff8801cafbadc0: 80 6d f3 07 00 ea ff ff c0 a7 e8 07 00 ea ff ff
>> .m..............
>> btrfs: ffff8801cafbadd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00
>> ........
>> btrfs: --- end eb contents at ffff8801cafbacc0 ---
>> ------------------------------------------------------------
>>
>> And the successive diffs are
>>
>> ------------------------------------------------------------
>> --- cc1.txt 2012-07-10 06:57:21.564665577 +0300
>> +++ cc2.txt 2012-07-10 06:59:10.272634578 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>
> the spinlock
>
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> @@ -20,9 +20,9 @@
>> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> -btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00
>> ................
>> -btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> -btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 01 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801cafbae20: 4d 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00
>> M.......N.......
>> +btrfs: ffff8801cafbae30: 07 78 02 00 00 00 00 00 00 00 00 00 00 00 00 00
>> .x..............
>
> and eb->debug
>
>> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00
>> ........
>> btrfs: --- end eb contents at ffff8801cafbacc0 ---
>> --- cc2.txt 2012-07-10 06:59:10.272634578 +0300
>> +++ cc3.txt 2012-07-10 06:59:10.016634663 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> --- cc3.txt 2012-07-10 06:59:10.016634663 +0300
>> +++ cc4.txt 2012-07-10 06:59:09.752634749 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> --- cc4.txt 2012-07-10 06:59:09.752634749 +0300
>> +++ cc5.txt 2012-07-10 06:59:09.504634831 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> --- cc5.txt 2012-07-10 06:59:09.504634831 +0300
>> +++ cc6.txt 2012-07-10 06:59:09.240634917 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> --- cc6.txt 2012-07-10 06:59:09.240634917 +0300
>> +++ cc7.txt 2012-07-10 06:59:08.944635015 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> --- cc7.txt 2012-07-10 06:59:08.944635015 +0300
>> +++ cc8.txt 2012-07-10 06:59:08.656635109 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> +btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> !!..............
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> --- cc8.txt 2012-07-10 06:59:08.656635109 +0300
>> +++ cc9.txt 2012-07-10 06:59:08.328635216 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> !!..............
>> +btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ##..............
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> --- cc9.txt 2012-07-10 06:59:08.328635216 +0300
>> +++ cc10.txt 2012-07-10 06:58:33.180646115 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ##..............
>> +btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> %%..............
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> --- cc10.txt 2012-07-10 06:58:33.180646115 +0300
>> +++ cc11.txt 2012-07-10 06:58:41.340643696 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00
>> ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff
>> 5.......0.......
>> -btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> %%..............
>> +btrfs: ffff8801cafbacf0: 27 27 00 00 03 00 00 00 00 00 00 00 00 00 00 00
>> ''..............
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> "...............
>> ------------------------------------------------------------
>
> All spinlocks.
>
> I'll go through this again with Jan to see if his eb->debug might give
> any clue here. It might be good to have a look at the contents of the
> pages though, as written in the previous mail.
>
> -Arne
>
>>
>> And since it's of course possible there's something wrong with my
>> modifications, here's the patch I used:
>>
>> ------------------------------------------------------------
>> diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
>> index 7452ecb..aadb82c 100644
>> --- a/fs/btrfs/extent_io.c
>> +++ b/fs/btrfs/extent_io.c
>> @@ -4527,6 +4527,9 @@ void read_extent_buffer(struct extent_buffer *eb, void
>> *dstv,
>> len,
>> eb->debug[0], eb->debug[1], eb->debug[2], eb->debug[3],
>> eb->debug[4], eb->debug[5], eb->debug[6], eb->debug[7]);
>> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb);
>> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1,
>> eb, sizeof(*eb), true);
>> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb);
>> WARN_ON(1);
>> }
>> WARN_ON(start + len > eb->start + eb->len);
>> diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c
>> index ea81bd4..663d6c4 100644
>> --- a/fs/btrfs/reada.c
>> +++ b/fs/btrfs/reada.c
>> @@ -130,9 +130,13 @@ static int __readahead_hook(struct btrfs_root *root,
>> struct extent_buffer *eb,
>> kref_get(&re->refcnt);
>> spin_unlock(&fs_info->reada_lock);
>>
>> - if (!err && btrfs_csum_tree_block(root, eb))
>> + if (!err && btrfs_csum_tree_block(root, eb)) {
>> printk(KERN_ERR "btrfs: checksum mismatch 4 on %llu\n",
>> eb->start);
>> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb);
>> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1,
>> eb, sizeof(*eb), true);
>> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb);
>> + }
>>
>> if (!re)
>> return -1;
>> @@ -150,9 +154,14 @@ static int __readahead_hook(struct btrfs_root *root,
>> struct extent_buffer *eb,
>> if (err == 0) {
>> nritems = level ? btrfs_header_nritems(eb) : 0;
>> if (level > BTRFS_MAX_LEVEL ||
>> - nritems > BTRFS_NODEPTRS_PER_BLOCK(root))
>> + nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) {
>> printk(KERN_ERR "btrfs: node seems invalid now.
>> checksum ok = %d\n",
>> btrfs_csum_tree_block(root, eb));
>> + printk(KERN_ERR "btrfs: --- start eb contents at %p
>> ---\n", eb);
>> + print_hex_dump(KERN_ERR, "btrfs: ",
>> DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true);
>> + printk(KERN_ERR "btrfs: --- end eb contents at %p
>> ---\n", eb);
>> +
>> + }
>> generation = btrfs_header_generation(eb);
>> /*
>> * FIXME: currently we just set nritems to 0 if this is a leaf,
>>
>> ------------------------------------------------------------
>>
>> Sami
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
