Any news on this? I you give me some hints, I can try to reproduce it here.
-Arne On 10.07.2012 08:57, Arne Jansen wrote: > On 10.07.2012 06:16, Sami Liedes wrote: >> On Mon, Jul 09, 2012 at 11:05:47AM +0200, Arne Jansen wrote: >>>> * Just before the crash: >>>> btrfs: invalid parameters for read_extent_buffer: start (32771) > >>>> eb->len (32768). eb start is 2261163409408, level 100, generation >>>> 4412718571037421157, nritems 538968254. len param 17. debug >>>> 2/989/538968254/4412718571037421157/0x0/0/0x0/0x0 >>>> >>> >>> At a first glance: the generation converted to ascii is: "ent() ==", >>> so someone is patching the memory with ascii text, possibly C source. >>> It might be interesting to dump the full contents of the eb, to get >>> a clue on the source of the data. >> >> I changed the code to dump the contents of the eb struct at the point >> where that error (btrfs: invalid parameters...) is printed, at the >> "checksum mismatch 4" site and at the "node seems invalid now" site. >> Now I have a big log of 1795 corrupted ebs. So far nothing that looks >> remotely like ascii text, though. But I have two different versions of >> the eb that caused that warning, a less corrupted one and a more >> corrupted one: >> >> ------------------------------------------------------------ >> btrfs: --- start eb contents at ffff8801b13cc4c8 --- >> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 >> ...f............ >> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> .......0....... >> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff >> X.<.....X.<..... >> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff >> ........p.<..... >> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 >> p.<............. >> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff >> ._.............. >> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff >> @L.......f...... >> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff >> ........@....... >> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff >> @............... >> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 >> ..<............. >> btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 >> ........ > > this one looks good so far. > >> btrfs: --- end eb contents at ffff8801b13cc4c8 --- >> btrfs: dm-6 checksum verify failed on 2239404212224 wanted B5F632BC found >> 3579FB59 level 160 >> btrfs: node seems invalid now. checksum ok = 1 >> btrfs: --- start eb contents at ffff8801b13cc4c8 --- >> [... identical dump to above ...] >> btrfs: --- end eb contents at ffff8801b13cc4c8 --- >> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len >> (32768). eb start is 2239404212224, level 160, generation >> 4716553384049587249, nritems 295705211. len param 17. debug >> 2/989/295705211/4716553384049587249/0x0/0/0x0/0x0 >> btrfs: --- start eb contents at ffff8801b13cc4c8 --- >> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 >> ...f............ >> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> .......0....... >> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff >> X.<.....X.<..... >> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff >> ........p.<..... >> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 >> p.<............. >> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff >> ._.............. >> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff >> @L.......f...... >> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff >> ........@....... >> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff >> @............... >> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 >> ..<............. >> btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 >> ........{....... >> btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 >> 14q<P.tA........ >> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 >> ........ >> btrfs: --- end eb contents at ffff8801b13cc4c8 --- >> ------------[ cut here ]------------ >> WARNING: at fs/btrfs/extent_io.c:4533 read_extent_buffer+0x1ef/0x220 >> [btrfs]() >> Hardware name: System Product Name >> Modules linked in: <omitted> [last unloaded: scsi_wait_scan] >> Pid: 25829, comm: btrfs-endio-met Tainted: G W 3.4.4+btrfsdebug3+ >> #4 >> [...] >> ------------------------------------------------------------ >> >> Here's the difference (lines reordered from diff to make comparing >> easier): >> >> ------------------------------------------------------------ >> -btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 >> ..<............. >> +btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 >> ..<............. >> -btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 >> ........{....... >> -btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 >> 14q<P.tA........ >> ------------------------------------------------------------ > > The diff is just the eb->debug[] Jan added. > >> >> If there's one pattern that catches eye in the dumps, it's that in >> many places where the same eb is dumped multiple times due to multiple >> "checksum mismatch 4"s, there are bytes at offsets 0x30 and 0x31 that >> always seem to have the same value and both separately increase >> between the dumps, usually(?) by two. Then that might be normal, I >> haven't looked at what should be at a struct eb :) There are some >> instances where they wrap, for example >> >> f8 f8 -> fa fa -> fc fc -> fe fe -> 00 00 -> 02 02 -> ..., > > this is the spinlock to protect the refcnt, so some reference taking/releasing > has been going on. > >> >> with no other changes visible in the corrupted ebs. But that is by no >> means the only change in the struct eb contents that can be observed, >> only the most obvious pattern. >> >> For example, here's a 11-long string of eb contents. First the dmesg: >> >> ------------------------------------------------------------ >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 1] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 2] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 3] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 4] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 5] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 6] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 7] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 8] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 9] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 10] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found >> C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 11] >> ------------------------------------------------------------ >> >> Where DUMP 1 contents (file cc1.txt if you'd want to apply the diffs >> below) is >> >> ------------------------------------------------------------ >> btrfs: --- start eb contents at ffff8801cafbacc0 --- >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> btrfs: ffff8801cafbad30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad40: 00 00 10 00 00 00 00 00 04 04 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad50: 50 ad fb ca 01 88 ff ff 50 ad fb ca 01 88 ff ff >> P.......P....... >> btrfs: ffff8801cafbad60: 02 02 00 00 00 00 00 00 68 ad fb ca 01 88 ff ff >> ........h....... >> btrfs: ffff8801cafbad70: 68 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 >> h............... >> btrfs: ffff8801cafbad80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad90: 00 a2 20 08 00 ea ff ff c0 2a f4 07 00 ea ff ff .. >> ......*...... >> btrfs: ffff8801cafbada0: 00 ac 89 07 00 ea ff ff c0 42 79 07 00 ea ff ff >> .........By..... >> btrfs: ffff8801cafbadb0: 80 af 77 07 00 ea ff ff 40 37 49 08 00 ea ff ff >> ..w.....@7I..... >> btrfs: ffff8801cafbadc0: 80 6d f3 07 00 ea ff ff c0 a7 e8 07 00 ea ff ff >> .m.............. >> btrfs: ffff8801cafbadd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 >> ........ >> btrfs: --- end eb contents at ffff8801cafbacc0 --- >> ------------------------------------------------------------ >> >> And the successive diffs are >> >> ------------------------------------------------------------ >> --- cc1.txt 2012-07-10 06:57:21.564665577 +0300 >> +++ cc2.txt 2012-07-10 06:59:10.272634578 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ > > the spinlock > >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> @@ -20,9 +20,9 @@ >> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> -btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 >> ................ >> -btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> -btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 01 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801cafbae20: 4d 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 >> M.......N....... >> +btrfs: ffff8801cafbae30: 07 78 02 00 00 00 00 00 00 00 00 00 00 00 00 00 >> .x.............. > > and eb->debug > >> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 >> ........ >> btrfs: --- end eb contents at ffff8801cafbacc0 --- >> --- cc2.txt 2012-07-10 06:59:10.272634578 +0300 >> +++ cc3.txt 2012-07-10 06:59:10.016634663 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> --- cc3.txt 2012-07-10 06:59:10.016634663 +0300 >> +++ cc4.txt 2012-07-10 06:59:09.752634749 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> --- cc4.txt 2012-07-10 06:59:09.752634749 +0300 >> +++ cc5.txt 2012-07-10 06:59:09.504634831 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> --- cc5.txt 2012-07-10 06:59:09.504634831 +0300 >> +++ cc6.txt 2012-07-10 06:59:09.240634917 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> --- cc6.txt 2012-07-10 06:59:09.240634917 +0300 >> +++ cc7.txt 2012-07-10 06:59:08.944635015 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> --- cc7.txt 2012-07-10 06:59:08.944635015 +0300 >> +++ cc8.txt 2012-07-10 06:59:08.656635109 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> +btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> !!.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> --- cc8.txt 2012-07-10 06:59:08.656635109 +0300 >> +++ cc9.txt 2012-07-10 06:59:08.328635216 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> !!.............. >> +btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ##.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> --- cc9.txt 2012-07-10 06:59:08.328635216 +0300 >> +++ cc10.txt 2012-07-10 06:58:33.180646115 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ##.............. >> +btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> %%.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> --- cc10.txt 2012-07-10 06:58:33.180646115 +0300 >> +++ cc11.txt 2012-07-10 06:58:41.340643696 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 >> ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff >> 5.......0....... >> -btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> %%.............. >> +btrfs: ffff8801cafbacf0: 27 27 00 00 03 00 00 00 00 00 00 00 00 00 00 00 >> ''.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> "............... >> ------------------------------------------------------------ > > All spinlocks. > > I'll go through this again with Jan to see if his eb->debug might give > any clue here. It might be good to have a look at the contents of the > pages though, as written in the previous mail. > > -Arne > >> >> And since it's of course possible there's something wrong with my >> modifications, here's the patch I used: >> >> ------------------------------------------------------------ >> diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c >> index 7452ecb..aadb82c 100644 >> --- a/fs/btrfs/extent_io.c >> +++ b/fs/btrfs/extent_io.c >> @@ -4527,6 +4527,9 @@ void read_extent_buffer(struct extent_buffer *eb, void >> *dstv, >> len, >> eb->debug[0], eb->debug[1], eb->debug[2], eb->debug[3], >> eb->debug[4], eb->debug[5], eb->debug[6], eb->debug[7]); >> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); >> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, >> eb, sizeof(*eb), true); >> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); >> WARN_ON(1); >> } >> WARN_ON(start + len > eb->start + eb->len); >> diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c >> index ea81bd4..663d6c4 100644 >> --- a/fs/btrfs/reada.c >> +++ b/fs/btrfs/reada.c >> @@ -130,9 +130,13 @@ static int __readahead_hook(struct btrfs_root *root, >> struct extent_buffer *eb, >> kref_get(&re->refcnt); >> spin_unlock(&fs_info->reada_lock); >> >> - if (!err && btrfs_csum_tree_block(root, eb)) >> + if (!err && btrfs_csum_tree_block(root, eb)) { >> printk(KERN_ERR "btrfs: checksum mismatch 4 on %llu\n", >> eb->start); >> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); >> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, >> eb, sizeof(*eb), true); >> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); >> + } >> >> if (!re) >> return -1; >> @@ -150,9 +154,14 @@ static int __readahead_hook(struct btrfs_root *root, >> struct extent_buffer *eb, >> if (err == 0) { >> nritems = level ? btrfs_header_nritems(eb) : 0; >> if (level > BTRFS_MAX_LEVEL || >> - nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) >> + nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) { >> printk(KERN_ERR "btrfs: node seems invalid now. >> checksum ok = %d\n", >> btrfs_csum_tree_block(root, eb)); >> + printk(KERN_ERR "btrfs: --- start eb contents at %p >> ---\n", eb); >> + print_hex_dump(KERN_ERR, "btrfs: ", >> DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); >> + printk(KERN_ERR "btrfs: --- end eb contents at %p >> ---\n", eb); >> + >> + } >> generation = btrfs_header_generation(eb); >> /* >> * FIXME: currently we just set nritems to 0 if this is a leaf, >> >> ------------------------------------------------------------ >> >> Sami > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html