2012-01-13 4:12, Jim Klimov wrote:
As I recently wrote, my data pool has experienced some
"unrecoverable errors". It seems that a userdata block
of deduped data got corrupted and no longer matches the
stored checksum. For whatever reason, raidz2 did not
help in recovery of this data, so I rsync'ed the files
over from another copy. Then things got interesting...
Well, after some crawling over my data with zdb, od and dd,
I guess ZFS was right about finding checksum errors - the
metadata's checksum matched that of a block on original
system, and the data block was indeed erring.
Just in case it helps others, the SHA256 checksums can be
tested with openssl as I show below. I still search for a
command-line fletcher4/fletcher2 checker (as that weak
hash is used on metadata; I wonder why).
Here's a tail from on-disk blkptr_t, bytes with checksum:
# tail -2 /tmp/osx.l0+1100000.blkptr.txt
000460 1f 6f 4c 73 5d c1 ab 15 00 cc 56 90 38 8e b4 dd
000470 a9 8e 54 6f f1 a7 db 43 7d 61 9e 01 23 45 2e 70
In byte 0x435 I have value 0x8 - SHA256. And here is the
SHA256 hash for the excerpt from original file (128Kb
cut out with dd):
# dd if=osx.zip of=/tmp/osx.l0+1100000.bin.orig bs=512 skip=34816 count=256
# openssl dgst -sha256 < /tmp/osx.l0+1100000.bin.orig
As my x86 is little-endian, the four 8-byte words of
the checksum appear reversed. But you can see it matches,
so my source file is okay.
I did not find the DDT entries (yet), so I don't know
what hash is there or what addresses it references for
how many files. The block pointer has the dedup bit set,
However, of all my files with errors, there are no DVA
I hexdumped (with od) the two 128Kb excerpts (one from the
original file, another fetched with zdb) and diffed them,
and while some lines matched, others did not.
What is more interesting, is that most of the error area
contains a repeating pattern like this, sometimes with
"extra" chars thrown in:
fc 42 fc 42 fc 42 fc 42 fc 42 fc
fc 42 1f fc 42 fc 42
42 ff fc 42 fc 42 fc 42 fc 42
I have seen similar patterns when I zdb-dumped compressed
blocks without decompression, so I guess this could be a
miswrite of compressed data and/or parity destined for
another file (which also did not get it).
The erroneous data starts and ends at "round" offsets like
0x1000-0x2000, 0x9000-0xa000, 0x11000-0x12000 (step 0x8000
between both sets of mismatches, size 4kb is my disk sector
size), which also suggests a non-coincidental problem.
However, part of the differing data is "normal-looking
random noise", while some part is that pattern above,
starting and ending at a seemingly random location mid-sector.
Here's about all I have to say and share so far :)
Open to suggestions how to compute fletcher checksums
zfs-discuss mailing list