Hi all, following the thread "Adventures in btrfs raid5 disk recovery", I investigated a bit the BTRFS capability to scrub a corrupted raid5 filesystem. To test it, I first find where a file was stored, and then I tried to corrupt the data disks (when unmounted) or the parity disk. The result showed that sometime the kernel recomputed the parity wrongly.
I tested the following kernel
- 4.6.1
- 4.5.4
and both showed the same behavior.
The test was performed as described below:
1) create a filesystem in raid5 mode (for data and metadata) of 1500MB
truncate -s 500M disk1.img; losetup -f disk1.img
truncate -s 500M disk2.img; losetup -f disk2.img
truncate -s 500M disk3.img; losetup -f disk3.img
sudo mkfs.btrfs -d raid5 -m raid5 /dev/loop[0-2]
sudo mount /dev/loop0 mnt/
2) I created a file with a length of 128kb:
python -c "print 'ad'+'a'*65534+'bd'+'b'*65533" | sudo tee mnt/out.txt
sudo umount mnt/
3) I looked at the output of 'btrfs-debug-tree /dev/loop0' and I was able to
find where the file stripe is located:
/dev/loop0: offset=81788928+16*4096 (64k, second half of the file:
'bdbbbb.....)
/dev/loop1: offset=61865984+16*4096 (64k, first half of the file:
'adaaaa.....)
/dev/loop2: offset=61865984+16*4096 (64k, parity:
'\x03\x00\x03\x03\x03.....)
4) I tried to corrupt each disk (one disk per test), and then run a scrub:
for example for the disk /dev/loop2:
sudo dd if=/dev/zero of=/dev/loop2 bs=1 \
seek=$((61865984+16*4096)) count=5
sudo mount /dev/loop0 mnt
sudo btrfs scrub start mnt/.
5) I check the disks at the offsets above, to verify that the data/parity is
correct
However I found that:
1) if I corrupt the parity disk (/dev/loop2), scrub don't find any corruption,
but recomputed the parity (always correctly);
2) when I corrupted the other disks (/dev/loop[01]) btrfs was able to find the
corruption. But I found two main behaviors:
2.a) the kernel repaired the damage, but compute the wrong parity. Where it was
the parity, the kernel copied the data of the second disk on the parity disk
2.b) the kernel repaired the damage, and rebuild a correct parity
I have to point out another strange thing: in dmesg I found two kinds of
messages:
msg1)
[....]
[ 1021.366944] BTRFS info (device loop2): disk space caching is enabled
[ 1021.366949] BTRFS: has skinny extents
[ 1021.399208] BTRFS warning (device loop2): checksum error at logical
142802944 on dev /dev/loop0, sector 159872, root 5, inode 257, offset 65536,
length 4096, links 1 (path: out.txt)
[ 1021.399214] BTRFS error (device loop2): bdev /dev/loop0 errs: wr 0,
rd 0, flush 0, corrupt 1, gen 0
[ 1021.399291] BTRFS error (device loop2): fixed up error at logical
142802944 on dev /dev/loop0
msg2)
[ 1017.435068] BTRFS info (device loop2): disk space caching is enabled
[ 1017.435074] BTRFS: has skinny extents
[ 1017.436778] BTRFS info (device loop2): bdev /dev/loop0 errs: wr 0,
rd 0, flush 0, corrupt 1, gen 0
[ 1017.463403] BTRFS warning (device loop2): checksum error at logical
142802944 on dev /dev/loop0, sector 159872, root 5, inode 257, offset
65536, length 4096, links 1 (path: out.txt)
[ 1017.463409] BTRFS error (device loop2): bdev /dev/loop0 errs: wr 0,
rd 0, flush 0, corrupt 2, gen 0
[ 1017.463467] BTRFS warning (device loop2): checksum error at logical
142802944 on dev /dev/loop0, sector 159872, root 5, inode 257, offset 65536,
length 4096, links 1 (path: out.txt)
[ 1017.463472] BTRFS error (device loop2): bdev /dev/loop0 errs: wr 0,
rd 0, flush 0, corrupt 3, gen 0
[ 1017.463512] BTRFS error (device loop2): unable to fixup (regular)
error at logical 142802944 on dev /dev/loop0
[ 1017.463535] BTRFS error (device loop2): fixed up error at logical
142802944 on dev /dev/loop0
but these seem to be UNrelated to the kernel behavior 2.a) or 2.b)
Another strangeness is that SCRUB sometime reports
ERROR: there are uncorrectable errors
and sometime reports
WARNING: errors detected during scrubbing, corrected
but also these seems UNrelated to the behavior 2.a) or 2.b) or msg1 or msg2
Enclosed you can find the script which I used to trigger the bug. I have to
rerun it several times to show the problem because it doesn't happen every
time. Pay attention that the offset and the loop device name are hard coded.
You must run the script in the same directory where it is: eg "bash test.sh".
Br
G.Baroncelli
--
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
--
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
test.sh
Description: Bourne shell script
