On Wed, Feb 17, 2021 at 7:43 PM Daniel Dawson <danielcdaw...@gmail.com> wrote:
>
> I was attempting to replace the drives in an array with RAID6 profile.
metadata raid6 as well?
What replacement command(s) are you using?
> The first replacement was seemingly successful (and there was a scrub
> afterward, with no errors). However, about 0.6% into the second
> replacement (sdc), something went wrong, and it went read-only (I should
> have copied the log of that somehow). Now it refuses to mount, and a
> (readonly) check cannot get started.
>
>
> # mount -o ro,degraded /dev/sda3 /mnt
> mount: /mnt: can't read superblock on /dev/sda3.
> # btrfs rescue super-recover /dev/sda3
> All supers are valid, no need to recover
>
>
> For this, dmesg shows:
>
> [ 202.675384] BTRFS info (device sdc3): allowing degraded mounts
> [ 202.675387] BTRFS info (device sdc3): disk space caching is enabled
> [ 202.675389] BTRFS info (device sdc3): has skinny extents
> [ 202.676302] BTRFS warning (device sdc3): devid 3 uuid
> 911a642e-0a4c-4483-9a1f-cde7b87c5519 is missing
> [ 202.676601] BTRFS warning (device sdc3): devid 3 uuid
> 911a642e-0a4c-4483-9a1f-cde7b87c5519 is missing
What device is devid 3?
> [ 202.985528] BTRFS info (device sdc3): bdev /dev/sdb3 errs: wr 0, rd
> 0, flush 0, corrupt 26, gen 0
> [ 202.985533] BTRFS info (device sdc3): bdev /dev/sdd3 errs: wr 0, rd
> 0, flush 0, corrupt 98, gen 0
> [ 203.278131] BTRFS info (device sdc3): start tree-log replay
> [ 203.454496] BTRFS critical (device sdc3): corrupt leaf: root=7
> block=371567214592 slot=0, unexpected item end, have 16315 expect 16283
> [ 203.454499] BTRFS error (device sdc3): block=371567214592 read time
> tree block corruption detected
> [ 203.454634] BTRFS critical (device sdc3): corrupt leaf: root=7
> block=371567214592 slot=0, unexpected item end, have 16315 expect 16283
> [ 203.454636] BTRFS error (device sdc3): block=371567214592 read time
> tree block corruption detected
> [ 203.455794] BTRFS critical (device sdc3): corrupt leaf: root=7
> block=371567214592 slot=0, unexpected item end, have 16315 expect 16283
16315=0x3fbb, 16283=0x3f9b, 16315^16283 = 32 or 0x20
11111110111011
11111110011011
^
Do a RAM test for as long as you can tolerate it, or it finds the
defect. Sometimes they show up quickly, other times days.
> [ 203.455796] BTRFS error (device sdc3): block=371567214592 read time
> tree block corruption detected
> [ 203.455820] BTRFS: error (device sdc3) in __btrfs_free_extent:3105:
> errno=-5 IO failure
> [ 203.455823] BTRFS: error (device sdc3) in
> btrfs_run_delayed_refs:2208: errno=-5 IO failure
> [ 203.455833] BTRFS: error (device sdc3) in btrfs_replay_log:2287:
> errno=-5 IO failure (Failed to recover log tree)
> [ 203.747758] BTRFS error (device sdc3): open_ctree failed
>
>
> I've looked for, but can't find, any bad blocks on the devices. Also, if
> it adds any info...
>
> # btrfs check --readonly /dev/sda3
> Opening filesystem to check...
> warning, device 3 is missing
> checksum verify failed on 371587727360 found 000000FF wanted 00000049
> checksum verify failed on 371587727360 found 00000005 wanted 00000010
> checksum verify failed on 371587727360 found 00000005 wanted 00000010
> bad tree block 371587727360, bytenr mismatch, want=371587727360,
> have=1076190010624
> ERROR: could not setup extent tree
> ERROR: cannot open file system
>
>
> Note: I'm running this off of System Rescue 7.01, which has earlier
> versions of things than what the machine in question has installed (the
> latter being Linux 5.10.16, with btrfs-progs v5.10.1).
>
> # uname -a
> Linux sysrescue 5.4.78-1-lts #1 SMP Wed, 18 Nov 2020 19:51:49 +0000
> x86_64 GNU/Linux
> # btrfs --version
> btrfs-progs v5.4.1
> # btrfs filesystem show
> Label: 'vroot2020' uuid: 5214d903-783a-4d14-ac78-046da5ac1db7
> Total devices 4 FS bytes used 65.98GiB
> devid 0 size 457.64GiB used 39.53GiB path /dev/sdc3
> devid 1 size 457.64GiB used 39.56GiB path /dev/sda3
> devid 2 size 457.64GiB used 39.56GiB path /dev/sdb3
> devid 4 size 457.64GiB used 39.53GiB path /dev/sdd3
This is confusing. devid 3 is claimed to be missing, but fi show isn't
showing any missing devices. If none of sd[abcd] are devid 3, then
what dev node is devid 3 and where is it?
But yeah you're probably best off not trying to fix this file system
until the memory is sorted out.
--
Chris Murphy
