Chris Murphy wrote on 2015/12/13 21:16 -0700:
Part 1= What to do about it? This post.
Part 2 = How I got here? I'm still working on the write up, so it's
not yet posted.
Summary:
2 dev (spinning rust) raid1 for data and metadata.
kernel 4.2.6, btrfs-progs 4.2.2
btrfs check with devid 1 and 2 present produces thousands of scary
messages, e.g.
checksum verify failed on 714189357056 found E4E3BDB6 wanted 00000000
Checked the full output.
The interesting part is, the calculated result is always E4E3BDB6, and
wanted is always all 0.
I assume E4E3BDB6 is crc32 of all 0 data.
If there is a full disk dump, it will be much easier to find where the
problem is.
But I'm a afraid it won't be possible.
At least, 'btrfs-debug-tree -t 2' should help to locate what's wrong
with the bytenr in the warning.
The good news is, the fs seems to be OK without major problem.
As except the csum error, btrfsck doesn't give other error/warning.
btrfs check with devid 1 or devid2 separate (the other is missing)
produces no such scary messages at all, but instead messages e.g.
failed to load free space cache for block group 357585387520
a. This inconsistency is unexpected.
b. the 'btrfs check' with combined devices gives no insight to the
seriousness of "checksum verify failed" messages, or what the solution
is.
I guess btrfsck did the wrong device assemble, but that's just my
personal guess.
And since I can't reproduce in my test environment, it won't be easy to
find the root cause.
c. combined or separate+degraded, read-only mounts succeed with no
errors in user space or dmesg; only normal mount messages happen. With
both devs ro mounted, I was able to completely btrfs send/receive the
most recent two ro snapshots comprising 100% (minus stale historical)
data on the drive, with zero errors reported.
d. no read-write mount attempt has happened since "the incident" which
will be detailed in part 2.
Details:
The full devid1&2 btrfs check is long and not very interesting, so
I've put that here:
https://drive.google.com/open?id=0B_2Asp8DGjJ9Vjd0VlNYb09LVFU
btrfs-show-super shows some differences, values denoted as
devid1/devid2. If there's no split, those values are the same for both
devids.
generation 4924/4923
root 714189258752/714188554240
sys_array_size 129
chunk_root_generation 4918
root_level 1
chunk_root 715141414912
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 1500312748032
bytes_used 537228206080
sectorsize 4096
nodesize 16384
[snip]
cache_generation 4924/4923
uuid_tree_generation 4924/4923
[snip]
dev_item.total_bytes 750156374016
dev_item.bytes_used 541199433728
Perhaps useful, is at the time of "the incident" this volume was rw
mounted, but was being used by a single process only: btrfs send. So
it was used as a source. No writes, other than btrfs's own generation
increment, were happening.
So in theory, this should perhaps be the simplest case of "what do I
do now?" and even makes me wonder if a normal rw mount should just fix
this up: either btrfs uses generation 4924 and updates all changes
from 4923 and 4924 automatically to devid2 so they are now in sync, or
it automatically discards generation 4924 from devid1, so both devices
are in sync.
The workload, circumstances of "the incident", the general purpose of
btrfs, and the likelihood a typical user would never have even become
aware of "the incident" until much later than I did, makes me strongly
feel like Btrfs should be able to completely recover from this, with
just a rw mount and eventually the missync'd generations will
autocorrect. But I don't know that. And I get essentially no advice
from btrfs check results.
So. What's the theory in this case? And then does it differ from reality?
Personally speaking, it may be a false alert from btrfsck.
So in this case, I can't provide much help.
If you're brave enough, mount it rw to see what will happen(although it
may mount just OK).
Thanks,
Qu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
