Re: BTRFS disaster (of my own making). Is this recoverable?
On 2015-08-05 22:13, Chris Murphy wrote: On Wed, Aug 5, 2015 at 6:45 PM, Paul Jones p...@pauljones.id.au wrote: Would it be possible to store this type of critical information twice on each disk, at the beginning and end? I thought BTRFS already did that, but I might be thinking of some other filesystem. I've had my share of these types of oops! moments as well. That option is metadata profile raid1. To do an automatic -mconvert=raid1 when the user does 'btrfs device add' breaks any use case where you want to temporarily add a small device, maybe a USB stick, and now hundreds of MiBs possibly GiBs of metadata have to be copied over to this device without warning. It could be made smart, autoconvert to raid1 when the added device is at least 4x the size of metadata allocation, but then that makes it inconsistent. OK so it could be made interactive, but now that breaks scripts. So... where do you draw the line? Maybe this would work if the system chunk only was raid1? I don't know what the minimum necessary information is for such a case. Possibly it make more sense if 'btrfs device add' always does -dconvert=raid1 unless a --quick option is passed? Perhaps we could print out a big noisy warning that could be silenced? smime.p7s Description: S/MIME Cryptographic Signature
Re: BTRFS disaster (of my own making). Is this recoverable?
On Tue, Aug 4, 2015 at 4:23 PM, Sonic sonicsm...@gmail.com wrote: Seems that if there was someway to edit something in those first overwritten 32MB of disc 2 to say hey, I'm really here, just a bit screwed up maybe some of the recovery tools could actually work. Just want to reiterate this thought. The basic error in most cases with the tools at hand is that Disc 2 is missing so there's little the tools can do. Somewhere in those first 32MB should be something to properly identify the disc as part of the array. If the btrfs tools can't fix it maybe dd can. Is there anything can be gained from the beginning of disc 1 (can dd this to a file) in order to create the necessary bits needed at the beginning of disc2? Or some other way to overwrite the beginning of disc 2 (using dd again) with some identification information so that the automated btrfs tools can take it from there? Thanks, Chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
On Wed, Aug 5, 2015 at 8:31 AM, Sonic sonicsm...@gmail.com wrote: The basic error in most cases with the tools at hand is that Disc 2 is missing so there's little the tools can do. Somewhere in those first 32MB should be something to properly identify the disc as part of the array. Somehow manually create the missing chunk root if this is the core problem?? -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
On Wed, Aug 5, 2015 at 6:31 AM, Sonic sonicsm...@gmail.com wrote: On Tue, Aug 4, 2015 at 4:23 PM, Sonic sonicsm...@gmail.com wrote: Seems that if there was someway to edit something in those first overwritten 32MB of disc 2 to say hey, I'm really here, just a bit screwed up maybe some of the recovery tools could actually work. Just want to reiterate this thought. The basic error in most cases with the tools at hand is that Disc 2 is missing so there's little the tools can do. Somewhere in those first 32MB should be something to properly identify the disc as part of the array. Yes but it was probably uniquely only on that disk, because there's no redundancy for metadata or system chunks. Therefore there's no copy on the other disk to use as a model. The btrfs check command has an option to use other superblocks, so you could try that switch and see if it makes a difference but it sounds like it's finding backup superblocks automatically. That's the one thing that is pretty much always duplicated on the same disk; for sure the first superblock is munged and would need repair. But there's still other chunks missing... so I don't think it'll help. If the btrfs tools can't fix it maybe dd can. Is there anything can be gained from the beginning of disc 1 (can dd this to a file) in order to create the necessary bits needed at the beginning of disc2? Not if there's no metadata or system redundancy profile like raid1. Or some other way to overwrite the beginning of disc 2 (using dd again) with some identification information so that the automated btrfs tools can take it from there? I think to have a viable reference, you need two disks (virtual or real) and you need to exactly replicate how you got to this two disk setup to find out what's in those 32MB that might get the file system to mount even if it complaints of some corrupt files. That's work that's way beyond my skill level. The tools don't do this right now as far as I'm aware. You'd be making byte by byte insertions to multiple sectors. Tedious. But I can't even guess how many steps it is. It might be 10. It might be 1. -- Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
On Wed, Aug 5, 2015 at 6:45 PM, Paul Jones p...@pauljones.id.au wrote: Would it be possible to store this type of critical information twice on each disk, at the beginning and end? I thought BTRFS already did that, but I might be thinking of some other filesystem. I've had my share of these types of oops! moments as well. That option is metadata profile raid1. To do an automatic -mconvert=raid1 when the user does 'btrfs device add' breaks any use case where you want to temporarily add a small device, maybe a USB stick, and now hundreds of MiBs possibly GiBs of metadata have to be copied over to this device without warning. It could be made smart, autoconvert to raid1 when the added device is at least 4x the size of metadata allocation, but then that makes it inconsistent. OK so it could be made interactive, but now that breaks scripts. So... where do you draw the line? Maybe this would work if the system chunk only was raid1? I don't know what the minimum necessary information is for such a case. Possibly it make more sense if 'btrfs device add' always does -dconvert=raid1 unless a --quick option is passed? -- Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: BTRFS disaster (of my own making). Is this recoverable?
-Original Message-
From: linux-btrfs-ow...@vger.kernel.org [mailto:linux-btrfs-
ow...@vger.kernel.org] On Behalf Of Chris Murphy
Sent: Thursday, 6 August 2015 2:54 AM
To: Sonic sonicsm...@gmail.com
Cc: Btrfs BTRFS linux-btrfs@vger.kernel.org; Hugo Mills
h...@carfax.org.uk
Subject: Re: BTRFS disaster (of my own making). Is this recoverable?
On Wed, Aug 5, 2015 at 6:31 AM, Sonic sonicsm...@gmail.com wrote:
On Tue, Aug 4, 2015 at 4:23 PM, Sonic sonicsm...@gmail.com wrote:
Seems that if there was someway to edit something in those first
overwritten 32MB of disc 2 to say hey, I'm really here, just a bit
screwed up maybe some of the recovery tools could actually work.
Just want to reiterate this thought.
The basic error in most cases with the tools at hand is that Disc 2 is
missing so there's little the tools can do. Somewhere in those first
32MB should be something to properly identify the disc as part of the
array.
Yes but it was probably uniquely only on that disk, because there's no
redundancy for metadata or system chunks. Therefore there's no copy on
the other disk to use as a model. The btrfs check command has an option to
use other superblocks, so you could try that switch and see if it makes a
difference but it sounds like it's finding backup superblocks automatically.
That's the one thing that is pretty much always duplicated on the same disk;
for sure the first superblock is munged and would need repair. But there's
still other chunks missing... so I don't think it'll help.
Would it be possible to store this type of critical information twice on each
disk, at the beginning and end? I thought BTRFS already did that, but I might
be thinking of some other filesystem. I've had my share of these types of oops!
moments as well.
Paul.
N�r��yb�X��ǧv�^�){.n�+{�n�߲)w* jg��� �ݢj/���z�ޖ��2�ޙ�)ߡ�a���� �G���h��j:+v���w��٥
Re: BTRFS disaster (of my own making). Is this recoverable?
On Tue, Aug 4, 2015 at 1:10 AM, Duncan 1i5t5.dun...@cox.net wrote: I don't remember you saying anything about trying btrfs restore. I did try it earlier in dry-run mode and it can't do anything: = btrfs restore -D /dev/sdc /mnt/saved/ warning, device 2 is missing bytenr mismatch, want=20971520, have=0 Couldn't read chunk root Could not open root, trying backup super warning, device 2 is missing bytenr mismatch, want=20971520, have=0 Couldn't read chunk root Could not open root, trying backup super warning, device 2 is missing bytenr mismatch, want=20971520, have=0 Couldn't read chunk root Could not open root, trying backup super = Find-roots, list-roots, none of it works. = btrfs restore --list-roots /dev/sde checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 bytenr mismatch, want=20971520, have=8330001001141004672 Couldn't read chunk root Could not open root, trying backup super warning, device 1 is missing checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 bytenr mismatch, want=20971520, have=8330001001141004672 Couldn't read chunk root Could not open root, trying backup super warning, device 1 is missing checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 bytenr mismatch, want=20971520, have=8330001001141004672 Couldn't read chunk root Could not open root, trying backup super = btrfs-find-root -a /dev/sdc warning, device 2 is missing Couldn't read chunk root btrfs-find-root -a /dev/sde Couldn't read chunk root Open ctree failed = In defense of BTRFS it's been absolutely trouble free keeping my data through many power outages, etc. and I plan to keep using it. It was my own doing that caused the problem. Seems that if there was someway to edit something in those first overwritten 32MB of disc 2 to say hey, I'm really here, just a bit screwed up maybe some of the recovery tools could actually work. Thanks to all. Chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
On Mon, Aug 03, 2015 at 11:05:46AM -0400, Sonic wrote: On Mon, Aug 3, 2015 at 10:43 AM, Sonic sonicsm...@gmail.com wrote: Is btrfs rescue super-recover safe to run? IOW, will it ask before doing anything possibly destructive (assuming I don't give it a -y)? Seemed a bit safe so I went for it and: sartre ~ # btrfs rescue super-recover /dev/sdc All supers are valid, no need to recover sartre ~ # btrfs rescue super-recover /dev/sde All supers are valid, no need to recover So it may not be a superblock issue. From the dmesg earlier: [ 3421.193734] BTRFS (device sde): bad tree block start 8330001001141004672 20971520 [ 3421.193738] BTRFS: failed to read chunk root on sde [ 3421.203221] BTRFS: open_ctree failed I may need a chunk-recover and also wonder if zero-log is advisable. Any ideas in those directions? Very unlikely and definitely not, respectively. There's nothing at all here to indicate that you've got a broken log, so dropping it would be at best pointless. The chunk tree is also most likely undamaged on both copies. Hugo. -- Hugo Mills | The early bird gets the worm, but the second mouse hugo@... carfax.org.uk | gets the cheese. http://carfax.org.uk/ | PGP: E2AB1DE4 | signature.asc Description: Digital signature
Re: BTRFS disaster (of my own making). Is this recoverable?
On Mon, Aug 3, 2015 at 1:17 AM, Duncan 1i5t5.dun...@cox.net wrote: The first thing you need to do in terms of trying to recover, is restore the superblock on the damaged device. Since btrfs keeps multiple copies (up to three, once the filesystem is large enough, as yours is) per device, that's actually relatively easy. Use... btrfs rescue super-recover Not sure how to tell if there is a superblock issue: === btrfs-show-super -f /dev/sdc superblock: bytenr=65536, device=/dev/sdc - dev_item.type 0 dev_item.total_bytes4000787030016 dev_item.bytes_used 3527267057664 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) chunk length 16777216 owner 2 type SYSTEM|RAID0 num_stripes 2 stripe 0 devid 2 offset 1048576 stripe 1 devid 1 offset 20971520 backup_roots[4]: backup 0: backup_tree_root: 1517037699072 gen: 9025 level: 1 backup_chunk_root: 20971520gen: 8957 level: 1 backup_extent_root: 576585728 gen: 9025 level: 2 backup_fs_root: 2056568832 gen: 1106 level: 0 backup_dev_root:52576256gen: 9021 level: 1 backup_csum_root: 1517028753408 gen: 9025 level: 3 backup_total_bytes: 8001574060032 backup_bytes_used: 7038625824768 backup_num_devices: 2 backup 1: backup_tree_root: 1517167755264 gen: 9026 level: 1 backup_chunk_root: 20971520gen: 8957 level: 1 backup_extent_root: 1517167771648 gen: 9026 level: 2 backup_fs_root: 2056568832 gen: 1106 level: 0 backup_dev_root:52576256gen: 9021 level: 1 backup_csum_root: 2503711637504 gen: 9026 level: 3 backup_total_bytes: 8001574060032 backup_bytes_used: 7038625824768 backup_num_devices: 2 backup 2: backup_tree_root: 980877312 gen: 9023 level: 1 backup_chunk_root: 20971520gen: 8957 level: 1 backup_extent_root: 1026768896 gen: 9023 level: 2 backup_fs_root: 2056568832 gen: 1106 level: 0 backup_dev_root:52576256gen: 9021 level: 1 backup_csum_root: 1790377984 gen: 9023 level: 3 backup_total_bytes: 8001574060032 backup_bytes_used: 7038617616384 backup_num_devices: 2 backup 3: backup_tree_root: 1960509440 gen: 9024 level: 1 backup_chunk_root: 20971520gen: 8957 level: 1 backup_extent_root: 1960525824 gen: 9024 level: 2 backup_fs_root: 2056568832 gen: 1106 level: 0 backup_dev_root:52576256gen: 9021 level: 1 backup_csum_root: 2106736640 gen: 9024 level: 3 backup_total_bytes: 8001574060032 backup_bytes_used: 7038617616384 backup_num_devices: 2 btrfs-show-super -f /dev/sde superblock: bytenr=65536, device=/dev/sde - csum0x9634c164 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid09024c28-7932-4ddb-960d-becc1ea839e5 label terrafirm generation 9026 root1517167755264 sys_array_size 129 chunk_root_generation 8957 root_level 1 chunk_root 20971520 chunk_root_level1 log_root0 log_root_transid0 log_root_level 0 total_bytes 8001574060032 bytes_used 7038625824768 sectorsize 4096 nodesize16384 leafsize16384 stripesize 4096 root_dir6 num_devices 2 compat_flags0x0 compat_ro_flags 0x0 incompat_flags 0x21 ( MIXED_BACKREF | BIG_METADATA ) csum_type 0 csum_size 4 cache_generation9026
Re: BTRFS disaster (of my own making). Is this recoverable?
On Mon, Aug 3, 2015 at 11:28 AM, Hugo Mills h...@carfax.org.uk wrote: Very unlikely and definitely not, respectively. There's nothing at all here to indicate that you've got a broken log, so dropping it would be at best pointless. The chunk tree is also most likely undamaged on both copies. Will it hurt to try a chunk-recover? I'm guessing it wont do anything unless I answer Y to some prompt. Is there anything else you would suggest? Thanks, Chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
Also tried: mount -o recovery,ro /mnt/butter/ And dmesg gives: [88228.756622] BTRFS info (device sde): enabling auto recovery [88228.756635] BTRFS info (device sde): disk space caching is enabled [88228.757244] BTRFS (device sde): bad tree block start 8330001001141004672 20971520 [88228.757248] BTRFS: failed to read chunk root on sde [88228.769877] BTRFS: open_ctree failed On Mon, Aug 3, 2015 at 12:03 PM, Sonic sonicsm...@gmail.com wrote: On Mon, Aug 3, 2015 at 11:28 AM, Hugo Mills h...@carfax.org.uk wrote: Very unlikely and definitely not, respectively. There's nothing at all here to indicate that you've got a broken log, so dropping it would be at best pointless. The chunk tree is also most likely undamaged on both copies. Will it hurt to try a chunk-recover? I'm guessing it wont do anything unless I answer Y to some prompt. Is there anything else you would suggest? Thanks, Chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
Output of btrfs check: btrfs check /dev/sdc warning, device 2 is missing bytenr mismatch, want=20971520, have=0 Couldn't read chunk root Couldn't open file system btrfs check /dev/sde checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 bytenr mismatch, want=20971520, have=8330001001141004672 Couldn't read chunk root Couldn't open file system In case the above helps. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
On Mon, Aug 3, 2015 at 12:32 PM, Sonic sonicsm...@gmail.com wrote: btrfs rescue chunk-recover Ran this: btrfs rescue chunk-recover -v /dev/sde |tee brcr.txt Got this (very end of output): == Unrecoverable Chunks: Total Chunks: 3292 Recoverable: 3292 Unrecoverable:0 Orphan Block Groups: Orphan Device Extents: Fail to recover the chunk tree. == If earlier output of this process might be helpful I can provide it (used tee to create a file). Didn't work - seems all chunks found were recoverable - it's apparently the missing chunks that are the problem. I'm thinking this is a lost cause. Also thinking I need to provide for some redundancy in the future :-) Any other suggestions before I give up the ghost on this? Thanks, Chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
Out of frustration and vodka I tried: btrfs check --repair /dev/sde To instantly be met with: enabling repair mode checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 bytenr mismatch, want=20971520, have=8330001001141004672 Couldn't read chunk root Couldn't open file system On Mon, Aug 3, 2015 at 10:20 PM, Sonic sonicsm...@gmail.com wrote: On Mon, Aug 3, 2015 at 12:32 PM, Sonic sonicsm...@gmail.com wrote: btrfs rescue chunk-recover Ran this: btrfs rescue chunk-recover -v /dev/sde |tee brcr.txt Got this (very end of output): == Unrecoverable Chunks: Total Chunks: 3292 Recoverable: 3292 Unrecoverable:0 Orphan Block Groups: Orphan Device Extents: Fail to recover the chunk tree. == If earlier output of this process might be helpful I can provide it (used tee to create a file). Didn't work - seems all chunks found were recoverable - it's apparently the missing chunks that are the problem. I'm thinking this is a lost cause. Also thinking I need to provide for some redundancy in the future :-) Any other suggestions before I give up the ghost on this? Thanks, Chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
Sonic posted on Mon, 03 Aug 2015 22:20:38 -0400 as excerpted: I'm thinking this is a lost cause. Also thinking I need to provide for some redundancy in the future :-) Any other suggestions before I give up the ghost on this? I don't remember you saying anything about trying btrfs restore. I'm not sure it can do anything for this case, and it definitely requires enough additional space on some other filesystem to store the data it restores, but if you had stuff on there you'd really like to get back, it's worth a shot, and if you're going to be backing up in the future, you'll need the additional devices for the backup in any case, so might as well get 'em now, mkfs them to whatever (doesn't have to be btrfs since all you're doing is using it as a place to store the files, it's the one you're restoring /from/ that's btrfs), mount 'em, and try restoring to them. Worst-case, it doesn't work and you have your set of backup devices already prepped and ready to go. Best-case, it recovers pretty much everything. But I am guessing restore won't be able to do anything on it's own, you'll need to try btrfs-find-root, and feed the bytenr from roots it finds into restore until you get something useful. Take a look at the link and brief instructions I posted in my first reply. Of course, if btrfs-find-root doesn't show you any roots (these are the filesystem master roots), or if fed those roots using -t bytenr, btrfs restore --list-roots (these are the various subtree roots) doesn't give you anything worthwhile, then it /would/ appear you're looking at a lost- cause... unless anyone else has a bright idea at this point. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
On Sun, Aug 02, 2015 at 11:42:25PM -0600, Chris Murphy wrote: I can't tell what the data and metadata profile are? That it won't mount degraded makes me think the metadata is not explicitly raid1; and it either raid0 or accidentally single or dup which can happen at mkfs time on single device, and just doing btrfs dev add to add another device. I think recovery is difficult but it depends on what sort of critical information is in those first 32MB other than the superblock. There are copies of the superblock so that can probably be reconstructed. It's probably that the small empty single chunks left behind by mkfs are still there. I don't think we have a good solution to this yet (other than fixing mkfs so it doesn't happen in the first place). Hugo. -- Hugo Mills | Our so-called leaders speak hugo@... carfax.org.uk | with words they try to jail ya http://carfax.org.uk/ | They subjugate the meek PGP: E2AB1DE4 | but it's the rhetoric of failure. The Police signature.asc Description: Digital signature
Re: BTRFS disaster (of my own making). Is this recoverable?
Are either one of these called for? btrfs check --repair or btrfs check --repair --init-csum-tree Seems like they might be a last ditch attempt. Is one preferred over the other? Is: btrfs rescue chunk-recover a much less dangerous attempt (IOW it wont hurt to try it first)? Thanks, Chris On Mon, Aug 3, 2015 at 12:22 PM, Sonic sonicsm...@gmail.com wrote: Output of btrfs check: btrfs check /dev/sdc warning, device 2 is missing bytenr mismatch, want=20971520, have=0 Couldn't read chunk root Couldn't open file system btrfs check /dev/sde checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238 bytenr mismatch, want=20971520, have=8330001001141004672 Couldn't read chunk root Couldn't open file system In case the above helps. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
Sonic posted on Mon, 03 Aug 2015 12:32:21 -0400 as excerpted: Are either one of these called for? btrfs check --repair or btrfs check --repair --init-csum-tree Seems like they might be a last ditch attempt. Is one preferred over the other? The read-only check (without --repair) couldn't read the chunk tree, so check --repair probably should run into the same issue and not be able to do anything as well. check --init-csum-tree reinitializes the checksums, but to do that you have to have something to reinitialize on, so that's unlikely to do anything, either. (I did have that in mind as a possibility after the superblock recovery and chunk-recover operations, tho, if necessary. That's why I mentioned check at the end. But the message was long enough already and that would have been getting ahead of things, so...) Is: btrfs rescue chunk-recover a much less dangerous attempt (IOW it wont hurt to try it first)? I'd not call it less dangerous, but given that the chunk tree can't be read, you're already dealing with dangerous. That's what I'd try here (tho Hugo's suggesting it won't help has me doubting, but that's where things do point from what I see). Which is why I mentioned doing a raw device backup, before attempting it. Chunk-recover is pretty invasive, and if it doesn't work, it could make the filesystem unrecoverable. It's basically a one-shot deal. A raw device backup can be used to avoid it being one-shot, but of course that does require that you have enough devices of sufficient size to backup to, which certainly in the terabyte range, is a potential problem unless you're a big-budget corporate user with a stack of spare devices on hand. I believe Hugo's right on zero-log, tho. That's a fix for a very specific situation, seen most often due to a specific bug that was short lived and has been fixed for quite some time, now, tho in certain very narrow post-crash situations the same fix has been known to work too. There's no evidence you're anywhere near that very specific situation, so zero-log's just not the tool for this job. And it could make things slightly worse, too, tho in theory at least, all you're doing is cutting off the last 30 seconds or so of activity, so the chances of it doing major harm are small, unless you were actively rebalancing or something when the filesystem was last unmounted (gracefully or not). -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
BTRFS disaster (of my own making). Is this recoverable?
I have had bad dreams about this particular fat finger but after a few years it has finally happened. Scenario: 2 drives in a raw btrfs array (no previous partitions and non-redundant) with various subvols as well. One was sdc the other was sde, although sde never displays with mount command and blkid is the same for both. Thinking I was writing to a flash drive I sent 32MB via dd === dd if=file of=/dev/sde === to sde (instead of what I wanted - sdf) and now the volume nor any of it's subvol's can mount (of course that seems entirely reasonable, although you can imagine how unhappy I am). With: === mount -t btrfs /mnt/butter/ === I get: === [ 3421.193103] BTRFS info (device sde): disk space caching is enabled [ 3421.193734] BTRFS (device sde): bad tree block start 8330001001141004672 20971520 [ 3421.193738] BTRFS: failed to read chunk root on sde [ 3421.203221] BTRFS: open_ctree failed === If I specify /dev/sdc instead of relying on fstab I get: === mount -t btrfs -o degraded /dev/sdc /mnt/butter/ === [ 3839.506766] BTRFS info (device sde): allowing degraded mounts [ 3839.506769] BTRFS info (device sde): disk space caching is enabled [ 3839.507154] BTRFS (device sde): bad tree block start 8330001001141004672 20971520 [ 3839.507159] BTRFS: failed to read chunk root on sde [ 3839.515023] BTRFS: open_ctree failed === Obligatory details: === uname -a Linux sartre 4.1.3-gentoo #1 SMP Sat Jul 25 22:34:14 EDT 2015 x86_64 Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz GenuineIntel GNU/Linux === btrfs --version btrfs-progs v4.1.2 === btrfs fi show Label: 'garage' uuid: 896f615e-6437-41c6-8f2a-25f73ff1af7a Total devices 1 FS bytes used 89.33GiB devid1 size 200.00GiB used 92.02GiB path /dev/sdb3 warning, device 2 is missing bytenr mismatch, want=20971520, have=0 Couldn't read chunk root Label: 'terrafirm' uuid: 09024c28-7932-4ddb-960d-becc1ea839e5 Total devices 2 FS bytes used 6.40TiB devid1 size 3.64TiB used 3.21TiB path /dev/sdc *** Some devices missing btrfs-progs v4.1.2 === btrfs fi df /mnt/butter ERROR: couldn't get space info - Inappropriate ioctl for device ERROR: get_df failed Inappropriate ioctl for device === Please advise. Thank you, Chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
Sonic posted on Sun, 02 Aug 2015 13:13:41 -0400 as excerpted: I have had bad dreams about this particular fat finger but after a few years it has finally happened. Scenario: 2 drives in a raw btrfs array (no previous partitions and non-redundant) with various subvols as well. One was sdc the other was sde, although sde never displays with mount command and blkid is the same for both. Thinking I was writing to a flash drive I sent 32MB via dd === dd if=file of=/dev/sde === to sde (instead of what I wanted - sdf) and now the volume nor any of it's subvol's can mount (of course that seems entirely reasonable, although you can imagine how unhappy I am). [...] Obligatory details: === uname -a Linux sartre 4.1.3-gentoo [...] === btrfs --version btrfs-progs v4.1.2 === btrfs fi show [...] warning, device 2 is missing bytenr mismatch, want=20971520, have=0 Couldn't read chunk root Label: 'terrafirm' [...] Total devices 2 FS bytes used 6.40TiB devid1 size 3.64TiB used 3.21TiB path /dev/sdc *** Some devices missing [...] Of course the first thing is sysadmin's backups rule -- if you don't have it backed up, by definition and action, that data is worth less to you than the time/effort/media to back it up, any claims to the contrary not withstanding, as your actions spoke louder than your words. And of course btrfs isn't fully stable yet, so that rule applies double, compared to a fully mature and stable filesystem. Therefore, you're in luck. Either you didn't lose anything of value because you had it backed up, or it wasn't of /that/ much value to you anyway, so again, you didn't lose anything of value. =:^) (And actually, that's truer than you might think at the moment. Consider my next-door neighbor, who a couple weeks ago escaped a fire with his life and the pair of shorts he was wearing. No wallet, no keys, nothing. I don't think he's going to be worrying much about that last fat-fingering he did on the computer for awhile! If you have your life, your health, a roof over your head, and either food on the kitchen shelves or money to eat out with, you're a lucky one! They say everybody fat-fingers at some point and has to learn the value of those backups the hard way. Now you have that behind you. =:^) I've been there too, and a bit of perspective really does help.) Beyond that... You're running current kernel and userspace/progs already, good job there. =:^) The first thing you need to do in terms of trying to recover, is restore the superblock on the damaged device. Since btrfs keeps multiple copies (up to three, once the filesystem is large enough, as yours is) per device, that's actually relatively easy. Use... btrfs rescue super-recover ... for that. Being a gentooer (as am I), I'm confident you can read the manpage for the details. =:^) You can use... btrfs-show-super ... to examine the various superblock copies in ordered to pick a good one to restore. Beyond that, as a simple btrfs user and list regular who hasn't had that particular problem here (my brown-bag day was a different fat-fingering! ), not a dev, I'm a bit beyond my depth, but were it me, at that point I'd see what I had, then try... btrfs restore There's a page on the wiki with details on how to try to get it to work manually, if it's not going to work automatically for you: https://btrfs.wiki.kernel.org/index.php/Restore I've used it here (tho not in the situation you're in) when I had backups, but they were a bit stale. Basically, I valued the data enough to have backups, but valued saving the hassle over keeping them current, given the amount of actually changed data. Still, since restore worked for me and thus gave me the choice of somewhat stale backups, or rather less stale btrfs restore, I took less stale. =:^) The nice thing about restore, other than that it often works when the filesystem won't mount, is that it doesn't write to the damaged filesystem at all, so there's no chance of making the problem worse instead of better. Restore simply reads what files it can get, restoring them to some other (working and mounted) filesystem. So you'll need some alternate place to store all those files... Tho the wiki is somewhat behind current code, so read the manpage for more useful options. In particular, the dry-run option can give you a good idea whether you're going to get much of anything useful with that generation/transid, or not, and there's now options to restore metadata (ownership and permissions) and symlinks, as well as the file data itself. Also, where it talks about picking a tree root with as many of the filesystem trees as possible, use the list-roots option. This is what displays all those trees it's talking about. If restore worked for you, you have a choice.
Re: BTRFS disaster (of my own making). Is this recoverable?
I can't tell what the data and metadata profile are? That it won't mount degraded makes me think the metadata is not explicitly raid1; and it either raid0 or accidentally single or dup which can happen at mkfs time on single device, and just doing btrfs dev add to add another device. I think recovery is difficult but it depends on what sort of critical information is in those first 32MB other than the superblock. There are copies of the superblock so that can probably be reconstructed. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
