Recovering data from old corrupted file system
I have a corrupted multi-device file system that got corrupted ages ago (as I recall, one of the drives stopped responding, causing btrfs to panic). I am hoping to recover some of the data. For what it's worth, here is the dmesg output from trying to mount the file system on a 3.0 kernel: device label Media devid 6 transid 816153 /dev/sdq device label Media devid 7 transid 816153 /dev/sdl device label Media devid 11 transid 816153 /dev/sdj device label Media devid 9 transid 816153 /dev/sdk device label Media devid 10 transid 816153 /dev/sdi device label Media devid 3 transid 816152 /dev/sdh device label Media devid 4 transid 816152 /dev/sdf device label Media devid 1 transid 816152 /dev/sdg device label Media devid 2 transid 816152 /dev/sde device label Media devid 8 transid 816153 /dev/sdb parent transid verify failed on 47051597279232 wanted 816153 found 816151 parent transid verify failed on 47051597279232 wanted 816153 found 816151 parent transid verify failed on 47051597279232 wanted 816153 found 816151 Most of the data is unimportant at this point, and I don't have enough extra space elsewhere to store it all anyway. Is there any way I could get a read-only view of the file system to pull off the bits I need? Any help would be greatly appreciated. Thanks. -- Erik Jensen -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Honest timeline for btrfsck
Chris, Now that you're back from vacation, I was wondering if you would be able to provide a revised estimate on how long this will take. Also, of the four parts, which will be necessary to correct a 'parent transid verify failed' error? Thank you for your time, --Erik On Thu, Aug 18, 2011 at 1:50 PM, Chris Mason chris.ma...@oracle.com wrote: Excerpts from Yalonda Gishtaka's message of 2011-08-17 21:09:37 -0400: Chris Mason chris.mason at oracle.com writes: Aside from making sure the kernel code is stable, btrfsck is all I'm working on right now. I do expect a release in the next two weeks that can recover your data (and many others). Thanks, Chris -- Chris, We're all on the edge of our seats. Can you provide an updated ETA on the release of the first functional btrfsck tool? No pressure or anything ;) Hi everyone, I've been working non-stop on this. Currently fsck has four parts: 1) mount -o recovery mode. I've posted smaller forms of these patches in the past that bypass log tree replay. The new versions have code to create stub roots for trees that can't be read (like the extent allocation tree) and will allow the mount to proceed. 2) fsck that scans for older roots. This takes advantage of older copies of metadata to look for consistent tree roots on disk. The downside is that it is currently very slow. I'm trying to speed it up by limiting the search to only the metadata block groups and a few other tricks. 3) fsck that fixes the extent allocation tree and the chunk tree. This is where I've been spending most of my time. The problem is that it tends to recover some filesystems and badly break others. While I'm fixing up the corner cases that work poorly, I'm adding an undo log to the fsck code so that you can get the FS back into its original state if you don't like the result of the fsck. 4) The rest of the corruptions can be dealt with fairly well from the kernel. I have a series of patches to make the extent allocation tree less strict about reference counts and other rules, basically allowing the FS to limp along instead of crash. These four things together are basically my minimal set of features required for fedora and our own internal projects at Oracle to start treating us as production filesystem. There are always bugs to fix, and I have #1 and #2 mostly ready. I had hoped to get #1 out the door before I left on vacation and I still might post it tonight. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Honest timeline for btrfsck
The lack of any information on when btrfsck might be ready is a real headache to those deciding what to do with a corrupted file system. I am currently sitting on a btrfs array of 10 disks that has been reporting parent transid verify failed since last November. While the data on the drive is by no means irreplaceable, it would take a fair amount of effort. At the time I was told that a btrfsck would almost certainly be released by the end of the year. In January, it was finally almost ready, and toward the end of May it was going to be released in a couple of days (hopefully). Had I known back in November 9 months would go by with no such tool, I would have certainly wiped the array and started over, as it was certainly not worth the wait. So here I am, several assurances of imminent release later, still wondering whether it would be better to wait or cut my losses. I understand that everyone is working hard, and I deeply appreciate the effort being put into this filesystem. I'm not looking for an exact date, just a rough order of magnitude on which to base decisions. Thank you very much. --Erik Jensen -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Mis-Design of Btrfs?
On Wed, Jun 29, 2011 at 3:47 AM, A. James Lewis ja...@fsck.co.uk wrote: Is there a possibility that one could have a 3 disk RAID5 array, and then add a 4th disk and then do a balance, growing the RAID5 onto 4 disks and gaining the space still with RAID5? It seems that to be consistent, BTRFS would have to do this. If this is the case, then I think that the BTRFS implementation of RAID5 would have to be quite different to the MD implementation. James. My understanding, gleaned from IRC several months ago, is that Btrfs would use the new drive, but not change the stripe size. Each allocation would then be written across some selection of three of the four drives. In other words, if you started with four stripes across three drives: AAA BBB CCC DDD and then added a drive and balanced, you might get something like: AAAB BBCC CDDD which would give you more space, but still use ⅓ of the space for parity. Trying to remove a drive from the original three-drive configuration would be an error, similarly to trying to remove the second to last drive in RAID 1, currently. Actually changing the stripe size would be done using the same mechanism as changing RAID levels. Again, this is just an interested but uninvolved person's understanding based on an IRC conversation, so please salt to taste. -- Erik -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Mis-Design of Btrfs?
On Thu, Jul 14, 2011 at 12:50 PM, John Stoffel j...@stoffel.org wrote: Alasdair == Alasdair G Kergon a...@redhat.com writes: Alasdair On Thu, Jul 14, 2011 at 04:38:36PM +1000, Neil Brown wrote: It might make sense for a device to be able to report what the maximum 'N' supported is... that might make stacked raid easier to manage... Alasdair I'll just say that any solution ought to be stackable. I've been mulling this over too and wondering how you'd handle this, because upper layers really can't peak down into lower layers easily. As far as I understand things. So if you have btrfs - luks - raid1 - raid6 - nbd - remote disks How does btrfs handle errors (or does it even see them!) from the raid6 level when a single nbd device goes away? Or taking the original example, when btrfs notices a checksum isn't correct, how would it push down multiple levels to try and find the correct data? Alasdair This means understanding both that the number of data access Alasdair routes may vary as you move through the stack, and that this Alasdair number may depend on the offset within the device. It almost seems to me that the retry needs to be done at each level on it's own, without pushing down or up the stack. But this doesn't handle the wrong file checksum issue. Hmm... maybe instead of just one number, we need another to count the levels down you go (or just split 16bit integer in half, bottom half being count of tries, the upper half being levels down to try that read?) It seems to defeat the purpose of layers if you can go down and find out how many layers there are underneath you John A random thought: What if we allow the number to wrap at each level, and, each time it wraps, increment the number passed to the next lower level. A zero would propagate down, letting each level do what it wants: luks: 0 raid1: 0 raid6: 0 nbd: 0 And higher numbers would indicate the method at each level: For a 1: luks: 1 raid1: 1 raid6: 1 nbd: 1 For a 3: luks: 1 (only one possibility, passes three down) raid1: 1 (two possibilities, so we wrap back to one and pass two down, since we wrapped once) raid6: 2 (not wrapped) nbd: 1 When the bottom-most level gets an N that it can't handle, it would return EINVAL, which would be propagated up the stack. This would allow the same algorithm of incrementing N until we receive good data or EINVAL, and would exhaust all ways of reading the data at all levels. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel BUG when removing missing drive (Take 2)
So, I ended up just applying the relevant commit to my existing source tree, which did allow me to successfully remove the missing drive, so I seem to be back up and running. Thank you very much! -- Erik On Thu, Oct 28, 2010 at 1:57 PM, Chris Mason chris.ma...@oracle.com wrote: On Tue, Oct 19, 2010 at 07:17:16PM -0700, Erik Jensen wrote: One of my drives on my six drive btrfs setup recently died. I initially wasn't too worried about it, since both my data and metadata are raid1. However, I have so far not been able to remove the missing drive after several attempts. After discussing my problem on IRC, Chris Mason asked me to list everything I've tried on the mailing list, so here goes: Ok, so the current code in the scratch branch is probably going to get rebased. I've got some commits in there to add features to the bdi code, but those features are still being discussed. But, if you: git pull git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable.git scratch You'll get the scratch branch of the btrfs-unstable repo. It fixes the oops on an unwritable missing drive, which I did reproduce locally. Please let me know how this works -chris 1. I was attempting to cut commercials out of a TV recording when things seemed to stall. A look a dmesg told me that one of my drives was having many read failures. 2. I shut down my computer and removed the failed drive. 3. I booted back up and mounted the array in degraded mode. A quick ls showed all my files. 4. I checked my filesystem usage and concluded that I should have enough free space to build back up to full redundancy on the remaining drives, so I would be protected until my replacement arrived. 5. I executed btrfs-vol -r missing, which churned the hard drives for a little bit and then stalled. dmesg showed this kernel BUG: http://pastebin.com/KgjUUBq0 6. The system wouldn't reboot normally at this point, so I had to use SysRq 7. I temporarily booted a 2.6.35 kernel (I'm currently running 2.6.34) and tried to remove the missing drive again, with the same result. 8. [back on 2.6.34] My replacement drive arrived, so I installed it and added it to the btrfs pool. 9. I tried btrfs-vol -r missing again, and received the same kernel BUG once again. 10. After using SysRq to reboot, I tried doing a btrfs-vol -b, which moved some data around and halted with the same BUG. 11. I checked the kernel source to find why the bug was being thrown. The offending line was BUG_ON(rw == WRITE !dev-writeable); in btrfs_map_bio in volumes.c 12. I used badblocks -nsv to make sure of all my hard drives were writeable, which they were. A paste of all of the logged kernel messages from 8 and 9 is at http://pastebin.org/322902 I would like to get this figured out as quickly as possible, since my data is currently spread across 6 drives with (effectively) no redundancy. I do have C programming experience, so if there is a way that I can help track down the problem, please let me know. Thanks, Erik -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
kernel BUG when removing missing drive (Take 2)
One of my drives on my six drive btrfs setup recently died. I initially wasn't too worried about it, since both my data and metadata are raid1. However, I have so far not been able to remove the missing drive after several attempts. After discussing my problem on IRC, Chris Mason asked me to list everything I've tried on the mailing list, so here goes: 1. I was attempting to cut commercials out of a TV recording when things seemed to stall. A look a dmesg told me that one of my drives was having many read failures. 2. I shut down my computer and removed the failed drive. 3. I booted back up and mounted the array in degraded mode. A quick ls showed all my files. 4. I checked my filesystem usage and concluded that I should have enough free space to build back up to full redundancy on the remaining drives, so I would be protected until my replacement arrived. 5. I executed btrfs-vol -r missing, which churned the hard drives for a little bit and then stalled. dmesg showed this kernel BUG: http://pastebin.com/KgjUUBq0 6. The system wouldn't reboot normally at this point, so I had to use SysRq 7. I temporarily booted a 2.6.35 kernel (I'm currently running 2.6.34) and tried to remove the missing drive again, with the same result. 8. [back on 2.6.34] My replacement drive arrived, so I installed it and added it to the btrfs pool. 9. I tried btrfs-vol -r missing again, and received the same kernel BUG once again. 10. After using SysRq to reboot, I tried doing a btrfs-vol -b, which moved some data around and halted with the same BUG. 11. I checked the kernel source to find why the bug was being thrown. The offending line was BUG_ON(rw == WRITE !dev-writeable); in btrfs_map_bio in volumes.c 12. I used badblocks -nsv to make sure of all my hard drives were writeable, which they were. A paste of all of the logged kernel messages from 8 and 9 is at http://pastebin.org/322902 I would like to get this figured out as quickly as possible, since my data is currently spread across 6 drives with (effectively) no redundancy. I do have C programming experience, so if there is a way that I can help track down the problem, please let me know. Thanks, Erik -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Can't remove missing drive (kernel BUG)
I have an six-drive array I use for television recordings (4 2TB and 2 1.5TB). Yesterday, I had one of the 1.5TBs die on me, so I did a mount -o degraded /dev/sdb /mnt/media followed by a btrfs-vol -r missing /mnt/media, but the latter command almost immediately generated a kernel BUG message, and then hung. This happened on both 2.6.34 and 2.6.35 kernels. The relevant dmesg output is below. I'm fine with recompiling the kernel, if there is a setting you'd like me to enable to get more information. Also, the array was fairly full (about 76%), so I don't know if that might be contributing. Thank you. device label Media devid 2 transid 160727 /dev/sdb btrfs: allowing degraded mounts warning devid 5 missing btrfs: relocating block group 15629440516096 flags 17 [ cut here ] kernel BUG at fs/btrfs/volumes.c:3037! invalid opcode: [#1] SMP last sysfs file: /sys/devices/pci:00/:00:01.0/:06:00.0/boot_vga CPU 1 Modules linked in: ipv6 xt_tcpudp iptable_filter ipt_addrtype xt_dscp xt_string xt_multiport xt_hashlimit xt_conntrack xt_DSCP xt_NFQUEUE xt_mark xt_connmark nf_conntrack ip_tables x_tables snd_pcm_oss snd_mixer_oss snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device btrfs zlib_deflate crc32c libcrc32c ir_kbd_i2c cx88_dvb cx88_vp3054_i2c s5h1411 xc5000 s5h1409 tuner cx23885 firmware_class cx88_alsa arc4 ecb cx8800 cx8802 cx88xx cx2341x ath9k ath9k_common ftdi_sio ath9k_hw usbserial v4l2_common videodev videobuf_dvb dvb_core r8169 ath usblp i2c_i801 v4l2_compat_ioctl32 pcspkr mii firewire_ohci firewire_core evdev i2c_algo_bit mac80211 tveeprom videobuf_dma_sg cfg80211 videobuf_core rfkill btcx_risc snd_hda_codec_realtek led_class snd_hda_intel i2c_core button snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc fuse reiserfs scsi_wait_scan uhci_hcd usb_storage ehci_hcd sr_mod cdrom sg sata_sil24 Pid: 9011, comm: flush-btrfs-2 Not tainted 2.6.35-gentoo-r8 #2 Burbank/KT829AV-ABA d4999t-2 RIP: 0010:[a02ce6fc] [a02ce6fc] 0xa02ce6fc RSP: 0018:88019d4f3930 EFLAGS: 00010246 RAX: 0030 RBX: 88019e4862c0 RCX: 0001 RDX: 8801ae7d99c0 RSI: 8801ab22ea00 RDI: RBP: 8801acde4800 R08: 88019d47b680 R09: 4000 R10: 8801a5b6cab0 R11: 8801ae7d99c0 R12: 88019d47b680 R13: 0002 R14: R15: 0006eda1a000 FS: () GS:88000168() knlGS: CS: 0010 DS: ES: CR0: 8005003b CR2: 7f4d55a0b000 CR3: 0001a3879000 CR4: 06e0 DR0: DR1: DR2: DR3: DR6: 0ff0 DR7: 0400 Process flush-btrfs-2 (pid: 9011, threadinfo 88019d4f2000, task 8801a38d9890) Stack: 00011000 0001 8801ae7d99c0 0001 0 88019e4862c0 88019d4f3c60 0 0006eda1a080 00010fff a02c7ec4 Call Trace: [a02c7ec4] ? 0xa02c7ec4 [a02c8004] ? 0xa02c8004 [a02c9fcb] ? 0xa02c9fcb [a02cbf60] ? 0xa02cbf60 [a02c9fcb] ? 0xa02c9fcb [a02cc3dd] ? 0xa02cc3dd [a02cc5c8] ? 0xa02cc5c8 [a02b4b89] ? 0xa02b4b89 [810b5356] ? 0x810b5356 [810b58bc] ? 0x810b58bc [810b607b] ? 0x810b607b [810b621a] ? 0x810b621a [810b62d9] ? 0x810b62d9 [81037a29] ? 0x81037a29 [810b6415] ? 0x810b6415 [810409f7] ? 0x810409f7 [81084f74] ? 0x81084f74 [81084fd7] ? 0x81084fd7 [81084f74] ? 0x81084f74 [810406e1] ? 0x810406e1 [81002c54] ? 0x81002c54 [8104066c] ? 0x8104066c [81002c50] ? 0x81002c50 Code: c0 03 48 c1 e0 04 48 01 c2 48 8b 52 08 48 c1 ea 09 49 89 14 24 48 8b 54 24 10 83 7c 24 04 01 48 8b 34 10 75 0a 83 7e 64 00 75 04 0f 0b eb fe 48 85 f6 74 34 48 8b 46 70 48 85 c0 74 2b 49 89 44 RIP [a02ce6fc] 0xa02ce6fc RSP 88019d4f3930 ---[ end trace be039cec4b98333b ]--- btrfs: found 645 extents -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Unable to remove failing drive from storage pool
I currently have a btrfs filesystem set up across three drives. Unfortunately, one is failing and is returning read errors. I tried to remove the drive from the pool using btrfs-vol -r, but the process seems to hang when a read error is encountered. At this point, disk activity stops and no further progress is made, with the drive remaining in the pool. I would like to be able to remove the drive from the pool, retaining as much data as possible. (It's just TV recordings, so a little corruption shouldn't be too much of a problem). I am using version 2.6.32 of the kernel. The relevant part of the dmesg output: http://fpaste.org/jnz8/ The part of the kernel state that seemed like they might be relevant: http://fpaste.org/0KKR/ btrfs-vol is towards the bottom. The entire state dump didn't fit in my dmesg buffer so there might be something missing. Thanks, Erik Jensen -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
