Re: kernel BUG at fs/btrfs/extent-tree.c:8113! (4.1.3 kernel)
On Fri, Jul 24, 2015 at 09:24:46AM -0700, Marc MERLIN wrote: Screenshot: http://marc.merlins.org/tmp/btrfs_crash.jpg So it's 32bit system, 3.19.8, crashing during snapshot deletion and backref walking. EIP is in do_walk_down+0x142. I've tried to match it to the sources on a local 32bit build, but it does not point to the expected crash site: Thanks for looking. Unfortunately it's a mythtv where if I put a 64bit kernel, other things go wrong with the 32bit userland/64bit kernel split. But I'll put a newer 64bit kernel on it to see what happens and report back. I got home, built the last kernel and got netconsole working. 4.1.3/64bit and 32bit crash the same way. Here's the 64bit crash: [ 209.647162] BTRFS: device label bigbackup devid 1 transid 39536 /dev/mapper/crypt_sdd1 [ 209.647449] BTRFS: device label bigbackup devid 5 transid 39536 /dev/mapper/crypt_sdh1 [ 209.648069] BTRFS: device label bigbackup devid 4 transid 39536 /dev/mapper/crypt_sdg1 [ 209.648469] BTRFS: device label bigbackup devid 2 transid 39536 /dev/mapper/crypt_sde1 [ 209.648871] BTRFS: device label bigbackup devid 3 transid 39536 /dev/mapper/crypt_sdf1 [ 209.675030] BTRFS info (device dm-0): disk space caching is enabled [ 249.865515] [ cut here ] [ 249.865530] WARNING: CPU: 1 PID: 3556 at fs/btrfs/extent-tree.c:863 btrfs_lookup_extent_info+0x292/0x2e0() [ 249.865534] Modules linked in: xts gf128mul configs rc_hauppauge ir_kbd_i2c cpufreq_userspace cpufreq_powersave cpufreq_conservative cpufreq_stats autofs4 joydev hid_generic usbhid tuner_simple tuner_types tda9887 tda8290 hid tuner msp3400 firewire_sbp2 snd_hda_codec_hdmi rc_imon_mce saa7127 snd_hda_codec_realtek snd_hda_codec_generic hwmon_vid dm_crypt snd_hda_intel dm_mod snd_hda_controller imon saa7115 snd_hda_codec snd_hda_core coretemp snd_pcm_oss snd_mixer_oss ivtv bttv cx2341x tea575x v4l2_common snd_pcm videodev snd_hwdep snd_seq_midi videobuf_dma_sg videobuf_core rc_core snd_rawmidi snd_seq_midi_event snd_seq gpio_ich kvm_intel ehci_pci sr_mod media firewire_ohci cdrom floppy snd_timer ehci_hcd uhci_hcd psmouse snd_seq_device tveeprom acpi_cpufreq lpc_ich usbcore sg lp firewire_core asus_atk0110 evdev usb_common kvm atl1 serio_raw mii crc_itu_t parport snd soundcore processor microcode 8250_fintek [ 249.865672] CPU: 1 PID: 3556 Comm: btrfs-cleaner Not tainted 4.1.3-amd64-i915-volpreempt-20150421 #1 [ 249.865674] Hardware name: System manufacturer P5E-VM HDMI/P5E-VM HDMI, BIOS 060407/16/2008 [ 249.865676] 0009 880041e03b68 8169d72d 4fb9 [ 249.865683] 880041e03ba8 8105861f 0094 [ 249.865691] 81239ac5 8800772920b0 [ 249.865700] Call Trace: [ 249.865706] [8169d72d] dump_stack+0x45/0x57 [ 249.865711] [8105861f] warn_slowpath_common+0xa1/0xbb [ 249.865714] [81239ac5] ? btrfs_lookup_extent_info+0x292/0x2e0 [ 249.865717] [810586dc] warn_slowpath_null+0x1a/0x1c [ 249.865720] [81239ac5] btrfs_lookup_extent_info+0x292/0x2e0 [ 249.865724] [816a3752] ? _raw_write_lock+0xe/0x10 [ 249.865727] [8123c4af] do_walk_down+0x14d/0x735 [ 249.865731] [8123cb1e] walk_down_tree+0x87/0xb0 [ 249.865734] [8123f617] btrfs_drop_snapshot+0x2e7/0x696 [ 249.865739] [8124ee09] btrfs_clean_one_deleted_snapshot+0xce/0xdb [ 249.865742] [81248452] cleaner_kthread+0x112/0x146 [ 249.865745] [81248340] ? atomic_add_unless.constprop.56+0x24/0x24 [ 249.865748] [81248340] ? atomic_add_unless.constprop.56+0x24/0x24 [ 249.865751] [810706f6] kthread+0xae/0xb6 [ 249.865755] [81070648] ? __kthread_parkme+0x61/0x61 [ 249.865758] [816a3e62] ret_from_fork+0x42/0x70 [ 249.865761] [81070648] ? __kthread_parkme+0x61/0x61 [ 249.865764] ---[ end trace 826326bc6da53e4f ]--- [ 249.865767] BTRFS error (device dm-0): Missing references. [ 249.865781] [ cut here ] [ 249.867106] kernel BUG at fs/btrfs/extent-tree.c:8113! [ 249.868435] invalid opcode: [#1] SMP [ 249.869508] Modules linked in: xts gf128mul configs rc_hauppauge ir_kbd_i2c cpufreq_userspace cpufreq_powersave cpufreq_conservative cpufreq_stats autofs4 joydev hid_generic usbhid tuner_simple tuner_types tda9887 tda8290 hid tuner msp3400 firewire_sbp2 snd_hda_codec_hdmi rc_imon_mce saa7127 snd_hda_codec_realtek snd_hda_codec_generic hwmon_vid dm_crypt snd_hda_intel dm_mod snd_hda_controller imon saa7115 snd_hda_codec snd_hda_core coretemp snd_pcm_oss snd_mixer_oss ivtv bttv cx2341x tea575x v4l2_common snd_pcm videodev snd_hwdep snd_seq_midi videobuf_dma_sg videobuf_core rc_core snd_rawmidi snd_seq_midi_event snd_seq gpio_ich kvm_intel ehci_pci sr_mod media firewire_ohci cdrom floppy snd_timer ehci_hcd uhci_hcd psmouse snd_seq_device tveeprom
Re: Data single *and* raid?
Hello, Looking at the btrfs fi show output, you've probably run out of space during the conversion, probably due to an uneven distribution of the original single chunks. I think I would suggest balancing the single chunks, and trying the conversion (of the unconverted parts) again: # btrfs balance start -dprofiles=single -mprofile=raid1 /mnt/new_storage/ # btrfs balance start -dconvert=raid5,soft -mconvert=raid5,soft /mnt/new_storage/ Yep I bet that's it also. btrfs fi usage might be better at exposing this case. Thanks for your hints. The balance is running now for about 11h: The status is a bit surprising to me: 0 out of about 2619 chunks balanced (4165 considered), 100% left btrfs fi usage is also surprising: Overall: Device size: 8.19TiB Device allocated: 2.56TiB Device unallocated:5.62TiB Device missing: 0.00B Used: 2.56TiB Free (estimated): 11.65TiB (min: 2.81TiB) Data ratio: 0.48 Metadata ratio: 1.33 Global reserve: 512.00MiB (used: 0.00B) Data,single: Size:2.55TiB, Used:2.55TiB /dev/sdc1.60TiB /dev/sde 975.44GiB Data,RAID5: Size:2.73TiB, Used:2.72TiB /dev/sdc1.12TiB /dev/sdd2.73TiB /dev/sde1.61TiB Metadata,RAID1: Size:6.00GiB, Used:5.33GiB /dev/sdc5.00GiB /dev/sdd1.00GiB /dev/sde6.00GiB Metadata,RAID5: Size:3.00GiB, Used:2.99GiB /dev/sdc2.00GiB /dev/sdd1.00GiB /dev/sde3.00GiB System,RAID5: Size:32.00MiB, Used:736.00KiB /dev/sdd 32.00MiB /dev/sde 32.00MiB Unallocated: /dev/sdc1.02MiB /dev/sdd1.02MiB /dev/sde 164.59GiB I hope, that is because Raid5 is not implemented yet? Regards, Hendrik --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
BTRFS disaster (of my own making). Is this recoverable?
I have had bad dreams about this particular fat finger but after a few years it has finally happened. Scenario: 2 drives in a raw btrfs array (no previous partitions and non-redundant) with various subvols as well. One was sdc the other was sde, although sde never displays with mount command and blkid is the same for both. Thinking I was writing to a flash drive I sent 32MB via dd === dd if=file of=/dev/sde === to sde (instead of what I wanted - sdf) and now the volume nor any of it's subvol's can mount (of course that seems entirely reasonable, although you can imagine how unhappy I am). With: === mount -t btrfs /mnt/butter/ === I get: === [ 3421.193103] BTRFS info (device sde): disk space caching is enabled [ 3421.193734] BTRFS (device sde): bad tree block start 8330001001141004672 20971520 [ 3421.193738] BTRFS: failed to read chunk root on sde [ 3421.203221] BTRFS: open_ctree failed === If I specify /dev/sdc instead of relying on fstab I get: === mount -t btrfs -o degraded /dev/sdc /mnt/butter/ === [ 3839.506766] BTRFS info (device sde): allowing degraded mounts [ 3839.506769] BTRFS info (device sde): disk space caching is enabled [ 3839.507154] BTRFS (device sde): bad tree block start 8330001001141004672 20971520 [ 3839.507159] BTRFS: failed to read chunk root on sde [ 3839.515023] BTRFS: open_ctree failed === Obligatory details: === uname -a Linux sartre 4.1.3-gentoo #1 SMP Sat Jul 25 22:34:14 EDT 2015 x86_64 Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz GenuineIntel GNU/Linux === btrfs --version btrfs-progs v4.1.2 === btrfs fi show Label: 'garage' uuid: 896f615e-6437-41c6-8f2a-25f73ff1af7a Total devices 1 FS bytes used 89.33GiB devid1 size 200.00GiB used 92.02GiB path /dev/sdb3 warning, device 2 is missing bytenr mismatch, want=20971520, have=0 Couldn't read chunk root Label: 'terrafirm' uuid: 09024c28-7932-4ddb-960d-becc1ea839e5 Total devices 2 FS bytes used 6.40TiB devid1 size 3.64TiB used 3.21TiB path /dev/sdc *** Some devices missing btrfs-progs v4.1.2 === btrfs fi df /mnt/butter ERROR: couldn't get space info - Inappropriate ioctl for device ERROR: get_df failed Inappropriate ioctl for device === Please advise. Thank you, Chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Data single *and* raid?
On Sun, Aug 02, 2015 at 12:31:13PM -0600, Chris Murphy wrote: On Sat, Aug 1, 2015 at 9:46 PM, Duncan 1i5t5.dun...@cox.net wrote: If it was setup with something earlier (not sure about 4.1.0, was it affected? No. but 4.0.x and earlier should be fine for setup), however, once on a new kernel the usual ENOSPC workarounds can be given a try. That would include a first balance start -dusage=0 -musage=0, and if that didn't free up at least a gig on a second device, If I'm following this correctly, the reproduce steps would be to create a single device Btrfs that's ~94% full, add two devices, then convert to raid5. I think what's going on here is empty single profile data chunks aren't being deallocated, and it's effectively a 2 device raid5. This isn't supported by the btrfs fi df output: all of the allocated space is used. Hugo. So actually, you're right, either -dusage=0 might fix it, or better, newer kernel that automatically deallocated empty/converted single profile data chunks. But right now it will take another balance in the end because it looks like this is effectively a 2 device raid5, with the 3rd drive full of single only chunks (which might be empty?). -- Hugo Mills | I'll take your bet, but make it ten thousand francs. hugo@... carfax.org.uk | I'm only a _poor_ corrupt official. http://carfax.org.uk/ | PGP: E2AB1DE4 | Capt. Renaud, Casablanca signature.asc Description: Digital signature
Re: Data single *and* raid?
On Sat, Aug 1, 2015 at 9:46 PM, Duncan 1i5t5.dun...@cox.net wrote: If it was setup with something earlier (not sure about 4.1.0, was it affected? No. but 4.0.x and earlier should be fine for setup), however, once on a new kernel the usual ENOSPC workarounds can be given a try. That would include a first balance start -dusage=0 -musage=0, and if that didn't free up at least a gig on a second device, If I'm following this correctly, the reproduce steps would be to create a single device Btrfs that's ~94% full, add two devices, then convert to raid5. I think what's going on here is empty single profile data chunks aren't being deallocated, and it's effectively a 2 device raid5. So actually, you're right, either -dusage=0 might fix it, or better, newer kernel that automatically deallocated empty/converted single profile data chunks. But right now it will take another balance in the end because it looks like this is effectively a 2 device raid5, with the 3rd drive full of single only chunks (which might be empty?). -- Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS disaster (of my own making). Is this recoverable?
Sonic posted on Sun, 02 Aug 2015 13:13:41 -0400 as excerpted: I have had bad dreams about this particular fat finger but after a few years it has finally happened. Scenario: 2 drives in a raw btrfs array (no previous partitions and non-redundant) with various subvols as well. One was sdc the other was sde, although sde never displays with mount command and blkid is the same for both. Thinking I was writing to a flash drive I sent 32MB via dd === dd if=file of=/dev/sde === to sde (instead of what I wanted - sdf) and now the volume nor any of it's subvol's can mount (of course that seems entirely reasonable, although you can imagine how unhappy I am). [...] Obligatory details: === uname -a Linux sartre 4.1.3-gentoo [...] === btrfs --version btrfs-progs v4.1.2 === btrfs fi show [...] warning, device 2 is missing bytenr mismatch, want=20971520, have=0 Couldn't read chunk root Label: 'terrafirm' [...] Total devices 2 FS bytes used 6.40TiB devid1 size 3.64TiB used 3.21TiB path /dev/sdc *** Some devices missing [...] Of course the first thing is sysadmin's backups rule -- if you don't have it backed up, by definition and action, that data is worth less to you than the time/effort/media to back it up, any claims to the contrary not withstanding, as your actions spoke louder than your words. And of course btrfs isn't fully stable yet, so that rule applies double, compared to a fully mature and stable filesystem. Therefore, you're in luck. Either you didn't lose anything of value because you had it backed up, or it wasn't of /that/ much value to you anyway, so again, you didn't lose anything of value. =:^) (And actually, that's truer than you might think at the moment. Consider my next-door neighbor, who a couple weeks ago escaped a fire with his life and the pair of shorts he was wearing. No wallet, no keys, nothing. I don't think he's going to be worrying much about that last fat-fingering he did on the computer for awhile! If you have your life, your health, a roof over your head, and either food on the kitchen shelves or money to eat out with, you're a lucky one! They say everybody fat-fingers at some point and has to learn the value of those backups the hard way. Now you have that behind you. =:^) I've been there too, and a bit of perspective really does help.) Beyond that... You're running current kernel and userspace/progs already, good job there. =:^) The first thing you need to do in terms of trying to recover, is restore the superblock on the damaged device. Since btrfs keeps multiple copies (up to three, once the filesystem is large enough, as yours is) per device, that's actually relatively easy. Use... btrfs rescue super-recover ... for that. Being a gentooer (as am I), I'm confident you can read the manpage for the details. =:^) You can use... btrfs-show-super ... to examine the various superblock copies in ordered to pick a good one to restore. Beyond that, as a simple btrfs user and list regular who hasn't had that particular problem here (my brown-bag day was a different fat-fingering! ), not a dev, I'm a bit beyond my depth, but were it me, at that point I'd see what I had, then try... btrfs restore There's a page on the wiki with details on how to try to get it to work manually, if it's not going to work automatically for you: https://btrfs.wiki.kernel.org/index.php/Restore I've used it here (tho not in the situation you're in) when I had backups, but they were a bit stale. Basically, I valued the data enough to have backups, but valued saving the hassle over keeping them current, given the amount of actually changed data. Still, since restore worked for me and thus gave me the choice of somewhat stale backups, or rather less stale btrfs restore, I took less stale. =:^) The nice thing about restore, other than that it often works when the filesystem won't mount, is that it doesn't write to the damaged filesystem at all, so there's no chance of making the problem worse instead of better. Restore simply reads what files it can get, restoring them to some other (working and mounted) filesystem. So you'll need some alternate place to store all those files... Tho the wiki is somewhat behind current code, so read the manpage for more useful options. In particular, the dry-run option can give you a good idea whether you're going to get much of anything useful with that generation/transid, or not, and there's now options to restore metadata (ownership and permissions) and symlinks, as well as the file data itself. Also, where it talks about picking a tree root with as many of the filesystem trees as possible, use the list-roots option. This is what displays all those trees it's talking about. If restore worked for you, you have a choice.
Re: BTRFS disaster (of my own making). Is this recoverable?
I can't tell what the data and metadata profile are? That it won't mount degraded makes me think the metadata is not explicitly raid1; and it either raid0 or accidentally single or dup which can happen at mkfs time on single device, and just doing btrfs dev add to add another device. I think recovery is difficult but it depends on what sort of critical information is in those first 32MB other than the superblock. There are copies of the superblock so that can probably be reconstructed. Chris Murphy -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs partition converted from ext4 becomes read-only minutes after booting: WARNING: CPU: 2 PID: 2777 at ../fs/btrfs/super.c:260 __btrfs_abort_transaction+0x4b/0x120
Thanks for the log.
I'll investigate it to see whether we can resolve the infinite loop problem.
Thanks,
Qu
Robert Munteanu wrote on 2015/07/31 16:38 +0300:
On Fri, Jul 31, 2015 at 5:08 AM, Qu Wenruo quwen...@cn.fujitsu.com wrote:
Any full output about it?
Please see the attached log. I left the process running for about 4
hours, and after the first five minutes all it cared about was a
single inode. I ended up stopping it as it looks like it's not making
progress.
Not sure if it real loops, but maybe the inode number changed in later
output?
Looks to me like it's the same inode
$ awk '/ for inode/ { print $7 } ' screenlog.0 | sort | uniq
14214570
Thanks,
Robert
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs partition converted from ext4 becomes read-only minutes after booting: WARNING: CPU: 2 PID: 2777 at ../fs/btrfs/super.c:260 __btrfs_abort_transaction+0x4b/0x120
Yes, you're right, that's a dead loop.
But for better debugging, would you please upload the following info?
1) output of command btrfs-debug-tree -t 5 DEV.
The only important things are info about that inode.
Whose objectid(first item in a key) is 14214570 and type is one of the
following:
INODE_ITEM, INODE_REF, EXTENT_DATA
So you may only need to cut the things like below:
==
item 4 key (14214570 INODE_ITEM 0) itemoff 15881 itemsize 160
inode generation 6 transid 6 size 1073741824 nbytes
1073741824
block group 0 mode 100644 links 1 uid 0 gid 0
rdev 0 flags 0x0
item 5 key (14214570 INODE_REF XXX) itemoff 15866 itemsize 15
inode ref index 2 namelen 5 name: file1
item 6 key (14214570 EXTENT_DATA 0) itemoff 15813 itemsize 53
extent data disk byte 2176843776 nr 134217728
extent data offset 0 nr 134217728 ram 134217728
extent compression 0
item 7 key (14214570 EXTENT_DATA XXX) itemoff 15760 itemsize 53
extent data disk byte 2311061504 nr 134217728
extent data offset 0 nr 134217728 ram 134217728
extent compression 0
(All items with 14214570 objectid is needed to debug)
==
And it's highly recommended to only cut that part and paste it.
Not only to reduce the output, but also help your privacy.
As you can see, INODE_REF contains file name, which can be sometimes
leaking your personal infomation.
2) output of command btrfs-debug-tree -t 2 DEV
Just in case your extent tree mismatch with fs tree.
If you don't like to execute 2 commands and are OK with leaking file/dir
names, you can also use btrfs-debug-tree DEV to dump every metadata
info.
Alternatively, if btrfs-image -c9 DEV works without problem, it will
also helps a lot for debugging.
Thanks,
Qu
Robert Munteanu wrote on 2015/07/31 16:38 +0300:
On Fri, Jul 31, 2015 at 5:08 AM, Qu Wenruo quwen...@cn.fujitsu.com wrote:
Any full output about it?
Please see the attached log. I left the process running for about 4
hours, and after the first five minutes all it cared about was a
single inode. I ended up stopping it as it looks like it's not making
progress.
Not sure if it real loops, but maybe the inode number changed in later
output?
Looks to me like it's the same inode
$ awk '/ for inode/ { print $7 } ' screenlog.0 | sort | uniq
14214570
Thanks,
Robert
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
