Re: [BUG] Kernel Bug at fs/btrfs/volumes.c:3638
I just noticed that there's a bugreport from opensuse user tripping over the same BUG() during log replay (and his problem was solved by btrfs-zero-log), probably after some crash. The kernel version was 3.1 ie. without the corruption fixes, so while it happened during normal use (and not via a crafted fs image), I'm not sure if this is still the case with recent kernels. Turning the BUG in __btrfs_map_block to return needs checking the value in not-so-few callers and from various callpaths, it's not straightforward to do eg. a quick return during mount, as in your case. Good that Jeff Mahoney's error handling series reduce the number of callers to update. david [ cut here ] WARNING: at /home/abuild/rpmbuild/BUILD/kernel-desktop-3.1.0/linux-3.1/fs/btrfs/tree-log.c:1729 walk_down_log_tree+0x 15a/0x3e0 [btrfs]() Pid: 8978, comm: mount Not tainted 3.1.0-1.2-desktop #1 Call Trace: [810043fa] dump_trace+0xaa/0x2b0 [81582a4a] dump_stack+0x69/0x6f [8105386b] warn_slowpath_common+0x7b/0xc0 [a0573cba] walk_down_log_tree+0x15a/0x3e0 [btrfs] [a0574267] walk_log_tree+0xc7/0x1f0 [btrfs] [a057803c] btrfs_recover_log_trees+0x1ec/0x2d0 [btrfs] [a0544303] open_ctree+0x13c3/0x1740 [btrfs] [a0522733] btrfs_fill_super.isra.36+0x73/0x150 [btrfs] [a0523b29] btrfs_mount+0x359/0x3e0 [btrfs] [81156465] mount_fs+0x45/0x1d0 [8116fdb6] vfs_kern_mount+0x66/0xd0 [81171383] do_kern_mount+0x53/0x120 [81172e35] do_mount+0x1a5/0x260 [811732da] sys_mount+0x9a/0xf0 [815a3292] system_call_fastpath+0x16/0x1b [7fc524137daa] 0x7fc524137da9 ---[ end trace 2bf4520d35da960f ]--- unable to find logical 5493736079360 len 4096 [ cut here ] 1728 if (btrfs_header_level(cur) != *level) 1729 WARN_ON(1); kernel BUG at /home/abuild/rpmbuild/BUILD/kernel-desktop-3.1.0/linux-3.1/fs/btrfs/volumes.c:2891! invalid opcode: [#1] PREEMPT SMP CPU 1 Pid: 8978, comm: mount Tainted: GW 3.1.0-1.2-desktop #1 RIP: 0010:[a0568e28] [a0568e28] __btrfs_map_block+0x7c8/0x890 [btrfs] RSP: 0018:8801b7507798 EFLAGS: 00010296 RAX: 0043 RBX: 04ff1c30 RCX: 2a82 RDX: 723a RSI: 0046 RDI: 0202 RBP: 8801b7507860 R08: 000a R09: R10: R11: 0001 R12: 8801dcd10cc0 R13: 0001 R14: R15: 0001 FS: 7fc524c587e0() GS:88021fd0() knlGS: CS: 0010 DS: ES: CR0: 80050033 CR2: 7faea5cb8000 CR3: 0001b74f4000 CR4: 06e0 DR0: DR1: DR2: DR3: DR6: 0ff0 DR7: 0400 Process mount (pid: 8978, threadinfo 8801b7506000, task 8801b0d9c740) Call Trace: [a056baa7] btrfs_map_bio+0x57/0x210 [btrfs] [a05600d4] submit_one_bio+0x64/0xa0 [btrfs] [a05653c7] read_extent_buffer_pages+0x367/0x4a0 [btrfs] [a053fd10] btree_read_extent_buffer_pages.isra.63+0x80/0xc0 [btrfs] [a0542b3a] btrfs_read_buffer+0x2a/0x40 [btrfs] [a0576d56] replay_one_buffer+0x46/0x360 [btrfs] [a0573d6d] walk_down_log_tree+0x20d/0x3e0 [btrfs] [a0574267] walk_log_tree+0xc7/0x1f0 [btrfs] [a057803c] btrfs_recover_log_trees+0x1ec/0x2d0 [btrfs] [a0544303] open_ctree+0x13c3/0x1740 [btrfs] [a0522733] btrfs_fill_super.isra.36+0x73/0x150 [btrfs] [a0523b29] btrfs_mount+0x359/0x3e0 [btrfs] [81156465] mount_fs+0x45/0x1d0 [8116fdb6] vfs_kern_mount+0x66/0xd0 [81171383] do_kern_mount+0x53/0x120 [81172e35] do_mount+0x1a5/0x260 [811732da] sys_mount+0x9a/0xf0 [815a3292] system_call_fastpath+0x16/0x1b [7fc524137daa] 0x7fc524137da9 -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] Kernel Bug at fs/btrfs/volumes.c:3638
On Sun, Feb 26, 2012 at 11:42:58PM -0500, Jérôme Carretero wrote: At some point, I would appreciate some kind of thorough evaluation using a fuzzer on small disk images. The btrfs developers could for instance: - provide a script to create a filesystem image with a known layout (known corpus) - provide .config and reference to kernel sources to build the kernel - provide a minimal root filesystem to be run under qemu, it would run a procedure on the other disk image at boot crashes wouldn't affect the host, which is good. - provide a way to retrieve the test parameters and results for every test case in case of bug, the test can be reproduced by the developers since the configuration is known - expect volunteers to run the scenarios (I know I would) The tricky part is of course the potentially super-costly procedure... Simplest case: flipping every bit / writing blocks with pseudo-random data, even on meta-data only, as the outcome on data is supposed to be known. Smarter: flipping bits on every btrfs meta-data structure type at every possible logical location. There is a dangerdonteveruse(tm) utility btrfs-corrupt-block able to target at specific metadata structure and corrupt it, with the fsck counterpart for the rescue. I believe we'll see more updates in that area. The block checksums are supposed to catch bitflips after they were written down to device (provided the data were correct up to the checksum point). If you're talking about random bitflips in metadata structures during processing, that's very likely to crash in many ways of course. I think some logic needs to be added to those corruptions and accompanied by the fsck part. The kind of stuff that would help all this could be something like Python bindings for a *btrfs library*. Helpful even for prototyping fsck stuff, making illustrations, etc. We could see btrfsprogs turn into a library + tool, someday. Added to project page. As of today, how are btrfs developers testing the filesystem implementation (except with xfstests) ? If there is a patch fixing particular bug, I try to set up environment stressing exactly that bug (and sometimes finding another one ...). The xfstests suite is a must before any testing. There are common loads raising the chances to hit a bug like repeated snapshots (and deletions), exhausting data/metadata space, 'fi defrag', 'fi sync', 'fi balance'. Sometimes it's enough to run a specific xfstest in a loop. I have set of hackish scripts doing just these tasks or wrappers around xfstests to create filesystem with desired raid levels (where applicable) and let the suite run on top of it. Another dimension of testing are mount options, there are some combinations likely to execrise specific parts of code, or create files in a way that may confuse different mount options (like nodatasum). We've seen btrfs-specific tests added to xfstests, so it's mostly changing the outer environment for the testsuite. david -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] Kernel Bug at fs/btrfs/volumes.c:3638
On Fri, 24 Feb 2012 16:11:29 +0530 Nageswara R Sastry rnsas...@linux.vnet.ibm.com wrote: Hello, While working with 'fsfuzz - file system fuzzing tool' on 'btrfs' encountered the following kernel bug. I inquired about robustness a while ago and it seems it's at some point on the horizon, but not now. My concern was about hot-unplugged disk drives, but btrfs also doesn't appreciate meta-data corruption. btrfs-raid users could be concerned, because contrarily to on a real RAID, the btrfs meta-data is a potential weak link. At some point, I would appreciate some kind of thorough evaluation using a fuzzer on small disk images. The btrfs developers could for instance: - provide a script to create a filesystem image with a known layout (known corpus) - provide .config and reference to kernel sources to build the kernel - provide a minimal root filesystem to be run under qemu, it would run a procedure on the other disk image at boot crashes wouldn't affect the host, which is good. - provide a way to retrieve the test parameters and results for every test case in case of bug, the test can be reproduced by the developers since the configuration is known - expect volunteers to run the scenarios (I know I would) The tricky part is of course the potentially super-costly procedure... Simplest case: flipping every bit / writing blocks with pseudo-random data, even on meta-data only, as the outcome on data is supposed to be known. Smarter: flipping bits on every btrfs meta-data structure type at every possible logical location. The kind of stuff that would help all this could be something like Python bindings for a *btrfs library*. Helpful even for prototyping fsck stuff, making illustrations, etc. As of today, how are btrfs developers testing the filesystem implementation (except with xfstests) ? Best regards, -- cJ PS: don't be mistaken, I'm not asking for all that, just suggesting. My time goes to something else, but I do have sleepy computers at home, and they could help. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] Kernel Bug at fs/btrfs/volumes.c:3638
On ఫిబ్రవరి 25 శనివారం 2012 ఉ. 11:42, Liu Bo wrote: Hi, I guess you're mounting a quite small partition. Given that this oops is in such an early stage, could you please show 1) your mkfs.btrfs options and 2) the log of btrfs-debug-tree /dev/loop0? thanks, liubo Here are the steps with options, 1. dd if=/dev/zero of=filename.img bs=1M count=16 2. mkfs.btrfs filename.img 3. Corrupt the filename.img using 'mangle' 4. mount -t btrfs -o loop filename.img mount point 5. # btrfs-debug-tree filename.img Couldn't map the block 3221392384 Couldn't read chunk root unable to open filename.img # btrfs-debug-tree /dev/loop0 Couldn't map the block 3221392384 Couldn't read chunk root unable to open /dev/loop0 Regards, R.Nageswara Sastry -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] Kernel Bug at fs/btrfs/volumes.c:3638
On 02/24/2012 06:41 PM, Nageswara R Sastry wrote: Hello, While working with 'fsfuzz - file system fuzzing tool' on 'btrfs' encountered the following kernel bug. Environment: Kernel Version: 3.3.0-rc4 Architecture: s390, x86 Providing the kernel trace from 's390' arch. Btrfs loaded device fsid 346683e8-0fcc-4440-b421-4535e73d60d6 devid 1 transid 4 /dev/loop0 btrfs: disk space caching is enabled unable to find logical 131072 len 4096 [ cut here ] kernel BUG at fs/btrfs/volumes.c:3638! illegal operation: 0001 [#1] SMP Modules linked in: btrfs zlib_deflate crc32c libcrc32c loop dm_multipath dm_mod qeth_l3 ipv6 vmur dasd_eckd_mod dasd_mod scsi_dh_hp_sw scsi_dh_alua scsi_dh_rdac scsi_dh_emc scsi_dh scsi_mod qeth qdio ccwgroup ext3 mbcache jbd CPU: 0 Not tainted 3.3.0-rc4-0.27-default #1 Process mount (pid: 2396, task: 3f176738, ksp: 02ab7648) Krnl PSW : 070430018000 03e004c10e08 (__btrfs_map_block+0x794/0x8cc [btrfs]) R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:0 CC:3 PM:0 EA:3 Krnl GPRS: 0001 0708 002d 0400 004d3f26 004e21a8 02ab7828 02130d00 3ee5ed90 3e962108 0002 3e962110 03e004bb 03e004c4c990 03e004c10e04 02ab7620 Krnl Code: 03e004c10df8: e3405004 lg %r4,0(%r5) 03e004c10dfe: c0e5fffcfb87 brasl %r14,3e004bb050c #03e004c10e04: a7f40001 brc 15,3e004c10e06 03e004c10e08: a7f4 brc 15,3e004c10e08 03e004c10e0c: 12bb ltr %r11,%r11 03e004c10e0e: a7c4ffb7 brc 12,3e004c10d7c 03e004c10e12: e3109024 lg %r1,32(%r9) 03e004c10e18: d507d0001078 clc 0(8,%r13),120(%r1) Call Trace: ([03e004c10e04] __btrfs_map_block+0x790/0x8cc [btrfs]) [03e004c10f6e] btrfs_map_block+0x2e/0x3c [btrfs] [03e004c11db4] btrfs_map_bio+0x74/0x2ac [btrfs] [03e004be13c6] btree_submit_bio_hook+0xd6/0xf0 [btrfs] [03e004c06b4c] submit_one_bio+0xb4/0xf8 [btrfs] [03e004c0e292] read_extent_buffer_pages+0x292/0x630 [btrfs] [03e004bddd0c] btree_read_extent_buffer_pages+0xc8/0xfc [btrfs] [03e004bdf488] read_tree_block+0x48/0x7c [btrfs] [03e004be30d6] open_ctree+0xec6/0x15f8 [btrfs] [03e004bbb7d8] btrfs_fill_super+0x90/0x170 [btrfs] [03e004bbbefa] btrfs_mount+0x3ea/0x3f8 [btrfs] [00260b96] mount_fs+0x5a/0x188 [002852e6] vfs_kern_mount+0x6e/0x11c [00285442] do_kern_mount+0x52/0x114 [0028573c] do_mount+0x238/0x288 [0028584e] SyS_mount+0xc2/0xf0 [004d7d88] sysc_noemu+0x22/0x28 [03fffd1fab1e] 0x3fffd1fab1e Last Breaking-Event-Address: [03e004c10e04] __btrfs_map_block+0x790/0x8cc [btrfs] ---[ end trace 1e786b24696895a8 ]--- Steps to reproduce: # mount mangled file system image mount point -t btrfs -o loop Please let me know if you need more information. Thanks in advance. Hi, I guess you're mounting a quite small partition. Given that this oops is in such an early stage, could you please show 1) your mkfs.btrfs options and 2) the log of btrfs-debug-tree /dev/loop0? thanks, liubo Regards R.Nageswara Sastry -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
