Re: Kernel crash triggered by dd to file with memcg, worst on btrfs
Michal Hocko wrote: Richard Davies wrote: I have a test case in which I can often crash an entire machine by running dd to a file with a memcg with relatively generous limits. This is simplified from real world problems with heavy disk i/o inside containers. ... [I have also just reported a different but similar bug with untar in a memcg http://marc.info/?l=linux-mmm=139766321822891 That one is not btrfs-linked] ... Does this happen even if no kmem limit is specified? No, it only happens with a kmem limit. So it is due to the kmem limiting being broken, as we discussed in the other untar thread lined above. The kmem limit would explain allocation failures for ext3 logged below but I would be interested about the Thread overran stack, or stack corrupted message reported for btrfs. The stack doesn't seem very deep there. I would expect some issues in the writeback path during the limit reclaim but this looks quite innocent. Rulling out kmem accounting would be a good first step though . (I am keepinng the full email for Vladimir) The btrfs problems only occur with a kmem limit. So this is also kmem-linked even if that is surprising. Richard. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crash triggered by dd to file with memcg, worst on btrfs
On 04/24/2014 01:58 AM, Michal Hocko wrote: I have a test case in which I can often crash an entire machine by running dd to a file with a memcg with relatively generous limits. This is simplified from real world problems with heavy disk i/o inside containers. The crashes are easy to trigger when dding to create a file on btrfs. On ext3, typically there is just an error in the kernel log, although occasionally it also crashes. I have a suspicion that crashes occur, because btrfs doesn't always expect that kmalloc may fail. At least, when trying to reproduce the bug, I got the following stack trace, which clearly points to tree_mod_log_free_eb, where we panic if kmalloc doesn't succeed: [ 728.489378] kernel BUG at fs/btrfs/ctree.c:985! [ 728.489508] invalid opcode: [#1] SMP [ 728.489636] Modules linked in: btrfs raid6_pq xor xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle xt_multiport xt_limit xt_dscp fuse ebt_among ebtable_filter ebtables bridge stp llc binfmt_misc sbs ppdev sbshc parport_pc parport microcode pcspkr lpc_ich mfd_core e1000 raid1 virtio_balloon shpchp [last unloaded: speedstep_lib] [ 728.490739] CPU: 0 PID: 4053 Comm: dd Not tainted 3.15.0-rc2-mm1+ #160 [ 728.490913] Hardware name: Parallels Software International Inc. Parallels Virtual Platform/Parallels Virtual Platform, BIOS 5.0.19471.922416 10/26/2007 [ 728.491325] task: 880069fe9630 ti: 880003dd2000 task.ti: 880003dd2000 [ 728.491544] RIP: 0010:[a013d497] [a013d497] tree_mod_log_set_root_pointer+0x27/0x40 [btrfs] [ 728.491832] RSP: 0018:880003dd3898 EFLAGS: 00010282 [ 728.491975] RAX: fff4 RBX: 880076df8000 RCX: 0007 [ 728.492203] RDX: 3dc0 RSI: 880069fe9ed8 RDI: [ 728.492445] RBP: 880003dd3898 R08: 81d28d40 R09: 88007660dd00 [ 728.492634] R10: 0003 R11: 0001 R12: 88006db2c1d0 [ 728.492822] R13: 8800053d46e0 R14: R15: 88006db18000 [ 728.493010] FS: 7fa6df2cb700() GS:88007960() knlGS: [ 728.493239] CS: 0010 DS: ES: CR0: 80050033 [ 728.493441] CR2: 7fa6debea300 CR3: 4c064000 CR4: 06f0 [ 728.493630] Stack: [ 728.493691] 880003dd3948 a013dc28 8801 [ 728.493918] 8101c809 880003dd38d8 810b3a75 [ 728.494146] 8800053d46e0 810cfed5 880003dd3a08 [ 728.494469] Call Trace: [ 728.494551] [a013dc28] __btrfs_cow_block+0x478/0x530 [btrfs] [ 728.494733] [8101c809] ? sched_clock+0x9/0x10 [ 728.494881] [810b3a75] ? local_clock+0x25/0x30 [ 728.495033] [a013e2de] btrfs_cow_block+0x11e/0x1d0 [btrfs] [ 728.495230] [a01407e3] btrfs_search_slot+0x1f3/0x990 [btrfs] [ 728.495475] [811d4fcd] ? __memcg_kmem_get_cache+0x10d/0x290 [ 728.495658] [a01418c6] btrfs_insert_empty_items+0x76/0xd0 [btrfs] [ 728.495857] [a019c214] btrfs_insert_orphan_item+0x64/0x90 [btrfs] [ 728.496053] [a01678bf] btrfs_orphan_add+0xef/0x1d0 [btrfs] [ 728.496267] [a0173219] btrfs_setattr+0x1f9/0x300 [btrfs] [ 728.496498] [811fefc2] notify_change+0x1c2/0x320 [ 728.496679] [811e81a4] ? flush_old_exec+0x584/0x7d0 [ 728.496837] [811e0473] do_truncate+0x63/0xa0 [ 728.496980] [811ef7c6] do_last+0x646/0xec0 [ 728.497122] [811ee9eb] ? link_path_walk+0x7b/0x810 [ 728.497341] [811f2cf2] path_openat+0xc2/0x620 [ 728.497501] [810523b7] ? kvm_clock_read+0x27/0x40 [ 728.497654] [8101c809] ? sched_clock+0x9/0x10 [ 728.497800] [812002b1] ? __alloc_fd+0x31/0x160 [ 728.497946] [811f3385] do_filp_open+0x45/0xa0 [ 728.498090] [81200338] ? __alloc_fd+0xb8/0x160 [ 728.498272] [811e0bb5] do_sys_open+0x115/0x230 [ 728.498439] [810d07f5] ? trace_hardirqs_on_caller+0x105/0x1d0 [ 728.498623] [81355fbe] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 728.498796] [811e0d09] SyS_open+0x19/0x20 [ 728.498933] [816f23b9] system_call_fastpath+0x16/0x1b [ 728.499094] Code: ff 0f 1f 00 55 48 8b 87 e8 01 00 00 41 89 d0 48 89 f2 48 8b 37 b9 50 00 00 00 48 89 e5 48 89 c7 e8 9f fc ff ff 85 c0 78 02 c9 c3 0f 0b 0f 1f 80 00 00 00 00 eb f7 66 66 66 66 66 2e 0f 1f 84 00 [ 728.500178] RIP [a013d497] tree_mod_log_set_root_pointer+0x27/0x40 [btrfs] [ 728.500457] RSP 880003dd3898 Ext3 kernel error log = 17:20:05 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x20) 17:20:05 kernel: cache: ext4_extent_status(2:test), object size: 40, buffer size: 40, default order: 0, min order: 0 17:20:05 kernel: node 0: slabs: 375, objs: 38250, free: 0 17:20:05 kernel: node 1: slabs: 128, objs: 13056, free: 0 (many times) This looks like the kmem
Re: Kernel crash triggered by dd to file with memcg, worst on btrfs
On Thu 24-04-14 11:59:33, Richard Davies wrote: Michal Hocko wrote: Richard Davies wrote: I have a test case in which I can often crash an entire machine by running dd to a file with a memcg with relatively generous limits. This is simplified from real world problems with heavy disk i/o inside containers. ... [I have also just reported a different but similar bug with untar in a memcg http://marc.info/?l=linux-mmm=139766321822891 That one is not btrfs-linked] ... Does this happen even if no kmem limit is specified? No, it only happens with a kmem limit. So it is due to the kmem limiting being broken, It still might be interesting to debug, because it suggests that some caller doesn't cope with an allocation failure. That being said, kmem accounting is broken for real life usage but crashes produced in the limitted environment is still good to debug. -- Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel crash triggered by dd to file with memcg, worst on btrfs
[CCing Vladimir] On Wed 16-04-14 18:42:10, Richard Davies wrote: Hi all, I have a test case in which I can often crash an entire machine by running dd to a file with a memcg with relatively generous limits. This is simplified from real world problems with heavy disk i/o inside containers. The crashes are easy to trigger when dding to create a file on btrfs. On ext3, typically there is just an error in the kernel log, although occasionally it also crashes. I'm not a kernel developer, but I'm happy to help with any further debugging or try patches. [I have also just reported a different but similar bug with untar in a memcg http://marc.info/?l=linux-mmm=139766321822891 That one is not btrfs-linked] To replicate on Linux 3.14.0, run the following 8 commands: # mkdir -p /sys/fs/cgroup/test/ # cat /sys/fs/cgroup/cpuset.cpus /sys/fs/cgroup/test/cpuset.cpus # cat /sys/fs/cgroup/cpuset.mems /sys/fs/cgroup/test/cpuset.mems # echo $((130)) /sys/fs/cgroup/test/memory.limit_in_bytes # echo $((130)) /sys/fs/cgroup/test/memory.memsw.limit_in_bytes # echo $((128)) /sys/fs/cgroup/test/memory.kmem.limit_in_bytes Does this happen even if no kmem limit is specified? The kmem limit would explain allocation failures for ext3 logged bellow but I would be interested about the Thread overran stack, or stack corrupted message reported for btrfs. The stack doesn't seem very deep there. I would expect some issues in the writeback path during the limit reclaim but this looks quite innocent. Rulling out kmem accounting would be a good first step though . (I am keepinng the full email for Vladimir) # echo $$ /sys/fs/cgroup/test/tasks # dd if=/dev/zero of=./crashme bs=2M and leave until several GB of data have been written. When running into a btrfs filesystem, this dd crashes the entire machine about 50% of the time for me, generating a console log as copied below. If the initial dd is running smoothly, I can often get it to crash by stopping the dd with ctrl-c and starting it again with a different output file, perhaps repeating this a few times. When running into an ext3 filesystem, this dd typically doesn't crash the machine but just output errors in the kernel log as copied below. Occasionally it will still crash. I am happy to help with extra information on kernel configuration, but I hope that the above is sufficient for others to replicate. I'm also happy to try suggestions and patches. Thanks in advance for your help, Richard. Ext3 kernel error log = 17:20:05 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x20) 17:20:05 kernel: cache: ext4_extent_status(2:test), object size: 40, buffer size: 40, default order: 0, min order: 0 17:20:05 kernel: node 0: slabs: 375, objs: 38250, free: 0 17:20:05 kernel: node 1: slabs: 128, objs: 13056, free: 0 (many times) This looks like the kmem limit has been reached and all the further allocation fails. Btrfs kernel console crash log == BUG: unable to handle kernel paging request at fffe36a55230 IP: [810f5055] cpuacct_charge+0x35/0x58 PGD 1b5d067 PUD 0 Thread overran stack, or stack corrupted This is really unexpected. Especially when the stack dumped bellow is not a usual suspect. This is a simple interrupt handler which handles hrtimer and this shouldn't overflow the stack... Oops: [#1] PREEMPT SMP Modules linked in: CPU: 6 PID: 5729 Comm: dd Not tainted 3.14.0-elastic #1 Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 task: 88040a6fdac0 ti: 8800d69cc000 task.ti: 8800d69cc000 RIP: 0010:[810f5055] [810f5055] cpuacct_charge+0x35/0x58 RSP: 0018:880827d03d88 EFLAGS: 00010002 RAX: 60f7d80032d0 RBX: 88040a6fdac0 RCX: d69cc148 RDX: 88081191a180 RSI: 000ebb99 RDI: 88040a6fdac0 RBP: 880827d03da8 R08: R09: 880827ffc348 R10: 880827ffc2a0 R11: 880827ffc340 R12: d69cc148 R13: 000ebb99 R14: fffebb99 R15: 88040a6fdac0 FS: 7f508b54e6f0() GS:880827d0() knlGS: CS: 0010 DS: ES: CR0: 8005003b CR2: fffe36a55230 CR3: 00080e9d2000 CR4: 07e0 Stack: 88040a6fdac0 880810fe2800 000ebb99 880827d03dd8 810ebbb3 88040a6fdb28 880810fe2800 880827d11bc0 880827d03e28 810eeaaf Call Trace: IRQ [810ebbb3] update_curr+0xc2/0x11e [810eeaaf] task_tick_fair+0x3d/0x631 [810e5bb7] scheduler_tick+0x57/0xba [81108eaf] ? tick_nohz_handler+0xcf/0xcf [810cb73d] update_process_times+0x55/0x66 [81108f2b] tick_sched_timer+0x7c/0x9b [810dd0d2] __run_hrtimer+0x57/0xcc [810dd4c7] hrtimer_interrupt+0xd0/0x1db [810e761b] ? __vtime_account_system+0x2d/0x31
Re: Kernel crash triggered by dd to file with memcg, worst on btrfs
Hi, what kernel version are you running? Marian On 04/16/2014 08:42 PM, Richard Davies wrote: Hi all, I have a test case in which I can often crash an entire machine by running dd to a file with a memcg with relatively generous limits. This is simplified from real world problems with heavy disk i/o inside containers. The crashes are easy to trigger when dding to create a file on btrfs. On ext3, typically there is just an error in the kernel log, although occasionally it also crashes. I'm not a kernel developer, but I'm happy to help with any further debugging or try patches. [I have also just reported a different but similar bug with untar in a memcg http://marc.info/?l=linux-mmm=139766321822891 That one is not btrfs-linked] To replicate on Linux 3.14.0, run the following 8 commands: # mkdir -p /sys/fs/cgroup/test/ # cat /sys/fs/cgroup/cpuset.cpus /sys/fs/cgroup/test/cpuset.cpus # cat /sys/fs/cgroup/cpuset.mems /sys/fs/cgroup/test/cpuset.mems # echo $((130)) /sys/fs/cgroup/test/memory.limit_in_bytes # echo $((130)) /sys/fs/cgroup/test/memory.memsw.limit_in_bytes # echo $((128)) /sys/fs/cgroup/test/memory.kmem.limit_in_bytes # echo $$ /sys/fs/cgroup/test/tasks # dd if=/dev/zero of=./crashme bs=2M and leave until several GB of data have been written. When running into a btrfs filesystem, this dd crashes the entire machine about 50% of the time for me, generating a console log as copied below. If the initial dd is running smoothly, I can often get it to crash by stopping the dd with ctrl-c and starting it again with a different output file, perhaps repeating this a few times. When running into an ext3 filesystem, this dd typically doesn't crash the machine but just output errors in the kernel log as copied below. Occasionally it will still crash. I am happy to help with extra information on kernel configuration, but I hope that the above is sufficient for others to replicate. I'm also happy to try suggestions and patches. Thanks in advance for your help, Richard. Ext3 kernel error log = 17:20:05 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x20) 17:20:05 kernel: cache: ext4_extent_status(2:test), object size: 40, buffer size: 40, default order: 0, min order: 0 17:20:05 kernel: node 0: slabs: 375, objs: 38250, free: 0 17:20:05 kernel: node 1: slabs: 128, objs: 13056, free: 0 (many times) Btrfs kernel console crash log == BUG: unable to handle kernel paging request at fffe36a55230 IP: [810f5055] cpuacct_charge+0x35/0x58 PGD 1b5d067 PUD 0 Thread overran stack, or stack corrupted Oops: [#1] PREEMPT SMP Modules linked in: CPU: 6 PID: 5729 Comm: dd Not tainted 3.14.0-elastic #1 Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 task: 88040a6fdac0 ti: 8800d69cc000 task.ti: 8800d69cc000 RIP: 0010:[810f5055] [810f5055] cpuacct_charge+0x35/0x58 RSP: 0018:880827d03d88 EFLAGS: 00010002 RAX: 60f7d80032d0 RBX: 88040a6fdac0 RCX: d69cc148 RDX: 88081191a180 RSI: 000ebb99 RDI: 88040a6fdac0 RBP: 880827d03da8 R08: R09: 880827ffc348 R10: 880827ffc2a0 R11: 880827ffc340 R12: d69cc148 R13: 000ebb99 R14: fffebb99 R15: 88040a6fdac0 FS: 7f508b54e6f0() GS:880827d0() knlGS: CS: 0010 DS: ES: CR0: 8005003b CR2: fffe36a55230 CR3: 00080e9d2000 CR4: 07e0 Stack: 88040a6fdac0 880810fe2800 000ebb99 880827d03dd8 810ebbb3 88040a6fdb28 880810fe2800 880827d11bc0 880827d03e28 810eeaaf Call Trace: IRQ [810ebbb3] update_curr+0xc2/0x11e [810eeaaf] task_tick_fair+0x3d/0x631 [810e5bb7] scheduler_tick+0x57/0xba [81108eaf] ? tick_nohz_handler+0xcf/0xcf [810cb73d] update_process_times+0x55/0x66 [81108f2b] tick_sched_timer+0x7c/0x9b [810dd0d2] __run_hrtimer+0x57/0xcc [810dd4c7] hrtimer_interrupt+0xd0/0x1db [810e761b] ? __vtime_account_system+0x2d/0x31 [8105f8c1] local_apic_timer_interrupt+0x53/0x58 [81060475] smp_apic_timer_interrupt+0x3e/0x51 [8186299d] apic_timer_interrupt+0x6d/0x80 EOI Code: 54 53 48 89 fb 48 83 ec 08 48 8b 47 08 4c 63 60 18 e8 84 8c 00 00 48 8b 83 a0 06 00 00 4c 89 e1 48 8b 50 48 48 8b 82 80 00 00 00 48 03 04 cd f0 47 bf 81 4c 01 28 48 8b 52 40 48 85 d2 75 e5 e8 RIP [810f5055] cpuacct_charge+0x35/0x58 RSP 880827d03d88 CR2: fffe36a55230 ---[ end trace b449af50c3a0711c ]--- Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: 0x0 from 0x8100 (relocation range: 0x8000-0x9fff) -- To unsubscribe from this list: send the line unsubscribe cgroups in the body of a message to majord...@vger.kernel.org More majordomo info at
Re: Kernel crash triggered by dd to file with memcg, worst on btrfs
Richard Davies wrote: I have a test case in which I can often crash an entire machine by running dd to a file with a memcg with relatively generous limits. This is simplified from real world problems with heavy disk i/o inside containers. The crashes are easy to trigger when dding to create a file on btrfs. On ext3, typically there is just an error in the kernel log, although occasionally it also crashes. A further note - the ext3 SLUB errors occur when dding into a ext3 file alone. The few ext3 crashes occurred when dding into a btrfs file for a while without a crash, then switching to dding into an ext3 file. So the ext3 crashes could actually be due to btrfs cached data still in memory - i.e. all crashes could be due to btrfs use. Richard. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Kernel crash triggered by dd to file with memcg, worst on btrfs
Hi all, I have a test case in which I can often crash an entire machine by running dd to a file with a memcg with relatively generous limits. This is simplified from real world problems with heavy disk i/o inside containers. The crashes are easy to trigger when dding to create a file on btrfs. On ext3, typically there is just an error in the kernel log, although occasionally it also crashes. I'm not a kernel developer, but I'm happy to help with any further debugging or try patches. [I have also just reported a different but similar bug with untar in a memcg http://marc.info/?l=linux-mmm=139766321822891 That one is not btrfs-linked] To replicate on Linux 3.14.0, run the following 8 commands: # mkdir -p /sys/fs/cgroup/test/ # cat /sys/fs/cgroup/cpuset.cpus /sys/fs/cgroup/test/cpuset.cpus # cat /sys/fs/cgroup/cpuset.mems /sys/fs/cgroup/test/cpuset.mems # echo $((130)) /sys/fs/cgroup/test/memory.limit_in_bytes # echo $((130)) /sys/fs/cgroup/test/memory.memsw.limit_in_bytes # echo $((128)) /sys/fs/cgroup/test/memory.kmem.limit_in_bytes # echo $$ /sys/fs/cgroup/test/tasks # dd if=/dev/zero of=./crashme bs=2M and leave until several GB of data have been written. When running into a btrfs filesystem, this dd crashes the entire machine about 50% of the time for me, generating a console log as copied below. If the initial dd is running smoothly, I can often get it to crash by stopping the dd with ctrl-c and starting it again with a different output file, perhaps repeating this a few times. When running into an ext3 filesystem, this dd typically doesn't crash the machine but just output errors in the kernel log as copied below. Occasionally it will still crash. I am happy to help with extra information on kernel configuration, but I hope that the above is sufficient for others to replicate. I'm also happy to try suggestions and patches. Thanks in advance for your help, Richard. Ext3 kernel error log = 17:20:05 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x20) 17:20:05 kernel: cache: ext4_extent_status(2:test), object size: 40, buffer size: 40, default order: 0, min order: 0 17:20:05 kernel: node 0: slabs: 375, objs: 38250, free: 0 17:20:05 kernel: node 1: slabs: 128, objs: 13056, free: 0 (many times) Btrfs kernel console crash log == BUG: unable to handle kernel paging request at fffe36a55230 IP: [810f5055] cpuacct_charge+0x35/0x58 PGD 1b5d067 PUD 0 Thread overran stack, or stack corrupted Oops: [#1] PREEMPT SMP Modules linked in: CPU: 6 PID: 5729 Comm: dd Not tainted 3.14.0-elastic #1 Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 task: 88040a6fdac0 ti: 8800d69cc000 task.ti: 8800d69cc000 RIP: 0010:[810f5055] [810f5055] cpuacct_charge+0x35/0x58 RSP: 0018:880827d03d88 EFLAGS: 00010002 RAX: 60f7d80032d0 RBX: 88040a6fdac0 RCX: d69cc148 RDX: 88081191a180 RSI: 000ebb99 RDI: 88040a6fdac0 RBP: 880827d03da8 R08: R09: 880827ffc348 R10: 880827ffc2a0 R11: 880827ffc340 R12: d69cc148 R13: 000ebb99 R14: fffebb99 R15: 88040a6fdac0 FS: 7f508b54e6f0() GS:880827d0() knlGS: CS: 0010 DS: ES: CR0: 8005003b CR2: fffe36a55230 CR3: 00080e9d2000 CR4: 07e0 Stack: 88040a6fdac0 880810fe2800 000ebb99 880827d03dd8 810ebbb3 88040a6fdb28 880810fe2800 880827d11bc0 880827d03e28 810eeaaf Call Trace: IRQ [810ebbb3] update_curr+0xc2/0x11e [810eeaaf] task_tick_fair+0x3d/0x631 [810e5bb7] scheduler_tick+0x57/0xba [81108eaf] ? tick_nohz_handler+0xcf/0xcf [810cb73d] update_process_times+0x55/0x66 [81108f2b] tick_sched_timer+0x7c/0x9b [810dd0d2] __run_hrtimer+0x57/0xcc [810dd4c7] hrtimer_interrupt+0xd0/0x1db [810e761b] ? __vtime_account_system+0x2d/0x31 [8105f8c1] local_apic_timer_interrupt+0x53/0x58 [81060475] smp_apic_timer_interrupt+0x3e/0x51 [8186299d] apic_timer_interrupt+0x6d/0x80 EOI Code: 54 53 48 89 fb 48 83 ec 08 48 8b 47 08 4c 63 60 18 e8 84 8c 00 00 48 8b 83 a0 06 00 00 4c 89 e1 48 8b 50 48 48 8b 82 80 00 00 00 48 03 04 cd f0 47 bf 81 4c 01 28 48 8b 52 40 48 85 d2 75 e5 e8 RIP [810f5055] cpuacct_charge+0x35/0x58 RSP 880827d03d88 CR2: fffe36a55230 ---[ end trace b449af50c3a0711c ]--- Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: 0x0 from 0x8100 (relocation range: 0x8000-0x9fff) -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
