Re: Btrfs resize seems to deadlock
OK, an update: After unmouting and running btrfs check, the drive reverted to reporting the old size. Not sure if this was due to unmounting / mounting or doing btrfs check. Btrfs check should have been running in readonly mode. Since it looked like something was wrong with the resize process, I patched my kernel with the posted patch. This time the resize operation finished successfully. On Sun, Oct 21, 2018 at 1:56 AM Filipe Manana wrote: > > On Sun, Oct 21, 2018 at 6:05 AM Andrew Nelson > wrote: > > > > Also, is the drive in a safe state to use? Is there anything I should > > run on the drive to check consistency? > > It should be in a safe state. You can verify it running "btrfs check > /dev/" (it's a readonly operation). > > If you are able to patch and build a kernel, you can also try the > patch. I left it running tests overnight and haven't got any > regressions. > > Thanks. > > > On Sat, Oct 20, 2018 at 10:02 PM Andrew Nelson > > wrote: > > > > > > I have ran the "btrfs inspect-internal dump-tree -t 1" command, but > > > the output is ~55mb. Is there something in particular you are looking > > > for in this? > > > On Sat, Oct 20, 2018 at 1:34 PM Filipe Manana wrote: > > > > > > > > On Sat, Oct 20, 2018 at 9:27 PM Liu Bo wrote: > > > > > > > > > > On Fri, Oct 19, 2018 at 7:09 PM Andrew Nelson > > > > > wrote: > > > > > > > > > > > > I am having an issue with btrfs resize in Fedora 28. I am attempting > > > > > > to enlarge my Btrfs partition. Every time I run "btrfs filesystem > > > > > > resize max $MOUNT", the command runs for a few minutes and then > > > > > > hangs > > > > > > forcing the system to be reset. I am not sure what the state of the > > > > > > filesystem really is at this point. Btrfs usage does report the > > > > > > correct size for after resizing. Details below: > > > > > > > > > > > > > > > > Thanks for the report, the stack is helpful, but this needs a few > > > > > deeper debugging, may I ask you to post "btrfs inspect-internal > > > > > dump-tree -t 1 /dev/your_btrfs_disk"? > > > > > > > > I believe it's actually easy to understand from the trace alone and > > > > it's kind of a bad luck scenario. > > > > I made this fix a few hours ago: > > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/fdmanana/linux.git/commit/?h=fix_find_free_extent_deadlock > > > > > > > > But haven't done full testing yet and might have missed something. > > > > Bo, can you take a look and let me know what you think? > > > > > > > > Thanks. > > > > > > > > > > > > > > So I'd like to know what's the height of your tree "1" which refers to > > > > > root tree in btrfs. > > > > > > > > > > thanks, > > > > > liubo > > > > > > > > > > > $ sudo btrfs filesystem usage $MOUNT > > > > > > Overall: > > > > > > Device size: 90.96TiB > > > > > > Device allocated: 72.62TiB > > > > > > Device unallocated: 18.33TiB > > > > > > Device missing: 0.00B > > > > > > Used: 72.62TiB > > > > > > Free (estimated): 18.34TiB (min: 9.17TiB) > > > > > > Data ratio: 1.00 > > > > > > Metadata ratio: 2.00 > > > > > > Global reserve: 512.00MiB (used: 24.11MiB) > > > > > > > > > > > > Data,single: Size:72.46TiB, Used:72.45TiB > > > > > > $MOUNT72.46TiB > > > > > > > > > > > > Metadata,DUP: Size:86.00GiB, Used:84.96GiB > > > > > > $MOUNT 172.00GiB > > > > > > > > > > > > System,DUP: Size:40.00MiB, Used:7.53MiB > > > > > >$MOUNT80.00MiB > > > > > > > > > > > > Unallocated: > > > > > > $MOUNT18.33TiB > > > > > > > > > > > > $ uname -a > > > > > > Linux localhost.localdomain 4.18.14-200.fc28.x86_64 #1 SMP
Re: Btrfs resize seems to deadlock
Also, is the drive in a safe state to use? Is there anything I should run on the drive to check consistency? On Sat, Oct 20, 2018 at 10:02 PM Andrew Nelson wrote: > > I have ran the "btrfs inspect-internal dump-tree -t 1" command, but > the output is ~55mb. Is there something in particular you are looking > for in this? > On Sat, Oct 20, 2018 at 1:34 PM Filipe Manana wrote: > > > > On Sat, Oct 20, 2018 at 9:27 PM Liu Bo wrote: > > > > > > On Fri, Oct 19, 2018 at 7:09 PM Andrew Nelson > > > wrote: > > > > > > > > I am having an issue with btrfs resize in Fedora 28. I am attempting > > > > to enlarge my Btrfs partition. Every time I run "btrfs filesystem > > > > resize max $MOUNT", the command runs for a few minutes and then hangs > > > > forcing the system to be reset. I am not sure what the state of the > > > > filesystem really is at this point. Btrfs usage does report the > > > > correct size for after resizing. Details below: > > > > > > > > > > Thanks for the report, the stack is helpful, but this needs a few > > > deeper debugging, may I ask you to post "btrfs inspect-internal > > > dump-tree -t 1 /dev/your_btrfs_disk"? > > > > I believe it's actually easy to understand from the trace alone and > > it's kind of a bad luck scenario. > > I made this fix a few hours ago: > > > > https://git.kernel.org/pub/scm/linux/kernel/git/fdmanana/linux.git/commit/?h=fix_find_free_extent_deadlock > > > > But haven't done full testing yet and might have missed something. > > Bo, can you take a look and let me know what you think? > > > > Thanks. > > > > > > > > So I'd like to know what's the height of your tree "1" which refers to > > > root tree in btrfs. > > > > > > thanks, > > > liubo > > > > > > > $ sudo btrfs filesystem usage $MOUNT > > > > Overall: > > > > Device size: 90.96TiB > > > > Device allocated: 72.62TiB > > > > Device unallocated: 18.33TiB > > > > Device missing: 0.00B > > > > Used: 72.62TiB > > > > Free (estimated): 18.34TiB (min: 9.17TiB) > > > > Data ratio: 1.00 > > > > Metadata ratio: 2.00 > > > > Global reserve: 512.00MiB (used: 24.11MiB) > > > > > > > > Data,single: Size:72.46TiB, Used:72.45TiB > > > > $MOUNT72.46TiB > > > > > > > > Metadata,DUP: Size:86.00GiB, Used:84.96GiB > > > > $MOUNT 172.00GiB > > > > > > > > System,DUP: Size:40.00MiB, Used:7.53MiB > > > >$MOUNT80.00MiB > > > > > > > > Unallocated: > > > > $MOUNT18.33TiB > > > > > > > > $ uname -a > > > > Linux localhost.localdomain 4.18.14-200.fc28.x86_64 #1 SMP Mon Oct 15 > > > > 13:16:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux > > > > > > > > btrfs-transacti D0 2501 2 0x8000 > > > > Call Trace: > > > > ? __schedule+0x253/0x860 > > > > schedule+0x28/0x80 > > > > btrfs_commit_transaction+0x7aa/0x8b0 [btrfs] > > > > ? kmem_cache_alloc+0x166/0x1d0 > > > > ? join_transaction+0x22/0x3e0 [btrfs] > > > > ? finish_wait+0x80/0x80 > > > > transaction_kthread+0x155/0x170 [btrfs] > > > > ? btrfs_cleanup_transaction+0x550/0x550 [btrfs] > > > > kthread+0x112/0x130 > > > > ? kthread_create_worker_on_cpu+0x70/0x70 > > > > ret_from_fork+0x35/0x40 > > > > btrfs D0 2504 2502 0x0002 > > > > Call Trace: > > > > ? __schedule+0x253/0x860 > > > > schedule+0x28/0x80 > > > > btrfs_tree_read_lock+0x8e/0x120 [btrfs] > > > > ? finish_wait+0x80/0x80 > > > > btrfs_read_lock_root_node+0x2f/0x40 [btrfs] > > > > btrfs_search_slot+0xf6/0x9f0 [btrfs] > > > > ? evict_refill_and_join+0xd0/0xd0 [btrfs] > > > > ? inode_insert5+0x119/0x190 > > > > btrfs_lookup_inode+0x3a/0xc0 [btrfs] > > > > ? kmem_cache_alloc+0x166/0x1d0 > > > > btrfs_iget+0x113/0x690 [btrfs] > > > > __lookup_free_space_inode+0xd8/0x150 [btrfs] > > > > lookup_free_space_inode+0x5b/0
Re: Btrfs resize seems to deadlock
I have ran the "btrfs inspect-internal dump-tree -t 1" command, but the output is ~55mb. Is there something in particular you are looking for in this? On Sat, Oct 20, 2018 at 1:34 PM Filipe Manana wrote: > > On Sat, Oct 20, 2018 at 9:27 PM Liu Bo wrote: > > > > On Fri, Oct 19, 2018 at 7:09 PM Andrew Nelson > > wrote: > > > > > > I am having an issue with btrfs resize in Fedora 28. I am attempting > > > to enlarge my Btrfs partition. Every time I run "btrfs filesystem > > > resize max $MOUNT", the command runs for a few minutes and then hangs > > > forcing the system to be reset. I am not sure what the state of the > > > filesystem really is at this point. Btrfs usage does report the > > > correct size for after resizing. Details below: > > > > > > > Thanks for the report, the stack is helpful, but this needs a few > > deeper debugging, may I ask you to post "btrfs inspect-internal > > dump-tree -t 1 /dev/your_btrfs_disk"? > > I believe it's actually easy to understand from the trace alone and > it's kind of a bad luck scenario. > I made this fix a few hours ago: > > https://git.kernel.org/pub/scm/linux/kernel/git/fdmanana/linux.git/commit/?h=fix_find_free_extent_deadlock > > But haven't done full testing yet and might have missed something. > Bo, can you take a look and let me know what you think? > > Thanks. > > > > > So I'd like to know what's the height of your tree "1" which refers to > > root tree in btrfs. > > > > thanks, > > liubo > > > > > $ sudo btrfs filesystem usage $MOUNT > > > Overall: > > > Device size: 90.96TiB > > > Device allocated: 72.62TiB > > > Device unallocated: 18.33TiB > > > Device missing: 0.00B > > > Used: 72.62TiB > > > Free (estimated): 18.34TiB (min: 9.17TiB) > > > Data ratio: 1.00 > > > Metadata ratio: 2.00 > > > Global reserve: 512.00MiB (used: 24.11MiB) > > > > > > Data,single: Size:72.46TiB, Used:72.45TiB > > > $MOUNT72.46TiB > > > > > > Metadata,DUP: Size:86.00GiB, Used:84.96GiB > > > $MOUNT 172.00GiB > > > > > > System,DUP: Size:40.00MiB, Used:7.53MiB > > >$MOUNT80.00MiB > > > > > > Unallocated: > > > $MOUNT18.33TiB > > > > > > $ uname -a > > > Linux localhost.localdomain 4.18.14-200.fc28.x86_64 #1 SMP Mon Oct 15 > > > 13:16:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux > > > > > > btrfs-transacti D0 2501 2 0x8000 > > > Call Trace: > > > ? __schedule+0x253/0x860 > > > schedule+0x28/0x80 > > > btrfs_commit_transaction+0x7aa/0x8b0 [btrfs] > > > ? kmem_cache_alloc+0x166/0x1d0 > > > ? join_transaction+0x22/0x3e0 [btrfs] > > > ? finish_wait+0x80/0x80 > > > transaction_kthread+0x155/0x170 [btrfs] > > > ? btrfs_cleanup_transaction+0x550/0x550 [btrfs] > > > kthread+0x112/0x130 > > > ? kthread_create_worker_on_cpu+0x70/0x70 > > > ret_from_fork+0x35/0x40 > > > btrfs D0 2504 2502 0x0002 > > > Call Trace: > > > ? __schedule+0x253/0x860 > > > schedule+0x28/0x80 > > > btrfs_tree_read_lock+0x8e/0x120 [btrfs] > > > ? finish_wait+0x80/0x80 > > > btrfs_read_lock_root_node+0x2f/0x40 [btrfs] > > > btrfs_search_slot+0xf6/0x9f0 [btrfs] > > > ? evict_refill_and_join+0xd0/0xd0 [btrfs] > > > ? inode_insert5+0x119/0x190 > > > btrfs_lookup_inode+0x3a/0xc0 [btrfs] > > > ? kmem_cache_alloc+0x166/0x1d0 > > > btrfs_iget+0x113/0x690 [btrfs] > > > __lookup_free_space_inode+0xd8/0x150 [btrfs] > > > lookup_free_space_inode+0x5b/0xb0 [btrfs] > > > load_free_space_cache+0x7c/0x170 [btrfs] > > > ? cache_block_group+0x72/0x3b0 [btrfs] > > > cache_block_group+0x1b3/0x3b0 [btrfs] > > > ? finish_wait+0x80/0x80 > > > find_free_extent+0x799/0x1010 [btrfs] > > > btrfs_reserve_extent+0x9b/0x180 [btrfs] > > > btrfs_alloc_tree_block+0x1b3/0x4f0 [btrfs] > > > __btrfs_cow_block+0x11d/0x500 [btrfs] > > > btrfs_cow_block+0xdc/0x180 [btrfs] > > > btrfs_search_slot+0x3bd/0x9f0 [btrfs] > > > btrfs_lookup_inode+0x3a/0xc0 [btrfs] > > > ? kmem_cache_alloc+0x166/0x1d0 > > > btrf
Btrfs resize seems to deadlock
I am having an issue with btrfs resize in Fedora 28. I am attempting to enlarge my Btrfs partition. Every time I run "btrfs filesystem resize max $MOUNT", the command runs for a few minutes and then hangs forcing the system to be reset. I am not sure what the state of the filesystem really is at this point. Btrfs usage does report the correct size for after resizing. Details below: $ sudo btrfs filesystem usage $MOUNT Overall: Device size: 90.96TiB Device allocated: 72.62TiB Device unallocated: 18.33TiB Device missing: 0.00B Used: 72.62TiB Free (estimated): 18.34TiB (min: 9.17TiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 24.11MiB) Data,single: Size:72.46TiB, Used:72.45TiB $MOUNT72.46TiB Metadata,DUP: Size:86.00GiB, Used:84.96GiB $MOUNT 172.00GiB System,DUP: Size:40.00MiB, Used:7.53MiB $MOUNT80.00MiB Unallocated: $MOUNT18.33TiB $ uname -a Linux localhost.localdomain 4.18.14-200.fc28.x86_64 #1 SMP Mon Oct 15 13:16:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux btrfs-transacti D0 2501 2 0x8000 Call Trace: ? __schedule+0x253/0x860 schedule+0x28/0x80 btrfs_commit_transaction+0x7aa/0x8b0 [btrfs] ? kmem_cache_alloc+0x166/0x1d0 ? join_transaction+0x22/0x3e0 [btrfs] ? finish_wait+0x80/0x80 transaction_kthread+0x155/0x170 [btrfs] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs] kthread+0x112/0x130 ? kthread_create_worker_on_cpu+0x70/0x70 ret_from_fork+0x35/0x40 btrfs D0 2504 2502 0x0002 Call Trace: ? __schedule+0x253/0x860 schedule+0x28/0x80 btrfs_tree_read_lock+0x8e/0x120 [btrfs] ? finish_wait+0x80/0x80 btrfs_read_lock_root_node+0x2f/0x40 [btrfs] btrfs_search_slot+0xf6/0x9f0 [btrfs] ? evict_refill_and_join+0xd0/0xd0 [btrfs] ? inode_insert5+0x119/0x190 btrfs_lookup_inode+0x3a/0xc0 [btrfs] ? kmem_cache_alloc+0x166/0x1d0 btrfs_iget+0x113/0x690 [btrfs] __lookup_free_space_inode+0xd8/0x150 [btrfs] lookup_free_space_inode+0x5b/0xb0 [btrfs] load_free_space_cache+0x7c/0x170 [btrfs] ? cache_block_group+0x72/0x3b0 [btrfs] cache_block_group+0x1b3/0x3b0 [btrfs] ? finish_wait+0x80/0x80 find_free_extent+0x799/0x1010 [btrfs] btrfs_reserve_extent+0x9b/0x180 [btrfs] btrfs_alloc_tree_block+0x1b3/0x4f0 [btrfs] __btrfs_cow_block+0x11d/0x500 [btrfs] btrfs_cow_block+0xdc/0x180 [btrfs] btrfs_search_slot+0x3bd/0x9f0 [btrfs] btrfs_lookup_inode+0x3a/0xc0 [btrfs] ? kmem_cache_alloc+0x166/0x1d0 btrfs_update_inode_item+0x46/0x100 [btrfs] cache_save_setup+0xe4/0x3a0 [btrfs] btrfs_start_dirty_block_groups+0x1be/0x480 [btrfs] btrfs_commit_transaction+0xcb/0x8b0 [btrfs] ? btrfs_release_path+0x13/0x80 [btrfs] ? btrfs_update_device+0x8d/0x1c0 [btrfs] btrfs_ioctl_resize.cold.46+0xf4/0xf9 [btrfs] btrfs_ioctl+0xa25/0x2cf0 [btrfs] ? tty_write+0x1fc/0x330 ? do_vfs_ioctl+0xa4/0x620 do_vfs_ioctl+0xa4/0x620 ksys_ioctl+0x60/0x90 ? ksys_write+0x4f/0xb0 __x64_sys_ioctl+0x16/0x20 do_syscall_64+0x5b/0x160 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fcdc0d78c57 Code: Bad RIP value. RSP: 002b:7ffdd1ee6cf8 EFLAGS: 0246 ORIG_RAX: 0010 RAX: ffda RBX: 7ffdd1ee888a RCX: 7fcdc0d78c57 RDX: 7ffdd1ee6da0 RSI: 50009403 RDI: 0003 RBP: 7ffdd1ee6da0 R08: R09: 7ffdd1ee67e0 R10: R11: 0246 R12: 7ffdd1ee888e R13: 0003 R14: R15: kworker/u48:1 D0 2505 2 0x8000 Workqueue: btrfs-freespace-write btrfs_freespace_write_helper [btrfs] Call Trace: ? __schedule+0x253/0x860 schedule+0x28/0x80 btrfs_tree_lock+0x130/0x210 [btrfs] ? finish_wait+0x80/0x80 btrfs_search_slot+0x775/0x9f0 [btrfs] btrfs_mark_extent_written+0xb0/0xb20 [btrfs] ? join_transaction+0x22/0x3e0 [btrfs] btrfs_finish_ordered_io+0x2e0/0x7e0 [btrfs] ? syscall_return_via_sysret+0x13/0x83 ? __switch_to_asm+0x34/0x70 normal_work_helper+0xd3/0x2f0 [btrfs] process_one_work+0x1a1/0x350 worker_thread+0x30/0x380 ? pwq_unbound_release_workfn+0xd0/0xd0 kthread+0x112/0x130 ? kthread_create_worker_on_cpu+0x70/0x70 ret_from_fork+0x35/0x40
