I rebooted the system in the meantime. It now had Kernel 4.2.0-30.36 from previous updates:
cat /proc/version_signature Ubuntu 4.2.0-30.36~14.04.1-generic 4.2.8-ckt3 During the reboot, the system performed some disk I/O for several hours. Before services like sshd became available. I guess this disk operation happened before the file system was mounted in read-write mode. What I can say is that the system tried to continue the failed balance operation during the boot (from syslog) without intervention from my side: BTRFS info (device dm-0): continuing balance BTRFS info (device dm-0): relocating block group 9553211031552 flags 36 found 106150 extents The btrfs balance operation did not succeed again, but in a manner which I was used to before writing this bug report. This is MUCH better than going into an unpredictable state (ro mode with Input/Output errors). I guess the OS totally got out of sync. BTRFS info (device dm-0): 2 enospc errors during balance The btrfs file system was mounted in READ-WRITE-mode this time (not in read-only mode as before!). And this time, I can read the files and navigate through directories without any Input/Output errors (before the reboot, I had lots of them!). Finally, everything looks fine for a btrfs file system with no space left on the disk. Now that the fs is online and accessible, I can try to add another disk to the fs to have some empty space again. After that, I hopefully can delete some data and hopefully redo the btrfs balance operation. /var/log/syslog gives me: Mar 22 12:42:37 server kernel: [ 3481.725273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 22 12:42:37 server kernel: [ 3481.726122] mount D ffff88043e296640 0 587 455 0x00000000 Mar 22 12:42:37 server kernel: [ 3481.726125] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 Mar 22 12:42:37 server kernel: [ 3481.726127] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff Mar 22 12:42:37 server kernel: [ 3481.726129] ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 Mar 22 12:42:37 server kernel: [ 3481.726130] Call Trace: Mar 22 12:42:37 server kernel: [ 3481.726136] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 3481.726138] [<ffffffff817b8ac7>] schedule+0x37/0x80 Mar 22 12:42:37 server kernel: [ 3481.726140] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 Mar 22 12:42:37 server kernel: [ 3481.726143] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 Mar 22 12:42:37 server kernel: [ 3481.726146] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 Mar 22 12:42:37 server kernel: [ 3481.726147] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 3481.726149] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 Mar 22 12:42:37 server kernel: [ 3481.726151] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 Mar 22 12:42:37 server kernel: [ 3481.726152] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 Mar 22 12:42:37 server kernel: [ 3481.726155] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 Mar 22 12:42:37 server kernel: [ 3481.726157] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 Mar 22 12:42:37 server kernel: [ 3481.726159] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 Mar 22 12:42:37 server kernel: [ 3481.726160] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 Mar 22 12:42:37 server kernel: [ 3481.726175] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726184] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726190] [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726195] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726202] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726210] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726215] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726222] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726227] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726233] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726239] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726247] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726253] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726258] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726260] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 Mar 22 12:42:37 server kernel: [ 3481.726262] [<ffffffff8120c413>] do_mount+0x843/0xcf0 Mar 22 12:42:37 server kernel: [ 3481.726263] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 Mar 22 12:42:37 server kernel: [ 3481.726265] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 Mar 22 12:42:37 server kernel: [ 3481.726266] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 Mar 22 12:42:37 server kernel: [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds. Mar 22 12:42:37 server kernel: [ 4562.226559] Tainted: G U 4.2.0-30-generic #36~14.04.1-Ubuntu Mar 22 12:42:37 server kernel: [ 4562.227415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [...] Mar 22 12:42:37 server kernel: [ 3481.726122] mount D ffff88043e296640 0 587 455 0x00000000 Mar 22 12:42:37 server kernel: [ 3481.726125] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 Mar 22 12:42:37 server kernel: [ 3481.726127] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff Mar 22 12:42:37 server kernel: [ 3481.726129] ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 Mar 22 12:42:37 server kernel: [ 3481.726130] Call Trace: Mar 22 12:42:37 server kernel: [ 3481.726136] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 3481.726138] [<ffffffff817b8ac7>] schedule+0x37/0x80 Mar 22 12:42:37 server kernel: [ 3481.726140] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 Mar 22 12:42:37 server kernel: [ 3481.726143] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 Mar 22 12:42:37 server kernel: [ 3481.726146] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 Mar 22 12:42:37 server kernel: [ 3481.726147] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 3481.726149] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 Mar 22 12:42:37 server kernel: [ 3481.726151] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 Mar 22 12:42:37 server kernel: [ 3481.726152] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 Mar 22 12:42:37 server kernel: [ 3481.726155] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 Mar 22 12:42:37 server kernel: [ 3481.726157] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 Mar 22 12:42:37 server kernel: [ 3481.726159] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 Mar 22 12:42:37 server kernel: [ 3481.726160] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 Mar 22 12:42:37 server kernel: [ 3481.726175] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726184] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726190] [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726195] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726202] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726210] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726215] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726222] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726227] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726233] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726239] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726247] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726253] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726258] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726260] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 Mar 22 12:42:37 server kernel: [ 3481.726262] [<ffffffff8120c413>] do_mount+0x843/0xcf0 Mar 22 12:42:37 server kernel: [ 3481.726263] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 Mar 22 12:42:37 server kernel: [ 3481.726265] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 Mar 22 12:42:37 server kernel: [ 3481.726266] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 Mar 22 12:42:37 server kernel: [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds. [...] Mar 22 12:42:37 server kernel: [ 4562.228280] mount D ffff88043e296640 0 587 455 0x00000000 Mar 22 12:42:37 server kernel: [ 4562.228283] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 Mar 22 12:42:37 server kernel: [ 4562.228285] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff Mar 22 12:42:37 server kernel: [ 4562.228286] ffff88043e5c05e8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 Mar 22 12:42:37 server kernel: [ 4562.228288] Call Trace: Mar 22 12:42:37 server kernel: [ 4562.228294] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 4562.228296] [<ffffffff817b8ac7>] schedule+0x37/0x80 Mar 22 12:42:37 server kernel: [ 4562.228297] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 Mar 22 12:42:37 server kernel: [ 4562.228301] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 Mar 22 12:42:37 server kernel: [ 4562.228304] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 Mar 22 12:42:37 server kernel: [ 4562.228305] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 4562.228307] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 Mar 22 12:42:37 server kernel: [ 4562.228309] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 Mar 22 12:42:37 server kernel: [ 4562.228310] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 Mar 22 12:42:37 server kernel: [ 4562.228313] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 Mar 22 12:42:37 server kernel: [ 4562.228315] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 Mar 22 12:42:37 server kernel: [ 4562.228317] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 Mar 22 12:42:37 server kernel: [ 4562.228318] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 Mar 22 12:42:37 server kernel: [ 4562.228332] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228334] [<ffffffff81180df1>] ? set_page_dirty+0x41/0x70 Mar 22 12:42:37 server kernel: [ 4562.228343] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228348] [<ffffffffc0498071>] ? btrfs_release_path+0x21/0xa0 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228354] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228361] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228369] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228374] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228381] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228386] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228392] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228398] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228405] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228412] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228417] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228419] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 Mar 22 12:42:37 server kernel: [ 4562.228421] [<ffffffff8120c413>] do_mount+0x843/0xcf0 Mar 22 12:42:37 server kernel: [ 4562.228422] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 Mar 22 12:42:37 server kernel: [ 4562.228424] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 Mar 22 12:42:37 server kernel: [ 4562.228425] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 Mar 22 12:42:37 server kernel: [ 7975.560908] BTRFS info (device dm-0): continuing balance [...] Mar 22 12:44:03 server kernel: [ 8073.441009] BTRFS info (device dm-0): relocating block group 9553211031552 flags 36 [...] Mar 22 12:51:33 server kernel: [ 8524.324198] btrfs-transacti D ffff88043e3d6640 0 409 2 0x00000000 Mar 22 12:51:33 server kernel: [ 8524.324204] ffff880424983d68 0000000000000046 ffff88042b9b8000 ffff880425be5280 Mar 22 12:51:33 server kernel: [ 8524.324208] 0000000000000246 ffff880424984000 ffff8803ff9884e0 ffff8804265801b0 Mar 22 12:51:33 server kernel: [ 8524.324211] ffff88042bbe5000 0000000000000000 ffff880424983d88 ffffffff817b8ac7 Mar 22 12:51:33 server kernel: [ 8524.324215] Call Trace: Mar 22 12:51:33 server kernel: [ 8524.324225] [<ffffffff817b8ac7>] schedule+0x37/0x80 Mar 22 12:51:33 server kernel: [ 8524.324247] [<ffffffffc04c06d0>] wait_for_commit.isra.17+0x40/0x70 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324252] [<ffffffff810b7670>] ? prepare_to_wait_event+0xf0/0xf0 Mar 22 12:51:33 server kernel: [ 8524.324267] [<ffffffffc04c149c>] btrfs_commit_transaction+0x15c/0xb10 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324281] [<ffffffffc04c1ee3>] ? start_transaction+0x93/0x580 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324295] [<ffffffffc04bc99a>] transaction_kthread+0x1ba/0x240 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324307] [<ffffffffc04bc7e0>] ? btrfs_cleanup_transaction+0x540/0x540 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324311] [<ffffffff81095499>] kthread+0xc9/0xe0 Mar 22 12:51:33 server kernel: [ 8524.324315] [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0 Mar 22 12:51:33 server kernel: [ 8524.324318] [<ffffffff817bc75f>] ret_from_fork+0x3f/0x70 Mar 22 12:51:33 server kernel: [ 8524.324322] [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0 Mar 22 12:52:23 server kernel: [ 8574.196463] init: ureadahead main process (449) killed by KILL signal Mar 22 12:59:29 server kernel: [ 9000.470433] BTRFS info (device dm-0): found 106150 extents Mar 22 12:59:54 server kernel: [ 9025.568882] BTRFS info (device dm-0): 2 enospc errors during balance dmesg gives me [ 3481.723605] INFO: task mount:587 blocked for more than 120 seconds. [ 3481.724438] Tainted: G U 4.2.0-30-generic #36~14.04.1-Ubuntu [ 3481.725273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 3481.726122] mount D ffff88043e296640 0 587 455 0x00000000 [ 3481.726125] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 [ 3481.726127] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff [ 3481.726129] ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 [ 3481.726130] Call Trace: [ 3481.726136] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 [ 3481.726138] [<ffffffff817b8ac7>] schedule+0x37/0x80 [ 3481.726140] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 [ 3481.726143] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 [ 3481.726146] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 [ 3481.726147] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 [ 3481.726149] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 [ 3481.726151] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 [ 3481.726152] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 [ 3481.726155] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 [ 3481.726157] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 [ 3481.726159] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 [ 3481.726160] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 [ 3481.726175] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] [ 3481.726184] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] [ 3481.726190] [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs] [ 3481.726195] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] [ 3481.726202] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] [ 3481.726210] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] [ 3481.726215] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] [ 3481.726222] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] [ 3481.726227] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] [ 3481.726233] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] [ 3481.726239] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] [ 3481.726247] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] [ 3481.726253] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] [ 3481.726258] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] [ 3481.726260] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 [ 3481.726262] [<ffffffff8120c413>] do_mount+0x843/0xcf0 [ 3481.726263] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 [ 3481.726265] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 [ 3481.726266] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds. [ 4562.226559] Tainted: G U 4.2.0-30-generic #36~14.04.1-Ubuntu [ 4562.227415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4562.228280] mount D ffff88043e296640 0 587 455 0x00000000 [ 4562.228283] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 [ 4562.228285] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff [ 4562.228286] ffff88043e5c05e8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 [ 4562.228288] Call Trace: [ 4562.228294] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 [ 4562.228296] [<ffffffff817b8ac7>] schedule+0x37/0x80 [ 4562.228297] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 [ 4562.228301] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 [ 4562.228304] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 [ 4562.228305] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 [ 4562.228307] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 [ 4562.228309] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 [ 4562.228310] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 [ 4562.228313] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 [ 4562.228315] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 [ 4562.228317] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 [ 4562.228318] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 [ 4562.228332] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] [ 4562.228334] [<ffffffff81180df1>] ? set_page_dirty+0x41/0x70 [ 4562.228343] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] [ 4562.228348] [<ffffffffc0498071>] ? btrfs_release_path+0x21/0xa0 [btrfs] [ 4562.228354] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] [ 4562.228361] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] [ 4562.228369] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] [ 4562.228374] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] [ 4562.228381] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] [ 4562.228386] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] [ 4562.228392] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] [ 4562.228398] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] [ 4562.228405] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] [ 4562.228412] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] [ 4562.228417] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] [ 4562.228419] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 [ 4562.228421] [<ffffffff8120c413>] do_mount+0x843/0xcf0 [ 4562.228422] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 [ 4562.228424] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 [ 4562.228425] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 [ 7975.560908] BTRFS info (device dm-0): continuing balance [ 7977.590090] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) [ 7978.881574] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null) [ 7982.053560] init: failsafe main process (1047) killed by TERM signal [...] [ 8073.441009] BTRFS info (device dm-0): relocating block group 9553211031552 flags 36 [ 8081.239604] audit: type=1400 audit(1458647050.858:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=1299 comm="apparmor_parser" [ 8171.758863] init: mysql post-start process (1322) terminated with status 1 [ 8190.153034] audit: type=1400 audit(1458647159.668:23): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/var/tmp/mysql/" pid=1321 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=106 ouid=0 [ 8216.355398] init: plymouth-upstart-bridge main process ended, respawning [ 8524.324045] INFO: task btrfs-transacti:409 blocked for more than 120 seconds. [ 8524.324102] Tainted: G U 4.2.0-30-generic #36~14.04.1-Ubuntu [ 8524.324149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8524.324198] btrfs-transacti D ffff88043e3d6640 0 409 2 0x00000000 [ 8524.324204] ffff880424983d68 0000000000000046 ffff88042b9b8000 ffff880425be5280 [ 8524.324208] 0000000000000246 ffff880424984000 ffff8803ff9884e0 ffff8804265801b0 [ 8524.324211] ffff88042bbe5000 0000000000000000 ffff880424983d88 ffffffff817b8ac7 [ 8524.324215] Call Trace: [ 8524.324225] [<ffffffff817b8ac7>] schedule+0x37/0x80 [ 8524.324247] [<ffffffffc04c06d0>] wait_for_commit.isra.17+0x40/0x70 [btrfs] [ 8524.324252] [<ffffffff810b7670>] ? prepare_to_wait_event+0xf0/0xf0 [ 8524.324267] [<ffffffffc04c149c>] btrfs_commit_transaction+0x15c/0xb10 [btrfs] [ 8524.324281] [<ffffffffc04c1ee3>] ? start_transaction+0x93/0x580 [btrfs] [ 8524.324295] [<ffffffffc04bc99a>] transaction_kthread+0x1ba/0x240 [btrfs] [ 8524.324307] [<ffffffffc04bc7e0>] ? btrfs_cleanup_transaction+0x540/0x540 [btrfs] [ 8524.324311] [<ffffffff81095499>] kthread+0xc9/0xe0 [ 8524.324315] [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0 [ 8524.324318] [<ffffffff817bc75f>] ret_from_fork+0x3f/0x70 [ 8524.324322] [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0 [ 8574.196463] init: ureadahead main process (449) killed by KILL signal [ 9000.470433] BTRFS info (device dm-0): found 106150 extents [ 9025.568882] BTRFS info (device dm-0): 2 enospc errors during balance -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1559723 Title: soft lockup during btrfs balance To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1559723/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
