Having an issue with CentOS7.1 + brtfs w/ docker where /var/lib/docker is btrfs
This appears to be an issue related to brtfs so posting here.
We have not seen the stack traces (below) on a similar server upgraded to
kernel 4.1.1, but since 3.10.0-229 is LTS, we wanted to report it to get it
patched in the distribution.
We've tried the same set up on kernel 4.1.1 but we also get uninterruptible
processes. In this case, we did not observe any call traces from the kernel in
dmesg or journalctl. On that machine, we attempted a full PS, but only got
this:
[root@eg-mesos-jenkins-003 ~]# ps aux | grep D
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 238 0.0 0.0 0 0 ? DN Jul02 0:45 [khugepaged]
Below is the relevant system info for the base kernel machine plus the
dmesg.log as an attachment.
% cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
% uname -a
Linux server-001 3.10.0-229.4.2.el7.x86_64 #1 SMP Wed May 13 10:06:09 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux
% btrfs --version
Btrfs v3.16.2
% sudo btrfs fi show
Label: 'docker' uuid: 4d41939a-099d-4868-b692-c62ddf8eb1b2
Total devices 1 FS bytes used 15.14GiB
devid 1 size 1.07TiB used 71.04GiB path /dev/sda5
Btrfs v3.16.2
% sudo btrfs fi df /var/lib/docker
Data, single: total=62.01GiB, used=14.61GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=4.50GiB, used=534.55MiB
Metadata, single: total=8.00MiB, used=0.00
GlobalReserve, single: total=192.00MiB, used=0.00
Call Trace:
Jul 06 23:40:35 server-001 kernel: INFO: task kworker/u65:9:31973 blocked for
more than 120 seconds.
Jul 06 23:40:35 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:40:35 server-001 kernel: kworker/u65:9 D ffff881fffdb3680 0
31973 2 0x00000080
Jul 06 23:40:35 server-001 kernel: Workqueue: writeback bdi_writeback_workfn
(flush-btrfs-1)
Jul 06 23:40:35 server-001 kernel: ffff8816e1a87738 0000000000000046
ffff8816e1a87fd8 0000000000013680
Jul 06 23:40:36 server-001 kernel: ffff8816e1a87fd8 0000000000013680
ffff8810e1b7cfa0 ffff881fffdb3f48
Jul 06 23:40:36 server-001 kernel: ffff8816e1a877c0 0000000000000002
ffffffff81156330 ffff8816e1a877b0
Jul 06 23:40:36 server-001 kernel: Call Trace:
Jul 06 23:40:36 server-001 kernel: [<ffffffff81156330>] ?
wait_on_page_read+0x60/0x60
Jul 06 23:40:36 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140
Jul 06 23:40:36 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20
Jul 06 23:40:36 server-001 kernel: [<ffffffff816083db>]
__wait_on_bit_lock+0x5b/0xc0
Jul 06 23:40:36 server-001 kernel: [<ffffffff81156458>] __lock_page+0x78/0xa0
Jul 06 23:40:36 server-001 kernel: [<ffffffff81098390>] ?
autoremove_wake_function+0x40/0x40
Jul 06 23:40:36 server-001 kernel: [<ffffffffa07fe715>]
lock_delalloc_pages+0x1e5/0x1f0 [btrfs]
Jul 06 23:40:36 server-001 kernel: [<ffffffffa0800f13>]
find_lock_delalloc_range.constprop.43+0x153/0x200 [btrfs]
Jul 06 23:40:36 server-001 kernel: [<ffffffffa080104b>]
writepage_delalloc.isra.33+0x8b/0x180 [btrfs]
Jul 06 23:40:36 server-001 kernel: [<ffffffffa0801cba>]
__extent_writepage+0xca/0x2b0 [btrfs]
Jul 06 23:40:36 server-001 kernel: [<ffffffffa08021ea>]
extent_write_cache_pages.isra.28.constprop.48+0x34a/0x420 [btrfs]
Jul 06 23:40:37 server-001 kernel: [<ffffffffa08040dc>]
extent_writepages+0x5c/0x90 [btrfs]
Jul 06 23:40:37 server-001 kernel: [<ffffffffa07e6e30>] ?
btrfs_submit_direct+0x6c0/0x6c0 [btrfs]
Jul 06 23:40:37 server-001 kernel: [<ffffffffa07e4738>]
btrfs_writepages+0x28/0x30 [btrfs]
Jul 06 23:40:37 server-001 kernel: [<ffffffff81162fae>] do_writepages+0x1e/0x40
Jul 06 23:40:37 server-001 kernel: [<ffffffff811f0670>]
__writeback_single_inode+0x40/0x220
Jul 06 23:40:37 server-001 kernel: [<ffffffff811f136e>]
writeback_sb_inodes+0x25e/0x420
Jul 06 23:40:37 server-001 kernel: [<ffffffff811f15cf>]
__writeback_inodes_wb+0x9f/0xd0
Jul 06 23:40:37 server-001 kernel: [<ffffffff811f1e13>]
wb_writeback+0x263/0x2f0
Jul 06 23:40:37 server-001 kernel: [<ffffffff811f32a5>]
bdi_writeback_workfn+0x115/0x460
Jul 06 23:40:37 server-001 kernel: [<ffffffff8108f1eb>]
process_one_work+0x17b/0x470
Jul 06 23:40:37 server-001 kernel: [<ffffffff8108ffbb>]
worker_thread+0x11b/0x400
Jul 06 23:40:37 server-001 kernel: [<ffffffff8108fea0>] ?
rescuer_thread+0x400/0x400
Jul 06 23:40:37 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0
Jul 06 23:40:37 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
Jul 06 23:40:38 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0
Jul 06 23:40:38 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
Jul 06 23:40:38 server-001 kernel: INFO: task git:8697 blocked for more than
120 seconds.
Jul 06 23:40:38 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:40:38 server-001 kernel: git D ffff881fffc93680 0
8697 8695 0x00000084
Jul 06 23:40:38 server-001 kernel: ffff881d50c43498 0000000000000082
ffff881d50c43fd8 0000000000013680
Jul 06 23:40:38 server-001 kernel: ffff881d50c43fd8 0000000000013680
ffff883488958b60 ffff881fffc93f48
Jul 06 23:40:38 server-001 kernel: ffff88207ffa6ee8 0000000000000002
ffffffff81156330 ffff881d50c43510
Jul 06 23:40:38 server-001 kernel: Call Trace:
Jul 06 23:40:38 server-001 kernel: [<ffffffff81156330>] ?
wait_on_page_read+0x60/0x60
Jul 06 23:40:38 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140
Jul 06 23:40:38 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20
Jul 06 23:40:38 server-001 kernel: [<ffffffff816082a0>] __wait_on_bit+0x60/0x90
Jul 06 23:40:38 server-001 kernel: [<ffffffff811560c6>]
wait_on_page_bit+0x86/0xb0
Jul 06 23:40:39 server-001 kernel: [<ffffffff81098390>] ?
autoremove_wake_function+0x40/0x40
Jul 06 23:40:39 server-001 kernel: [<ffffffff8116a1b2>]
shrink_page_list+0x6c2/0xad0
Jul 06 23:40:39 server-001 kernel: [<ffffffff813f9b80>] ?
scsi_request_fn+0x50/0x570
Jul 06 23:40:39 server-001 kernel: [<ffffffff8116ac7a>]
shrink_inactive_list+0x1ea/0x560
Jul 06 23:40:39 server-001 kernel: [<ffffffff8116b73d>]
shrink_lruvec+0x36d/0x730
Jul 06 23:40:39 server-001 kernel: [<ffffffff8116bb76>] shrink_zone+0x76/0x1a0
Jul 06 23:40:39 server-001 kernel: [<ffffffff8116c080>]
do_try_to_free_pages+0xf0/0x4e0
Jul 06 23:40:39 server-001 kernel: [<ffffffff8115d90a>] ? __rmqueue+0x8a/0x460
Jul 06 23:40:39 server-001 kernel: [<ffffffff8116c6ba>]
try_to_free_mem_cgroup_pages+0xca/0x160
Jul 06 23:40:39 server-001 kernel: [<ffffffff811bc9ce>]
mem_cgroup_reclaim+0x4e/0xe0
Jul 06 23:40:39 server-001 kernel: [<ffffffff811bceb9>]
__mem_cgroup_try_charge+0x459/0xbe0
Jul 06 23:40:39 server-001 kernel: [<ffffffffa07e4dd5>] ?
btrfs_split_extent_hook+0x35/0x40 [btrfs]
Jul 06 23:40:39 server-001 kernel: [<ffffffffa07c6055>] ?
block_rsv_release_bytes+0x95/0x180 [btrfs]
Jul 06 23:40:40 server-001 kernel: [<ffffffff811bdd69>]
mem_cgroup_charge_common+0x59/0xc0
Jul 06 23:40:40 server-001 kernel: [<ffffffff811bf9ba>]
mem_cgroup_cache_charge+0x8a/0xb0
Jul 06 23:40:40 server-001 kernel: [<ffffffff811571f2>]
__add_to_page_cache_locked+0x52/0x260
Jul 06 23:40:40 server-001 kernel: [<ffffffff81157457>]
add_to_page_cache_lru+0x37/0xb0
Jul 06 23:40:40 server-001 kernel: [<ffffffff811577de>]
find_or_create_page+0x5e/0xa0
Jul 06 23:40:40 server-001 kernel: [<ffffffffa07f3b00>]
prepare_pages.isra.19+0xc0/0x180 [btrfs]
Jul 06 23:40:40 server-001 kernel: [<ffffffffa07f472c>]
__btrfs_buffered_write+0x1dc/0x5c0 [btrfs]
Jul 06 23:40:40 server-001 kernel: [<ffffffff810a0898>] ?
__wake_up_common+0x58/0x90
Jul 06 23:40:40 server-001 kernel: [<ffffffffa07f4d5b>]
btrfs_file_aio_write+0x24b/0x5a0 [btrfs]
Jul 06 23:40:40 server-001 kernel: [<ffffffff811c650d>] do_sync_write+0x8d/0xd0
Jul 06 23:40:40 server-001 kernel: [<ffffffff811c6cad>] vfs_write+0xbd/0x1e0
Jul 06 23:40:40 server-001 kernel: [<ffffffff811c76f8>] SyS_write+0x58/0xb0
Jul 06 23:40:40 server-001 kernel: [<ffffffff81614de9>]
system_call_fastpath+0x16/0x1b
Jul 06 23:42:41 server-001 kernel: INFO: task kworker/u65:9:31973 blocked for
more than 120 seconds.
Jul 06 23:42:41 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:42:41 server-001 kernel: kworker/u65:9 D ffff881fffdb3680 0
31973 2 0x00000080
Jul 06 23:42:41 server-001 kernel: Workqueue: writeback bdi_writeback_workfn
(flush-btrfs-1)
Jul 06 23:42:41 server-001 kernel: ffff8816e1a87738 0000000000000046
ffff8816e1a87fd8 0000000000013680
Jul 06 23:42:41 server-001 kernel: ffff8816e1a87fd8 0000000000013680
ffff8810e1b7cfa0 ffff881fffdb3f48
Jul 06 23:42:41 server-001 kernel: ffff8816e1a877c0 0000000000000002
ffffffff81156330 ffff8816e1a877b0
Jul 06 23:42:41 server-001 kernel: Call Trace:
Jul 06 23:42:41 server-001 kernel: [<ffffffff81156330>] ?
wait_on_page_read+0x60/0x60
Jul 06 23:42:41 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140
Jul 06 23:42:41 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20
Jul 06 23:42:41 server-001 kernel: [<ffffffff816083db>]
__wait_on_bit_lock+0x5b/0xc0
Jul 06 23:42:41 server-001 kernel: [<ffffffff81156458>] __lock_page+0x78/0xa0
Jul 06 23:42:41 server-001 kernel: [<ffffffff81098390>] ?
autoremove_wake_function+0x40/0x40
Jul 06 23:42:42 server-001 kernel: [<ffffffffa07fe715>]
lock_delalloc_pages+0x1e5/0x1f0 [btrfs]
Jul 06 23:42:42 server-001 kernel: [<ffffffffa0800f13>]
find_lock_delalloc_range.constprop.43+0x153/0x200 [btrfs]
Jul 06 23:42:42 server-001 kernel: [<ffffffffa080104b>]
writepage_delalloc.isra.33+0x8b/0x180 [btrfs]
Jul 06 23:42:42 server-001 kernel: [<ffffffffa0801cba>]
__extent_writepage+0xca/0x2b0 [btrfs]
Jul 06 23:42:42 server-001 kernel: [<ffffffffa08021ea>]
extent_write_cache_pages.isra.28.constprop.48+0x34a/0x420 [btrfs]
Jul 06 23:42:42 server-001 kernel: [<ffffffffa08040dc>]
extent_writepages+0x5c/0x90 [btrfs]
Jul 06 23:42:42 server-001 kernel: [<ffffffffa07e6e30>] ?
btrfs_submit_direct+0x6c0/0x6c0 [btrfs]
Jul 06 23:42:42 server-001 kernel: [<ffffffffa07e4738>]
btrfs_writepages+0x28/0x30 [btrfs]
Jul 06 23:42:42 server-001 kernel: [<ffffffff81162fae>] do_writepages+0x1e/0x40
Jul 06 23:42:42 server-001 kernel: [<ffffffff811f0670>]
__writeback_single_inode+0x40/0x220
Jul 06 23:42:42 server-001 kernel: [<ffffffff811f136e>]
writeback_sb_inodes+0x25e/0x420
Jul 06 23:42:43 server-001 kernel: [<ffffffff811f15cf>]
__writeback_inodes_wb+0x9f/0xd0
Jul 06 23:42:43 server-001 kernel: [<ffffffff811f1e13>]
wb_writeback+0x263/0x2f0
Jul 06 23:42:43 server-001 kernel: [<ffffffff811f32a5>]
bdi_writeback_workfn+0x115/0x460
Jul 06 23:42:43 server-001 kernel: [<ffffffff8108f1eb>]
process_one_work+0x17b/0x470
Jul 06 23:42:43 server-001 kernel: [<ffffffff8108ffbb>]
worker_thread+0x11b/0x400
Jul 06 23:42:43 server-001 kernel: [<ffffffff8108fea0>] ?
rescuer_thread+0x400/0x400
Jul 06 23:42:43 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0
Jul 06 23:42:43 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
Jul 06 23:42:43 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0
Jul 06 23:42:43 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
Jul 06 23:42:43 server-001 kernel: INFO: task kworker/u65:22:27037 blocked for
more than 120 seconds.
Jul 06 23:42:43 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:42:43 server-001 kernel: kworker/u65:22 D ffff881fffc33680 0
27037 2 0x00000080
Jul 06 23:42:44 server-001 kernel: Workqueue: events_unbound
btrfs_async_reclaim_metadata_space [btrfs]
Jul 06 23:42:44 server-001 kernel: ffff88001ab8fb80 0000000000000046
ffff88001ab8ffd8 0000000000013680
Jul 06 23:42:44 server-001 kernel: ffff88001ab8ffd8 0000000000013680
ffff8812414571c0 ffff88001ab8fca8
Jul 06 23:42:44 server-001 kernel: ffff88001ab8fcb0 7fffffffffffffff
ffff8812414571c0 0000000000000000
Jul 06 23:42:44 server-001 kernel: Call Trace:
Jul 06 23:42:44 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70
Jul 06 23:42:44 server-001 kernel: [<ffffffff81608119>]
schedule_timeout+0x209/0x2d0
Jul 06 23:42:44 server-001 kernel: [<ffffffff8108d126>] ?
__queue_work+0x136/0x320
Jul 06 23:42:44 server-001 kernel: [<ffffffff8108d3da>] ?
__queue_delayed_work+0xaa/0x1a0
Jul 06 23:42:44 server-001 kernel: [<ffffffff8160a6e6>]
wait_for_completion+0x116/0x170
Jul 06 23:42:44 server-001 kernel: [<ffffffff810a9650>] ?
wake_up_state+0x20/0x20
Jul 06 23:42:44 server-001 kernel: [<ffffffff811f09ee>]
writeback_inodes_sb_nr+0x8e/0xd0
Jul 06 23:42:44 server-001 kernel: [<ffffffffa07c9ea8>]
flush_space+0x458/0x4f0 [btrfs]
Jul 06 23:42:44 server-001 kernel: [<ffffffffa07c9530>] ?
btrfs_get_alloc_profile+0x30/0x40 [btrfs]
Jul 06 23:42:44 server-001 kernel: [<ffffffffa07c9a04>] ?
can_overcommit+0xa4/0xf0 [btrfs]
Jul 06 23:42:45 server-001 kernel: [<ffffffffa07ca0d4>]
btrfs_async_reclaim_metadata_space+0x194/0x210 [btrfs]
Jul 06 23:42:45 server-001 kernel: [<ffffffff8108f1eb>]
process_one_work+0x17b/0x470
Jul 06 23:42:45 server-001 kernel: [<ffffffff8108ffbb>]
worker_thread+0x11b/0x400
Jul 06 23:42:45 server-001 kernel: [<ffffffff8108fea0>] ?
rescuer_thread+0x400/0x400
Jul 06 23:42:45 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0
Jul 06 23:42:45 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
Jul 06 23:42:45 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0
Jul 06 23:42:45 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
Jul 06 23:42:45 server-001 kernel: INFO: task git:8697 blocked for more than
120 seconds.
Jul 06 23:42:45 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:42:45 server-001 kernel: git D ffff881fffc93680 0
8697 8695 0x00000084
Jul 06 23:42:46 server-001 kernel: ffff881d50c43498 0000000000000082
ffff881d50c43fd8 0000000000013680
Jul 06 23:42:46 server-001 kernel: ffff881d50c43fd8 0000000000013680
ffff883488958b60 ffff881fffc93f48
Jul 06 23:42:46 server-001 kernel: ffff88207ffa6ee8 0000000000000002
ffffffff81156330 ffff881d50c43510
Jul 06 23:42:46 server-001 kernel: Call Trace:
Jul 06 23:42:46 server-001 kernel: [<ffffffff81156330>] ?
wait_on_page_read+0x60/0x60
Jul 06 23:42:46 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140
Jul 06 23:42:46 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20
Jul 06 23:42:46 server-001 kernel: [<ffffffff816082a0>] __wait_on_bit+0x60/0x90
Jul 06 23:42:46 server-001 kernel: [<ffffffff811560c6>]
wait_on_page_bit+0x86/0xb0
Jul 06 23:42:46 server-001 kernel: [<ffffffff81098390>] ?
autoremove_wake_function+0x40/0x40
Jul 06 23:42:46 server-001 kernel: [<ffffffff8116a1b2>]
shrink_page_list+0x6c2/0xad0
Jul 06 23:42:46 server-001 kernel: [<ffffffff813f9b80>] ?
scsi_request_fn+0x50/0x570
Jul 06 23:42:46 server-001 kernel: [<ffffffff8116ac7a>]
shrink_inactive_list+0x1ea/0x560
Jul 06 23:42:46 server-001 kernel: [<ffffffff8116b73d>]
shrink_lruvec+0x36d/0x730
Jul 06 23:42:46 server-001 kernel: [<ffffffff8116bb76>] shrink_zone+0x76/0x1a0
Jul 06 23:42:46 server-001 kernel: [<ffffffff8116c080>]
do_try_to_free_pages+0xf0/0x4e0
Jul 06 23:42:47 server-001 kernel: [<ffffffff8115d90a>] ? __rmqueue+0x8a/0x460
Jul 06 23:42:47 server-001 kernel: [<ffffffff8116c6ba>]
try_to_free_mem_cgroup_pages+0xca/0x160
Jul 06 23:42:47 server-001 kernel: [<ffffffff811bc9ce>]
mem_cgroup_reclaim+0x4e/0xe0
Jul 06 23:42:47 server-001 kernel: [<ffffffff811bceb9>]
__mem_cgroup_try_charge+0x459/0xbe0
Jul 06 23:42:47 server-001 kernel: [<ffffffffa07e4dd5>] ?
btrfs_split_extent_hook+0x35/0x40 [btrfs]
Jul 06 23:42:47 server-001 kernel: [<ffffffffa07c6055>] ?
block_rsv_release_bytes+0x95/0x180 [btrfs]
Jul 06 23:42:47 server-001 kernel: [<ffffffff811bdd69>]
mem_cgroup_charge_common+0x59/0xc0
Jul 06 23:42:47 server-001 kernel: [<ffffffff811bf9ba>]
mem_cgroup_cache_charge+0x8a/0xb0
Jul 06 23:42:47 server-001 kernel: [<ffffffff811571f2>]
__add_to_page_cache_locked+0x52/0x260
Jul 06 23:42:47 server-001 kernel: [<ffffffff81157457>]
add_to_page_cache_lru+0x37/0xb0
Jul 06 23:42:47 server-001 kernel: [<ffffffff811577de>]
find_or_create_page+0x5e/0xa0
Jul 06 23:42:47 server-001 kernel: [<ffffffffa07f3b00>]
prepare_pages.isra.19+0xc0/0x180 [btrfs]
Jul 06 23:42:48 server-001 kernel: [<ffffffffa07f472c>]
__btrfs_buffered_write+0x1dc/0x5c0 [btrfs]
Jul 06 23:42:48 server-001 kernel: [<ffffffff810a0898>] ?
__wake_up_common+0x58/0x90
Jul 06 23:42:48 server-001 kernel: [<ffffffffa07f4d5b>]
btrfs_file_aio_write+0x24b/0x5a0 [btrfs]
Jul 06 23:42:48 server-001 kernel: [<ffffffff811c650d>] do_sync_write+0x8d/0xd0
Jul 06 23:42:48 server-001 kernel: [<ffffffff811c6cad>] vfs_write+0xbd/0x1e0
Jul 06 23:42:48 server-001 kernel: [<ffffffff811c76f8>] SyS_write+0x58/0xb0
Jul 06 23:42:48 server-001 kernel: [<ffffffff81614de9>]
system_call_fastpath+0x16/0x1b
Jul 06 23:42:48 server-001 kernel: INFO: task tar:10489 blocked for more than
120 seconds.
Jul 06 23:42:48 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:42:48 server-001 kernel: tar D ffff881fffd93680 0
10489 10479 0x00000080
Jul 06 23:42:48 server-001 kernel: ffff883439adf8d0 0000000000000086
ffff883439adffd8 0000000000013680
Jul 06 23:42:48 server-001 kernel: ffff883439adffd8 0000000000013680
ffff88339d5396c0 ffff883439adf9f8
Jul 06 23:42:49 server-001 kernel: ffff883439adfa00 7fffffffffffffff
ffff88339d5396c0 0000000000000000
Jul 06 23:42:49 server-001 kernel: Call Trace:
Jul 06 23:42:49 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70
Jul 06 23:42:49 server-001 kernel: [<ffffffff81608119>]
schedule_timeout+0x209/0x2d0
Jul 06 23:42:49 server-001 kernel: [<ffffffff8108d126>] ?
__queue_work+0x136/0x320
Jul 06 23:42:49 server-001 kernel: [<ffffffff8108d3da>] ?
__queue_delayed_work+0xaa/0x1a0
Jul 06 23:42:49 server-001 kernel: [<ffffffff8160a6e6>]
wait_for_completion+0x116/0x170
Jul 06 23:42:49 server-001 kernel: [<ffffffff810a9650>] ?
wake_up_state+0x20/0x20
Jul 06 23:42:49 server-001 kernel: [<ffffffff811f09ee>]
writeback_inodes_sb_nr+0x8e/0xd0
Jul 06 23:42:49 server-001 kernel: [<ffffffffa07c9ea8>]
flush_space+0x458/0x4f0 [btrfs]
Jul 06 23:42:49 server-001 kernel: [<ffffffffa07c9530>] ?
btrfs_get_alloc_profile+0x30/0x40 [btrfs]
Jul 06 23:42:49 server-001 kernel: [<ffffffffa07c9a04>] ?
can_overcommit+0xa4/0xf0 [btrfs]
Jul 06 23:42:49 server-001 kernel: [<ffffffffa07ca31e>]
reserve_metadata_bytes+0x1ce/0x540 [btrfs]
Jul 06 23:42:49 server-001 kernel: [<ffffffff81295718>] ?
crypto_shash_update+0x38/0x100
Jul 06 23:42:49 server-001 kernel: [<ffffffffa07cac40>]
btrfs_block_rsv_add+0x30/0x60 [btrfs]
Jul 06 23:42:50 server-001 kernel: [<ffffffffa07e2ee3>]
start_transaction+0x453/0x5a0 [btrfs]
Jul 06 23:42:50 server-001 kernel: [<ffffffffa07b8b25>] ?
btrfs_release_path+0x25/0xb0 [btrfs]
Jul 06 23:42:50 server-001 kernel: [<ffffffffa07e304b>]
btrfs_start_transaction+0x1b/0x20 [btrfs]
Jul 06 23:42:50 server-001 kernel: [<ffffffffa07f08ea>]
btrfs_create+0x4a/0x230 [btrfs]
Jul 06 23:42:50 server-001 kernel: [<ffffffff8126986c>] ?
security_inode_permission+0x1c/0x30
Jul 06 23:42:50 server-001 kernel: [<ffffffff811d30ed>] vfs_create+0xcd/0x130
Jul 06 23:42:50 server-001 kernel: [<ffffffff811d632f>] do_last+0xb8f/0x1270
Jul 06 23:42:50 server-001 kernel: [<ffffffff811d6ad2>] path_openat+0xc2/0x490
Jul 06 23:42:50 server-001 kernel: [<ffffffffa07faf12>] ?
btrfs_removexattr+0x72/0xd0 [btrfs]
Jul 06 23:42:50 server-001 kernel: [<ffffffff811d829b>] do_filp_open+0x4b/0xb0
Jul 06 23:42:50 server-001 kernel: [<ffffffff811e5a4f>] ?
mnt_drop_write+0x1f/0x30
Jul 06 23:42:50 server-001 kernel: [<ffffffff811e4d07>] ? __alloc_fd+0xa7/0x130
Jul 06 23:42:50 server-001 kernel: [<ffffffff811c5f83>] do_sys_open+0xf3/0x1f0
Jul 06 23:42:50 server-001 kernel: [<ffffffff811c609e>] SyS_open+0x1e/0x20
Jul 06 23:42:51 server-001 kernel: [<ffffffff81614de9>]
system_call_fastpath+0x16/0x1b
Jul 06 23:44:51 server-001 kernel: INFO: task khugepaged:252 blocked for more
than 120 seconds.
Jul 06 23:44:51 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:44:51 server-001 kernel: khugepaged D ffff881fffcb3680 0
252 2 0x00000000
Jul 06 23:44:51 server-001 kernel: ffff881fd058bc98 0000000000000046
ffff881fd058bfd8 0000000000013680
Jul 06 23:44:51 server-001 kernel: ffff881fd058bfd8 0000000000013680
ffff883fd1e05b00 ffff883fd1e05b00
Jul 06 23:44:51 server-001 kernel: ffff881fced21fb8 ffff881fced21fc0
ffffffff00000000 ffff881fced21fc8
Jul 06 23:44:51 server-001 kernel: Call Trace:
Jul 06 23:44:51 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70
Jul 06 23:44:51 server-001 kernel: [<ffffffff8160bad5>]
rwsem_down_write_failed+0x115/0x220
Jul 06 23:44:51 server-001 kernel: [<ffffffff811bbf12>] ?
__mem_cgroup_commit_charge+0x152/0x390
Jul 06 23:44:51 server-001 kernel: [<ffffffff812e3493>]
call_rwsem_down_write_failed+0x13/0x20
Jul 06 23:44:51 server-001 kernel: [<ffffffff816095dd>] ? down_write+0x2d/0x30
Jul 06 23:44:52 server-001 kernel: [<ffffffff811b4485>]
khugepaged_scan_mm_slot+0x415/0xca0
Jul 06 23:44:52 server-001 kernel: [<ffffffff811b4f6f>] khugepaged+0x25f/0x4a0
Jul 06 23:44:52 server-001 kernel: [<ffffffff81098350>] ? wake_up_bit+0x30/0x30
Jul 06 23:44:52 server-001 kernel: [<ffffffff811b4d10>] ?
khugepaged_scan_mm_slot+0xca0/0xca0
Jul 06 23:44:52 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0
Jul 06 23:44:52 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
Jul 06 23:44:52 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0
Jul 06 23:44:52 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
Jul 06 23:44:52 server-001 kernel: INFO: task mesos-slave:1559 blocked for more
than 120 seconds.
Jul 06 23:44:52 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:44:52 server-001 kernel: mesos-slave D ffff88407fdf3680 0
1559 1 0x00000080
Jul 06 23:44:52 server-001 kernel: ffff881fc2a07cc8 0000000000000086
ffff881fc2a07fd8 0000000000013680
Jul 06 23:44:53 server-001 kernel: ffff881fc2a07fd8 0000000000013680
ffff881fd0894fa0 ffff881fd0894fa0
Jul 06 23:44:53 server-001 kernel: ffff881fced21fb8 ffffffffffffffff
ffff881fced21fc0 000000000000015c
Jul 06 23:44:53 server-001 kernel: Call Trace:
Jul 06 23:44:53 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70
Jul 06 23:44:53 server-001 kernel: [<ffffffff8160bcd5>]
rwsem_down_read_failed+0xf5/0x165
Jul 06 23:44:53 server-001 kernel: [<ffffffff812e3464>]
call_rwsem_down_read_failed+0x14/0x30
Jul 06 23:44:53 server-001 kernel: [<ffffffff816095a0>] ? down_read+0x20/0x30
Jul 06 23:44:53 server-001 kernel: [<ffffffff81183a41>]
__access_remote_vm+0x51/0x1f0
Jul 06 23:44:53 server-001 kernel: [<ffffffff81184880>]
access_process_vm+0x50/0x70
Jul 06 23:44:53 server-001 kernel: [<ffffffff8122fc1a>]
proc_pid_cmdline+0x8a/0x120
Jul 06 23:44:53 server-001 kernel: [<ffffffff8123107f>]
proc_info_read+0x8f/0xe0
Jul 06 23:44:53 server-001 kernel: [<ffffffff811c6b1c>] vfs_read+0x9c/0x170
Jul 06 23:44:53 server-001 kernel: [<ffffffff811c7648>] SyS_read+0x58/0xb0
Jul 06 23:44:53 server-001 kernel: [<ffffffff81614de9>]
system_call_fastpath+0x16/0x1b
Jul 06 23:44:53 server-001 kernel: INFO: task atop:17585 blocked for more than
120 seconds.
Jul 06 23:44:54 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:44:54 server-001 kernel: atop D ffff881fffcd3680 0
17585 1 0x00000084
Jul 06 23:44:54 server-001 kernel: ffff8816e1677cc8 0000000000000086
ffff8816e1677fd8 0000000000013680
Jul 06 23:44:54 server-001 kernel: ffff8816e1677fd8 0000000000013680
ffff881e441b6660 ffff881e441b6660
Jul 06 23:44:54 server-001 kernel: ffff881fced21fb8 ffffffffffffffff
ffff881fced21fc0 000000000000015c
Jul 06 23:44:54 server-001 kernel: Call Trace:
Jul 06 23:44:54 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70
Jul 06 23:44:54 server-001 kernel: [<ffffffff8160bcd5>]
rwsem_down_read_failed+0xf5/0x165
Jul 06 23:44:54 server-001 kernel: [<ffffffff812e3464>]
call_rwsem_down_read_failed+0x14/0x30
Jul 06 23:44:54 server-001 kernel: [<ffffffff816095a0>] ? down_read+0x20/0x30
Jul 06 23:44:54 server-001 kernel: [<ffffffff81183a41>]
__access_remote_vm+0x51/0x1f0
Jul 06 23:44:54 server-001 kernel: [<ffffffff81184880>]
access_process_vm+0x50/0x70
Jul 06 23:44:54 server-001 kernel: [<ffffffff8122fc1a>]
proc_pid_cmdline+0x8a/0x120
Jul 06 23:44:54 server-001 kernel: [<ffffffff8123107f>]
proc_info_read+0x8f/0xe0
Jul 06 23:44:55 server-001 kernel: [<ffffffff811c6b1c>] vfs_read+0x9c/0x170
Jul 06 23:44:55 server-001 kernel: [<ffffffff811c7648>] SyS_read+0x58/0xb0
Jul 06 23:44:55 server-001 kernel: [<ffffffff81614de9>]
system_call_fastpath+0x16/0x1b
Jul 06 23:44:55 server-001 kernel: INFO: task kworker/u65:9:31973 blocked for
more than 120 seconds.
Jul 06 23:44:55 server-001 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 06 23:44:55 server-001 kernel: kworker/u65:9 D ffff881fffdb3680 0
31973 2 0x00000080
Jul 06 23:44:55 server-001 kernel: Workqueue: writeback bdi_writeback_workfn
(flush-btrfs-1)
Jul 06 23:44:55 server-001 kernel: ffff8816e1a87738 0000000000000046
ffff8816e1a87fd8 0000000000013680
Jul 06 23:44:55 server-001 kernel: ffff8816e1a87fd8 0000000000013680
ffff8810e1b7cfa0 ffff881fffdb3f48
Jul 06 23:44:55 server-001 kernel: ffff8816e1a877c0 0000000000000002
ffffffff81156330 ffff8816e1a877b0
Jul 06 23:44:55 server-001 kernel: Call Trace:
Jul 06 23:44:55 server-001 kernel: [<ffffffff81156330>] ?
wait_on_page_read+0x60/0x60
Jul 06 23:44:55 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140
Jul 06 23:44:56 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20
Jul 06 23:44:56 server-001 kernel: [<ffffffff816083db>]
__wait_on_bit_lock+0x5b/0xc0
Jul 06 23:44:56 server-001 kernel: [<ffffffff81156458>] __lock_page+0x78/0xa0
Jul 06 23:44:56 server-001 kernel: [<ffffffff81098390>] ?
autoremove_wake_function+0x40/0x40
Jul 06 23:44:56 server-001 kernel: [<ffffffffa07fe715>]
lock_delalloc_pages+0x1e5/0x1f0 [btrfs]
Jul 06 23:44:56 server-001 kernel: [<ffffffffa0800f13>]
find_lock_delalloc_range.constprop.43+0x153/0x200 [btrfs]
Jul 06 23:44:56 server-001 kernel: [<ffffffffa080104b>]
writepage_delalloc.isra.33+0x8b/0x180 [btrfs]
Jul 06 23:44:56 server-001 kernel: [<ffffffffa0801cba>]
__extent_writepage+0xca/0x2b0 [btrfs]
Jul 06 23:44:56 server-001 kernel: [<ffffffffa08021ea>]
extent_write_cache_pages.isra.28.constprop.48+0x34a/0x420 [btrfs]
Jul 06 23:44:56 server-001 kernel: [<ffffffffa08040dc>]
extent_writepages+0x5c/0x90 [btrfs]
Jul 06 23:44:56 server-001 kernel: [<ffffffffa07e6e30>] ?
btrfs_submit_direct+0x6c0/0x6c0 [btrfs]
Jul 06 23:44:56 server-001 kernel: [<ffffffffa07e4738>]
btrfs_writepages+0x28/0x30 [btrfs]
Jul 06 23:44:57 server-001 kernel: [<ffffffff81162fae>] do_writepages+0x1e/0x40
Jul 06 23:44:57 server-001 kernel: [<ffffffff811f0670>]
__writeback_single_inode+0x40/0x220
Jul 06 23:44:57 server-001 kernel: [<ffffffff811f136e>]
writeback_sb_inodes+0x25e/0x420
Jul 06 23:44:57 server-001 kernel: [<ffffffff811f15cf>]
__writeback_inodes_wb+0x9f/0xd0
Jul 06 23:44:57 server-001 kernel: [<ffffffff811f1e13>]
wb_writeback+0x263/0x2f0
Jul 06 23:44:57 server-001 kernel: [<ffffffff811f32a5>]
bdi_writeback_workfn+0x115/0x460
Jul 06 23:44:57 server-001 kernel: [<ffffffff8108f1eb>]
process_one_work+0x17b/0x470
Jul 06 23:44:57 server-001 kernel: [<ffffffff8108ffbb>]
worker_thread+0x11b/0x400
Jul 06 23:44:57 server-001 kernel: [<ffffffff8108fea0>] ?
rescuer_thread+0x400/0x400
Jul 06 23:44:57 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0
Jul 06 23:44:57 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
Jul 06 23:44:57 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0
Jul 06 23:44:57 server-001 kernel: [<ffffffff810972d0>] ?
kthread_create_on_node+0x140/0x140
dmesg.log.gz
Description: dmesg.log.gz
