** Attachment added: "hung_kernel_messages.txt" https://bugs.launchpad.net/ubuntu/+source/linux-lts-xenial/+bug/1645187/+attachment/4783929/+files/hung_kernel_messages.txt
** Description changed: Server's load become high with tasks in D state, no choice but to reboot the system. Could be related to the following ? : https://bugzilla.kernel.org/show_bug.cgi?id=119841 https://www.redhat.com/archives/dm-devel/2016-June/msg00399.html https://patchwork.kernel.org/patch/9223697/ https://xen.crc.id.au/bugs/view.php?id=75 - ii xen-hypervisor-4.4-amd64 4.4.2-0ubuntu0.14.04.7 amd64 Xen Hypervisor on AMD64 ii linux-image-extra-4.4.0-47-generic 4.4.0-47.68~14.04.1 amd64 Linux kernel extra modules for version 4.4.0 on 64 bit x86 SMP ii linux-image-4.4.0-47-generic 4.4.0-47.68~14.04.1 amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP - kernel messages: ---------------- - - - Nov 28 07:03:31 server1 kernel: [890070.994700] INFO: task blkback.3.xvda2:5756 blocked for more than 120 seconds. - Nov 28 07:03:31 server1 kernel: [890070.994758] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu - Nov 28 07:03:31 server1 kernel: [890070.994806] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - Nov 28 07:03:31 server1 kernel: [890070.994884] blkback.3.xvda2 D ffff8800b7ff3928 0 5756 2 0x00000000 - Nov 28 07:03:31 server1 kernel: [890070.994890] ffff8800b7ff3928 ffffffff81e13500 ffff8800b6a4be80 ffff8800b7ff4000 - Nov 28 07:03:31 server1 kernel: [890070.994895] ffff88013a83cc18 ffff88013a83cc00 ffffffff00000000 fffffffe00000001 - Nov 28 07:03:31 server1 kernel: [890070.994898] ffff8800b7ff3940 ffffffff817fafc5 ffff8800b6a4be80 ffff8800b7ff39c0 - Nov 28 07:03:31 server1 kernel: [890070.994902] Call Trace: - Nov 28 07:03:31 server1 kernel: [890070.994912] [<ffffffff817fafc5>] schedule+0x35/0x80 - Nov 28 07:03:31 server1 kernel: [890070.994917] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320 - Nov 28 07:03:31 server1 kernel: [890070.994923] [<ffffffff81689577>] ? push+0x47/0x50 - Nov 28 07:03:31 server1 kernel: [890070.994927] [<ffffffff81689f07>] ? dm_kcopyd_copy+0x147/0x1f0 - Nov 28 07:03:31 server1 kernel: [890070.994931] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20 - Nov 28 07:03:31 server1 kernel: [890070.994933] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40 - Nov 28 07:03:31 server1 kernel: [890070.994939] [<ffffffffc0314dae>] __origin_write+0x6e/0x210 [dm_snapshot] - Nov 28 07:03:31 server1 kernel: [890070.994944] [<ffffffff81185625>] ? mempool_alloc_slab+0x15/0x20 - Nov 28 07:03:31 server1 kernel: [890070.994946] [<ffffffff8118574f>] ? mempool_alloc+0x5f/0x150 - Nov 28 07:03:31 server1 kernel: [890070.994949] [<ffffffffc0314fb7>] do_origin.isra.14+0x67/0x90 [dm_snapshot] - Nov 28 07:03:31 server1 kernel: [890070.994952] [<ffffffffc0315042>] origin_map+0x62/0x80 [dm_snapshot] - Nov 28 07:03:31 server1 kernel: [890070.994955] [<ffffffff8167f2da>] __map_bio+0x3a/0x110 - Nov 28 07:03:31 server1 kernel: [890070.994957] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0 - Nov 28 07:03:31 server1 kernel: [890070.994960] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0 - Nov 28 07:03:31 server1 kernel: [890070.994964] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0 - Nov 28 07:03:31 server1 kernel: [890070.994968] [<ffffffff813aa357>] submit_bio+0x77/0x150 - Nov 28 07:03:31 server1 kernel: [890070.994971] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0 - Nov 28 07:03:31 server1 kernel: [890070.994977] [<ffffffffc02c6a1d>] dispatch_rw_block_io+0x4fd/0x9b0 [xen_blkback] - Nov 28 07:03:31 server1 kernel: [890070.994981] [<ffffffff8101c244>] ? xen_load_sp0+0x84/0x180 - Nov 28 07:03:31 server1 kernel: [890070.994985] [<ffffffffc02c70c5>] __do_block_io_op+0x1f5/0x650 [xen_blkback] - Nov 28 07:03:31 server1 kernel: [890070.994990] [<ffffffff810e5e18>] ? del_timer_sync+0x48/0x50 - Nov 28 07:03:31 server1 kernel: [890070.994993] [<ffffffff817fd8ab>] ? schedule_timeout+0x16b/0x2d0 - Nov 28 07:03:31 server1 kernel: [890070.994997] [<ffffffffc02c7880>] xen_blkif_schedule+0xd0/0x820 [xen_blkback] - Nov 28 07:03:31 server1 kernel: [890070.995002] [<ffffffff810a4e1a>] ? finish_task_switch+0x7a/0x290 - Nov 28 07:03:31 server1 kernel: [890070.995004] [<ffffffff817fa969>] ? __schedule+0x359/0x980 - Nov 28 07:03:31 server1 kernel: [890070.995010] [<ffffffff810bde70>] ? prepare_to_wait_event+0xf0/0xf0 - Nov 28 07:03:31 server1 kernel: [890070.995014] [<ffffffffc02c77b0>] ? xen_blkif_be_int+0x30/0x30 [xen_blkback] - Nov 28 07:03:31 server1 kernel: [890070.995018] [<ffffffff8109ba29>] kthread+0xc9/0xe0 - Nov 28 07:03:31 server1 kernel: [890070.995021] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 - Nov 28 07:03:31 server1 kernel: [890070.995025] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70 - Nov 28 07:03:31 server1 kernel: [890070.995027] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 - Nov 28 07:03:31 server1 kernel: [890070.995033] INFO: task kworker/u4:1:8922 blocked for more than 120 seconds. - Nov 28 07:03:31 server1 kernel: [890070.995106] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu - Nov 28 07:03:31 server1 kernel: [890070.995148] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - Nov 28 07:03:31 server1 kernel: [890070.995200] kworker/u4:1 D ffff8800a7fbb628 0 8922 2 0x00000000 - Nov 28 07:03:31 server1 kernel: [890070.995208] Workqueue: writeback wb_workfn (flush-252:12) - Nov 28 07:03:31 server1 kernel: [890070.995210] ffff8800a7fbb628 ffff880139ad3200 ffff8800b7dfcb00 ffff8800a7fbc000 - Nov 28 07:03:31 server1 kernel: [890070.995212] ffff88013a83cc18 ffff88013a83cc00 ffffffff00000000 fffffffe00000001 - Nov 28 07:03:31 server1 kernel: [890070.995215] ffff8800a7fbb640 ffffffff817fafc5 ffff8800b7dfcb00 ffff8800a7fbb6c0 - Nov 28 07:03:31 server1 kernel: [890070.995217] Call Trace: - Nov 28 07:03:31 server1 kernel: [890070.995220] [<ffffffff817fafc5>] schedule+0x35/0x80 - Nov 28 07:03:31 server1 kernel: [890070.995223] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320 - Nov 28 07:03:31 server1 kernel: [890070.995229] [<ffffffff811fad2a>] ? __slab_alloc+0x4d/0x5c - Nov 28 07:03:31 server1 kernel: [890070.995232] [<ffffffff811dc3eb>] ? kmem_cache_alloc+0x1bb/0x200 - Nov 28 07:03:31 server1 kernel: [890070.995236] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20 - Nov 28 07:03:31 server1 kernel: [890070.995239] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40 - Nov 28 07:03:31 server1 kernel: [890070.995244] [<ffffffffc0315d12>] snapshot_map+0x62/0x390 [dm_snapshot] - Nov 28 07:03:31 server1 kernel: [890070.995272] [<ffffffff8167f2da>] __map_bio+0x3a/0x110 - Nov 28 07:03:31 server1 kernel: [890070.995275] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0 - Nov 28 07:03:31 server1 kernel: [890070.995278] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0 - Nov 28 07:03:31 server1 kernel: [890070.995282] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0 - Nov 28 07:03:31 server1 kernel: [890070.995284] [<ffffffff813aa357>] submit_bio+0x77/0x150 - Nov 28 07:03:31 server1 kernel: [890070.995286] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0 - Nov 28 07:03:31 server1 kernel: [890070.995289] [<ffffffff812347ef>] submit_bh_wbc+0x12f/0x160 - Nov 28 07:03:31 server1 kernel: [890070.995292] [<ffffffff81236615>] __block_write_full_page.constprop.39+0x125/0x360 - Nov 28 07:03:31 server1 kernel: [890070.995293] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20 - Nov 28 07:03:31 server1 kernel: [890070.995295] [<ffffffff8123692e>] block_write_full_page+0xde/0x100 - Nov 28 07:03:31 server1 kernel: [890070.995298] [<ffffffff81237378>] blkdev_writepage+0x18/0x20 - Nov 28 07:03:31 server1 kernel: [890070.995300] [<ffffffff8118d253>] __writepage+0x13/0x40 - Nov 28 07:03:31 server1 kernel: [890070.995302] [<ffffffff8118e5c1>] write_cache_pages+0x241/0x4c0 - Nov 28 07:03:31 server1 kernel: [890070.995304] [<ffffffff8118d240>] ? wb_update_dirty_ratelimit+0x1c0/0x1c0 - Nov 28 07:03:31 server1 kernel: [890070.995307] [<ffffffff8118e883>] generic_writepages+0x43/0x60 - Nov 28 07:03:31 server1 kernel: [890070.995310] [<ffffffff8118d3ff>] ? __wb_calc_thresh+0x2f/0x120 - Nov 28 07:03:31 server1 kernel: [890070.995313] [<ffffffff8118f46e>] do_writepages+0x1e/0x30 - Nov 28 07:03:31 server1 kernel: [890070.995316] [<ffffffff8122b215>] __writeback_single_inode+0x45/0x340 - Nov 28 07:03:31 server1 kernel: [890070.995319] [<ffffffff8122ba4b>] writeback_sb_inodes+0x26b/0x5c0 - Nov 28 07:03:31 server1 kernel: [890070.995322] [<ffffffff8122be26>] __writeback_inodes_wb+0x86/0xc0 - Nov 28 07:03:31 server1 kernel: [890070.995325] [<ffffffff8122c0b2>] wb_writeback+0x252/0x2e0 - Nov 28 07:03:31 server1 kernel: [890070.995328] [<ffffffff8122c818>] wb_workfn+0x238/0x3d0 - Nov 28 07:03:31 server1 kernel: [890070.995332] [<ffffffff810c4e11>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 - Nov 28 07:03:31 server1 kernel: [890070.995337] [<ffffffff81095b40>] process_one_work+0x150/0x3f0 - Nov 28 07:03:31 server1 kernel: [890070.995341] [<ffffffff810962ba>] worker_thread+0x11a/0x470 - Nov 28 07:03:31 server1 kernel: [890070.995345] [<ffffffff817fa969>] ? __schedule+0x359/0x980 - Nov 28 07:03:31 server1 kernel: [890070.995347] [<ffffffff810961a0>] ? rescuer_thread+0x310/0x310 - Nov 28 07:03:31 server1 kernel: [890070.995349] [<ffffffff8109ba29>] kthread+0xc9/0xe0 - Nov 28 07:03:31 server1 kernel: [890070.995351] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 - Nov 28 07:03:31 server1 kernel: [890070.995354] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70 - Nov 28 07:03:31 server1 kernel: [890070.995356] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 - Nov 28 07:03:31 server1 kernel: [890070.995361] INFO: task fsck.ext4:11082 blocked for more than 120 seconds. - Nov 28 07:03:31 server1 kernel: [890070.995433] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu - Nov 28 07:03:31 server1 kernel: [890070.995475] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - Nov 28 07:03:31 server1 kernel: [890070.995519] fsck.ext4 D ffff8800a7caf7d8 0 11082 11081 0x00000000 - Nov 28 07:03:31 server1 kernel: [890070.995522] ffff8800a7caf7d8 ffff88013b000000 ffff8800b8463200 ffff8800a7cb0000 - Nov 28 07:03:31 server1 kernel: [890070.995525] ffff88013a83cc18 ffff88013a83cc00 ffffffff00000000 fffffffe00000001 - Nov 28 07:03:31 server1 kernel: [890070.995527] ffff8800a7caf7f0 ffffffff817fafc5 ffff8800b8463200 ffff8800a7caf870 - Nov 28 07:03:31 server1 kernel: [890070.995529] Call Trace: - Nov 28 07:03:31 server1 kernel: [890070.995532] [<ffffffff817fafc5>] schedule+0x35/0x80 - Nov 28 07:03:31 server1 kernel: [890070.995535] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320 - Nov 28 07:03:31 server1 kernel: [890070.995537] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20 - Nov 28 07:03:31 server1 kernel: [890070.995539] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40 - Nov 28 07:03:31 server1 kernel: [890070.995543] [<ffffffffc0315d12>] snapshot_map+0x62/0x390 [dm_snapshot] - Nov 28 07:03:31 server1 kernel: [890070.995545] [<ffffffff8167f2da>] __map_bio+0x3a/0x110 - Nov 28 07:03:31 server1 kernel: [890070.995547] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0 - Nov 28 07:03:31 server1 kernel: [890070.995550] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0 - Nov 28 07:03:31 server1 kernel: [890070.995553] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0 - Nov 28 07:03:31 server1 kernel: [890070.995555] [<ffffffff813aa357>] submit_bio+0x77/0x150 - Nov 28 07:03:31 server1 kernel: [890070.995558] [<ffffffff81191245>] ? release_pages+0xc5/0x260 - Nov 28 07:03:31 server1 kernel: [890070.995560] [<ffffffff810c4e11>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 - Nov 28 07:03:31 server1 kernel: [890070.995563] [<ffffffff8123d3c9>] do_mpage_readpage+0x2d9/0x6d0 - Nov 28 07:03:31 server1 kernel: [890070.995565] [<ffffffff81191ade>] ? lru_cache_add+0xe/0x10 - Nov 28 07:03:31 server1 kernel: [890070.995567] [<ffffffff8123d8c3>] mpage_readpages+0x103/0x150 - Nov 28 07:03:31 server1 kernel: [890070.995569] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20 - Nov 28 07:03:31 server1 kernel: [890070.995571] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20 - Nov 28 07:03:31 server1 kernel: [890070.995573] [<ffffffff8123733d>] blkdev_readpages+0x1d/0x20 - Nov 28 07:03:31 server1 kernel: [890070.995575] [<ffffffff8118fb94>] __do_page_cache_readahead+0x174/0x200 - Nov 28 07:03:31 server1 kernel: [890070.995577] [<ffffffff8118fd55>] ondemand_readahead+0x135/0x260 - Nov 28 07:03:31 server1 kernel: [890070.995579] [<ffffffff8168033a>] ? dm_any_congested+0x4a/0x50 - Nov 28 07:03:31 server1 kernel: [890070.995581] [<ffffffff8118feec>] page_cache_async_readahead+0x6c/0x70 - Nov 28 07:03:31 server1 kernel: [890070.995584] [<ffffffff811846a0>] generic_file_read_iter+0x390/0x5c0 - Nov 28 07:03:31 server1 kernel: [890070.995586] [<ffffffff812375e7>] blkdev_read_iter+0x37/0x40 - Nov 28 07:03:31 server1 kernel: [890070.995589] [<ffffffff811fdc55>] new_sync_read+0x85/0xb0 - Nov 28 07:03:31 server1 kernel: [890070.995591] [<ffffffff811fdca7>] __vfs_read+0x27/0x40 - Nov 28 07:03:31 server1 kernel: [890070.995593] [<ffffffff811fe24f>] vfs_read+0x7f/0x130 - Nov 28 07:03:31 server1 kernel: [890070.995595] [<ffffffff811ff026>] SyS_read+0x46/0xa0 - Nov 28 07:03:31 server1 kernel: [890070.995597] [<ffffffff811fdf27>] ? SyS_lseek+0x87/0xb0 - Nov 28 07:03:31 server1 kernel: [890070.995599] [<ffffffff817fe836>] entry_SYSCALL_64_fastpath+0x16/0x75 - - - Nov 28 07:04:29 server4 kernel: [504268.435113] INFO: task blkback.44.xvda:8031 blocked for more than 120 seconds. - Nov 28 07:04:29 server4 kernel: [504268.435173] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu - Nov 28 07:04:29 server4 kernel: [504268.435233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - Nov 28 07:04:29 server4 kernel: [504268.435295] blkback.44.xvda D ffff8801709b7928 0 8031 2 0x00000000 - Nov 28 07:04:29 server4 kernel: [504268.435334] ffff8801709b7928 ffff88017ccf8dc0 ffff880119c30000 ffff8801709b8000 - Nov 28 07:04:29 server4 kernel: [504268.435339] ffff8800047c0c18 ffff8800047c0c00 ffffffff00000000 fffffffe00000001 - Nov 28 07:04:29 server4 kernel: [504268.435343] ffff8801709b7940 ffffffff817fafc5 ffff880119c30000 ffff8801709b79c0 - Nov 28 07:04:29 server4 kernel: [504268.435347] Call Trace: - Nov 28 07:04:29 server4 kernel: [504268.435358] [<ffffffff817fafc5>] schedule+0x35/0x80 - Nov 28 07:04:29 server4 kernel: [504268.435362] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320 - Nov 28 07:04:29 server4 kernel: [504268.435367] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435369] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40 - Nov 28 07:04:29 server4 kernel: [504268.435377] [<ffffffffc0397dae>] __origin_write+0x6e/0x210 [dm_snapshot] - Nov 28 07:04:29 server4 kernel: [504268.435382] [<ffffffff81185625>] ? mempool_alloc_slab+0x15/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435385] [<ffffffff8118574f>] ? mempool_alloc+0x5f/0x150 - Nov 28 07:04:29 server4 kernel: [504268.435388] [<ffffffffc0397fb7>] do_origin.isra.14+0x67/0x90 [dm_snapshot] - Nov 28 07:04:29 server4 kernel: [504268.435391] [<ffffffffc0398042>] origin_map+0x62/0x80 [dm_snapshot] - Nov 28 07:04:29 server4 kernel: [504268.435395] [<ffffffff8167f2da>] __map_bio+0x3a/0x110 - Nov 28 07:04:29 server4 kernel: [504268.435398] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0 - Nov 28 07:04:29 server4 kernel: [504268.435401] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0 - Nov 28 07:04:29 server4 kernel: [504268.435406] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0 - Nov 28 07:04:29 server4 kernel: [504268.435409] [<ffffffff813aa357>] submit_bio+0x77/0x150 - Nov 28 07:04:29 server4 kernel: [504268.435411] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0 - Nov 28 07:04:29 server4 kernel: [504268.435417] [<ffffffffc03a8a1d>] dispatch_rw_block_io+0x4fd/0x9b0 [xen_blkback] - Nov 28 07:04:29 server4 kernel: [504268.435420] [<ffffffff8101c244>] ? xen_load_sp0+0x84/0x180 - Nov 28 07:04:29 server4 kernel: [504268.435423] [<ffffffffc03a90c5>] __do_block_io_op+0x1f5/0x650 [xen_blkback] - Nov 28 07:04:29 server4 kernel: [504268.435427] [<ffffffff810e5e18>] ? del_timer_sync+0x48/0x50 - Nov 28 07:04:29 server4 kernel: [504268.435429] [<ffffffff817fd8ab>] ? schedule_timeout+0x16b/0x2d0 - Nov 28 07:04:29 server4 kernel: [504268.435432] [<ffffffffc03a9880>] xen_blkif_schedule+0xd0/0x820 [xen_blkback] - Nov 28 07:04:29 server4 kernel: [504268.435436] [<ffffffff810a4e1a>] ? finish_task_switch+0x7a/0x290 - Nov 28 07:04:29 server4 kernel: [504268.435438] [<ffffffff817fa969>] ? __schedule+0x359/0x980 - Nov 28 07:04:29 server4 kernel: [504268.435443] [<ffffffff810bde70>] ? prepare_to_wait_event+0xf0/0xf0 - Nov 28 07:04:29 server4 kernel: [504268.435446] [<ffffffffc03a97b0>] ? xen_blkif_be_int+0x30/0x30 [xen_blkback] - Nov 28 07:04:29 server4 kernel: [504268.435449] [<ffffffff8109ba29>] kthread+0xc9/0xe0 - Nov 28 07:04:29 server4 kernel: [504268.435451] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 - Nov 28 07:04:29 server4 kernel: [504268.435455] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70 - Nov 28 07:04:29 server4 kernel: [504268.435457] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 - Nov 28 07:04:29 server4 kernel: [504268.435473] INFO: task kworker/u4:2:10156 blocked for more than 120 seconds. - Nov 28 07:04:29 server4 kernel: [504268.435523] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu - Nov 28 07:04:29 server4 kernel: [504268.435568] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - Nov 28 07:04:29 server4 kernel: [504268.435617] kworker/u4:2 D ffff880160633628 0 10156 2 0x00000000 - Nov 28 07:04:29 server4 kernel: [504268.435625] Workqueue: writeback wb_workfn (flush-252:36) - Nov 28 07:04:29 server4 kernel: [504268.435627] ffff880160633628 ffff880007366040 ffff880076470dc0 ffff880160634000 - Nov 28 07:04:29 server4 kernel: [504268.435630] ffff8800047c0c18 ffff8800047c0c00 ffffffff00000000 fffffffe00000001 - Nov 28 07:04:29 server4 kernel: [504268.435632] ffff880160633640 ffffffff817fafc5 ffff880076470dc0 ffff8801606336c0 - Nov 28 07:04:29 server4 kernel: [504268.435635] Call Trace: - Nov 28 07:04:29 server4 kernel: [504268.435637] [<ffffffff817fafc5>] schedule+0x35/0x80 - Nov 28 07:04:29 server4 kernel: [504268.435640] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320 - Nov 28 07:04:29 server4 kernel: [504268.435642] [<ffffffff813a1a5c>] ? bvec_alloc+0x5c/0xf0 - Nov 28 07:04:29 server4 kernel: [504268.435645] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435647] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40 - Nov 28 07:04:29 server4 kernel: [504268.435650] [<ffffffffc0398d12>] snapshot_map+0x62/0x390 [dm_snapshot] - Nov 28 07:04:29 server4 kernel: [504268.435653] [<ffffffff8167f2da>] __map_bio+0x3a/0x110 - Nov 28 07:04:29 server4 kernel: [504268.435656] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0 - Nov 28 07:04:29 server4 kernel: [504268.435658] [<ffffffff810c4e11>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435661] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0 - Nov 28 07:04:29 server4 kernel: [504268.435664] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0 - Nov 28 07:04:29 server4 kernel: [504268.435667] [<ffffffff813aa357>] submit_bio+0x77/0x150 - Nov 28 07:04:29 server4 kernel: [504268.435669] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0 - Nov 28 07:04:29 server4 kernel: [504268.435673] [<ffffffff812347ef>] submit_bh_wbc+0x12f/0x160 - Nov 28 07:04:29 server4 kernel: [504268.435675] [<ffffffff81236615>] __block_write_full_page.constprop.39+0x125/0x360 - Nov 28 07:04:29 server4 kernel: [504268.435677] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435679] [<ffffffff8123692e>] block_write_full_page+0xde/0x100 - Nov 28 07:04:29 server4 kernel: [504268.435682] [<ffffffff81237378>] blkdev_writepage+0x18/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435684] [<ffffffff8118d253>] __writepage+0x13/0x40 - Nov 28 07:04:29 server4 kernel: [504268.435687] [<ffffffff8118e5c1>] write_cache_pages+0x241/0x4c0 - Nov 28 07:04:29 server4 kernel: [504268.435689] [<ffffffff8118d240>] ? wb_update_dirty_ratelimit+0x1c0/0x1c0 - Nov 28 07:04:29 server4 kernel: [504268.435692] [<ffffffff817fe23a>] ? _raw_spin_unlock_irqrestore+0x1a/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435701] [<ffffffffc0126226>] ? _base_get_chain_buffer_tracker+0x86/0xd0 [mpt3sas] - Nov 28 07:04:29 server4 kernel: [504268.435703] [<ffffffff8118e883>] generic_writepages+0x43/0x60 - Nov 28 07:04:29 server4 kernel: [504268.435706] [<ffffffff8118f46e>] do_writepages+0x1e/0x30 - Nov 28 07:04:29 server4 kernel: [504268.435708] [<ffffffff8122b215>] __writeback_single_inode+0x45/0x340 - Nov 28 07:04:29 server4 kernel: [504268.435710] [<ffffffff8122ba4b>] writeback_sb_inodes+0x26b/0x5c0 - Nov 28 07:04:29 server4 kernel: [504268.435713] [<ffffffff8122be26>] __writeback_inodes_wb+0x86/0xc0 - Nov 28 07:04:29 server4 kernel: [504268.435715] [<ffffffff8122c0b2>] wb_writeback+0x252/0x2e0 - Nov 28 07:04:29 server4 kernel: [504268.435717] [<ffffffff8122c8a2>] wb_workfn+0x2c2/0x3d0 - Nov 28 07:04:29 server4 kernel: [504268.435719] [<ffffffff810c4e11>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435724] [<ffffffff81095b40>] process_one_work+0x150/0x3f0 - Nov 28 07:04:29 server4 kernel: [504268.435727] [<ffffffff810962ba>] worker_thread+0x11a/0x470 - Nov 28 07:04:29 server4 kernel: [504268.435729] [<ffffffff817fa969>] ? __schedule+0x359/0x980 - Nov 28 07:04:29 server4 kernel: [504268.435732] [<ffffffff810961a0>] ? rescuer_thread+0x310/0x310 - Nov 28 07:04:29 server4 kernel: [504268.435734] [<ffffffff8109ba29>] kthread+0xc9/0xe0 - Nov 28 07:04:29 server4 kernel: [504268.435736] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 - Nov 28 07:04:29 server4 kernel: [504268.435739] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70 - Nov 28 07:04:29 server4 kernel: [504268.435741] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 - Nov 28 07:04:29 server4 kernel: [504268.435750] INFO: task fsck.ext4:17037 blocked for more than 120 seconds. - Nov 28 07:04:29 server4 kernel: [504268.435797] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu - Nov 28 07:04:29 server4 kernel: [504268.435842] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. - Nov 28 07:04:29 server4 kernel: [504268.435890] fsck.ext4 D ffff88016b6a7888 0 17037 17035 0x00000000 - Nov 28 07:04:29 server4 kernel: [504268.435893] ffff88016b6a7888 ffff880004569b80 ffff880010cb1b80 ffff88016b6a8000 - Nov 28 07:04:29 server4 kernel: [504268.435895] ffff8800047c0c18 ffff8800047c0c00 ffffffff00000000 fffffffe00000001 - Nov 28 07:04:29 server4 kernel: [504268.435898] ffff88016b6a78a0 ffffffff817fafc5 ffff880010cb1b80 ffff88016b6a7918 - Nov 28 07:04:29 server4 kernel: [504268.435901] Call Trace: - Nov 28 07:04:29 server4 kernel: [504268.435903] [<ffffffff817fafc5>] schedule+0x35/0x80 - Nov 28 07:04:29 server4 kernel: [504268.435905] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320 - Nov 28 07:04:29 server4 kernel: [504268.435908] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435910] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40 - Nov 28 07:04:29 server4 kernel: [504268.435913] [<ffffffffc0398d12>] snapshot_map+0x62/0x390 [dm_snapshot] - Nov 28 07:04:29 server4 kernel: [504268.435916] [<ffffffff8167f2da>] __map_bio+0x3a/0x110 - Nov 28 07:04:29 server4 kernel: [504268.435919] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0 - Nov 28 07:04:29 server4 kernel: [504268.435921] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0 - Nov 28 07:04:29 server4 kernel: [504268.435924] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0 - Nov 28 07:04:29 server4 kernel: [504268.435927] [<ffffffff813aa357>] submit_bio+0x77/0x150 - Nov 28 07:04:29 server4 kernel: [504268.435930] [<ffffffff8123cf6a>] mpage_bio_submit+0x2a/0x40 - Nov 28 07:04:29 server4 kernel: [504268.435932] [<ffffffff8123d8f4>] mpage_readpages+0x134/0x150 - Nov 28 07:04:29 server4 kernel: [504268.435934] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435936] [<ffffffff81236af0>] ? I_BDEV+0x20/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435938] [<ffffffff8123733d>] blkdev_readpages+0x1d/0x20 - Nov 28 07:04:29 server4 kernel: [504268.435940] [<ffffffff8118fb94>] __do_page_cache_readahead+0x174/0x200 - Nov 28 07:04:29 server4 kernel: [504268.435943] [<ffffffff8118fd55>] ondemand_readahead+0x135/0x260 - Nov 28 07:04:29 server4 kernel: [504268.435945] [<ffffffff8168033a>] ? dm_any_congested+0x4a/0x50 - Nov 28 07:04:29 server4 kernel: [504268.435948] [<ffffffff8118feec>] page_cache_async_readahead+0x6c/0x70 - Nov 28 07:04:29 server4 kernel: [504268.435951] [<ffffffff811846a0>] generic_file_read_iter+0x390/0x5c0 - Nov 28 07:04:29 server4 kernel: [504268.435953] [<ffffffff812375e7>] blkdev_read_iter+0x37/0x40 - Nov 28 07:04:29 server4 kernel: [504268.435957] [<ffffffff811fdc55>] new_sync_read+0x85/0xb0 - Nov 28 07:04:29 server4 kernel: [504268.435959] [<ffffffff811fdca7>] __vfs_read+0x27/0x40 - Nov 28 07:04:29 server4 kernel: [504268.435961] [<ffffffff811fe24f>] vfs_read+0x7f/0x130 - Nov 28 07:04:29 server4 kernel: [504268.435964] [<ffffffff811ff026>] SyS_read+0x46/0xa0 - Nov 28 07:04:29 server4 kernel: [504268.435966] [<ffffffff811fdf27>] ? SyS_lseek+0x87/0xb0 - Nov 28 07:04:29 server4 kernel: [504268.435969] [<ffffffff817fe836>] entry_SYSCALL_64_fastpath+0x16/0x75 + [890070.994700] INFO: task blkback.3.xvda2:5756 blocked for more than 120 seconds. + [890070.994758] Not tainted 4.4.0-47-generic #68~14.04.1-Ubuntu + [890070.994806] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. + [890070.994884] blkback.3.xvda2 D ffff8800b7ff3928 0 5756 2 0x00000000 + [890070.994890] ffff8800b7ff3928 ffffffff81e13500 ffff8800b6a4be80 ffff8800b7ff4000 + [890070.994895] ffff88013a83cc18 ffff88013a83cc00 ffffffff00000000 fffffffe00000001 + [890070.994898] ffff8800b7ff3940 ffffffff817fafc5 ffff8800b6a4be80 ffff8800b7ff39c0 + [890070.994902] Call Trace: + [890070.994912] [<ffffffff817fafc5>] schedule+0x35/0x80 + [890070.994917] [<ffffffff817fd46a>] rwsem_down_write_failed+0x1da/0x320 + [890070.994923] [<ffffffff81689577>] ? push+0x47/0x50 + [890070.994927] [<ffffffff81689f07>] ? dm_kcopyd_copy+0x147/0x1f0 + [890070.994931] [<ffffffff813e6a53>] call_rwsem_down_write_failed+0x13/0x20 + [890070.994933] [<ffffffff817fcd7d>] ? down_write+0x2d/0x40 + [890070.994939] [<ffffffffc0314dae>] __origin_write+0x6e/0x210 [dm_snapshot] + [890070.994944] [<ffffffff81185625>] ? mempool_alloc_slab+0x15/0x20 + [890070.994946] [<ffffffff8118574f>] ? mempool_alloc+0x5f/0x150 + [890070.994949] [<ffffffffc0314fb7>] do_origin.isra.14+0x67/0x90 [dm_snapshot] + [890070.994952] [<ffffffffc0315042>] origin_map+0x62/0x80 [dm_snapshot] + [890070.994955] [<ffffffff8167f2da>] __map_bio+0x3a/0x110 + [890070.994957] [<ffffffff816809c0>] __split_and_process_bio+0x240/0x3c0 + [890070.994960] [<ffffffff81680baa>] dm_make_request+0x6a/0xd0 + [890070.994964] [<ffffffff813aa221>] generic_make_request+0xe1/0x1a0 + [890070.994968] [<ffffffff813aa357>] submit_bio+0x77/0x150 + [890070.994971] [<ffffffff813a1c71>] ? bio_alloc_bioset+0x181/0x2a0 + [890070.994977] [<ffffffffc02c6a1d>] dispatch_rw_block_io+0x4fd/0x9b0 [xen_blkback] + [890070.994981] [<ffffffff8101c244>] ? xen_load_sp0+0x84/0x180 + [890070.994985] [<ffffffffc02c70c5>] __do_block_io_op+0x1f5/0x650 [xen_blkback] + [890070.994990] [<ffffffff810e5e18>] ? del_timer_sync+0x48/0x50 + [890070.994993] [<ffffffff817fd8ab>] ? schedule_timeout+0x16b/0x2d0 + [890070.994997] [<ffffffffc02c7880>] xen_blkif_schedule+0xd0/0x820 [xen_blkback] + [890070.995002] [<ffffffff810a4e1a>] ? finish_task_switch+0x7a/0x290 + [890070.995004] [<ffffffff817fa969>] ? __schedule+0x359/0x980 + [890070.995010] [<ffffffff810bde70>] ? prepare_to_wait_event+0xf0/0xf0 + [890070.995014] [<ffffffffc02c77b0>] ? xen_blkif_be_int+0x30/0x30 [xen_blkback] + [890070.995018] [<ffffffff8109ba29>] kthread+0xc9/0xe0 + [890070.995021] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 + [890070.995025] [<ffffffff817febcf>] ret_from_fork+0x3f/0x70 + [890070.995027] [<ffffffff8109b960>] ? kthread_park+0x60/0x60 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1645187 Title: 4.4.0-47-generic dm_snapshot random deadlock To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-lts-xenial/+bug/1645187/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
