We don't know what triggered this bug. While the server was operational
and hosting 5 virtual machines, we saw abnormal behavior to one of them
and after a while the whole host became unresponsive due to high loads.
We were unable to force shutdown guest vms due to device or resource
busy error. We were only able to find a few more info from the quest vm
that was first affected.
Please find bellow.
Oct 29 13:53:15 ws091 kernel: [201133.016407] INFO: rcu_sched detected stalls
on CPUs/tasks:
Oct 29 13:53:15 ws091 kernel: [201133.050472] 30-...0: (1 GPs behind)
idle=83e/140000000000000/0 softirq=294262/294263 fqs=4910
Oct 29 13:53:15 ws091 kernel: [201133.053751] (detected by 7, t=15011
jiffies, g=3090541, c=3090540, q=8389)
Oct 29 13:53:15 ws091 kernel: [201133.056359] Sending NMI from CPU 7 to CPUs 30:
Oct 29 13:53:15 ws091 kernel: [201142.979012] rcu_sched kthread starved for
6707 jiffies! g3090541 c3090540 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=45
Oct 29 13:53:15 ws091 kernel: [201142.983752] rcu_sched I 0 8
2 0x80000000
Oct 29 13:53:15 ws091 kernel: [201142.983755] Call Trace:
Oct 29 13:53:15 ws091 kernel: [201142.983763] __schedule+0x3d6/0x8b0
Oct 29 13:53:15 ws091 kernel: [201142.983765] schedule+0x36/0x80
Oct 29 13:53:15 ws091 kernel: [201142.983767] schedule_timeout+0x162/0x370
Oct 29 13:53:15 ws091 kernel: [201142.983770] ?
__next_timer_interrupt+0xe0/0xe0
Oct 29 13:53:15 ws091 kernel: [201142.983774] rcu_gp_kthread+0x5b4/0x960
Oct 29 13:53:15 ws091 kernel: [201142.983779] kthread+0x105/0x140
Oct 29 13:53:15 ws091 kernel: [201142.983780] ? rcu_barrier_sched+0x10/0x10
Oct 29 13:53:15 ws091 kernel: [201142.983781] ?
kthread_destroy_worker+0x50/0x50
Oct 29 13:53:15 ws091 kernel: [201142.983784] ret_from_fork+0x35/0x40
Oct 29 13:55:57 ws091 kernel: [201305.267245] INFO: task jbd2/vdc1-8:1412
blocked for more than 120 seconds.
Oct 29 13:55:57 ws091 kernel: [201305.340061] Not tainted
4.15.0-38-generic #41~16.04.1-Ubuntu
Oct 29 13:55:57 ws091 kernel: [201305.341300] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 13:55:57 ws091 kernel: [201305.342813] jbd2/vdc1-8 D 0 1412
2 0x80000000
Oct 29 13:55:57 ws091 kernel: [201305.342817] Call Trace:
Oct 29 13:55:57 ws091 kernel: [201305.342831] __schedule+0x3d6/0x8b0
Oct 29 13:55:57 ws091 kernel: [201305.342834] schedule+0x36/0x80
Oct 29 13:55:57 ws091 kernel: [201305.342840]
jbd2_journal_commit_transaction+0x241/0x1760
Oct 29 13:55:57 ws091 kernel: [201305.342846] ? __switch_to_asm+0x34/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342848] ? __switch_to_asm+0x40/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342849] ? __switch_to_asm+0x34/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342851] ? __switch_to_asm+0x40/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342853] ? __switch_to_asm+0x34/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342855] ? __switch_to_asm+0x40/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342857] ? __switch_to_asm+0x40/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342859] ? __switch_to_asm+0x34/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342860] ? __switch_to_asm+0x34/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342862] ? __switch_to_asm+0x40/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342873] ? __switch_to_asm+0x40/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342879] ? wait_woken+0x80/0x80
Oct 29 13:55:57 ws091 kernel: [201305.342882] ? __switch_to_asm+0x40/0x70
Oct 29 13:55:57 ws091 kernel: [201305.342890] ? lock_timer_base+0x75/0xa0
Oct 29 13:55:57 ws091 kernel: [201305.342895] kjournald2+0xc8/0x250
Oct 29 13:55:57 ws091 kernel: [201305.342897] ? kjournald2+0xc8/0x250
Oct 29 13:55:57 ws091 kernel: [201305.342899] ? wait_woken+0x80/0x80
Oct 29 13:55:57 ws091 kernel: [201305.342905] kthread+0x105/0x140
Oct 29 13:55:57 ws091 kernel: [201305.342907] ? commit_timeout+0x20/0x20
Oct 29 13:55:57 ws091 kernel: [201305.342909] ?
kthread_destroy_worker+0x50/0x50
Oct 29 13:55:57 ws091 kernel: [201305.342911] ret_from_fork+0x35/0x40
Oct 29 13:55:57 ws091 kernel: [201305.343058] INFO: task kworker/u112:1:45667
blocked for more than 120 seconds.
Oct 29 13:55:57 ws091 kernel: [201305.344520] Not tainted
4.15.0-38-generic #41~16.04.1-Ubuntu
Oct 29 13:55:58 ws091 kernel: [201305.362862] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 13:55:58 ws091 kernel: [201305.376988] kworker/u112:1 D 0 45667
2 0x80000000
Oct 29 13:55:58 ws091 kernel: [201305.377000] Workqueue: writeback wb_workfn
(flush-252:32)
Oct 29 13:55:58 ws091 kernel: [201305.377002] Call Trace:
Oct 29 13:55:58 ws091 kernel: [201305.377007] __schedule+0x3d6/0x8b0
Oct 29 13:55:58 ws091 kernel: [201305.377016] ?
blk_mq_sched_insert_requests+0x7a/0x80
Oct 29 13:55:58 ws091 kernel: [201305.377017] schedule+0x36/0x80
Oct 29 13:55:58 ws091 kernel: [201305.377026] wait_transaction_locked+0x8a/0xd0
Oct 29 13:55:58 ws091 kernel: [201305.377029] ? wait_woken+0x80/0x80
Oct 29 13:55:58 ws091 kernel: [201305.377031]
add_transaction_credits+0x1c1/0x2a0
Oct 29 13:55:58 ws091 kernel: [201305.377034] ? blk_mq_make_request+0x3ba/0x590
Oct 29 13:55:58 ws091 kernel: [201305.377036] start_this_handle+0x103/0x400
Oct 29 13:55:58 ws091 kernel: [201305.377038] ? _cond_resched+0x1a/0x50
Oct 29 13:55:58 ws091 kernel: [201305.377045] ? kmem_cache_alloc+0x10f/0x1c0
Oct 29 13:55:58 ws091 kernel: [201305.377047] jbd2__journal_start+0xdb/0x1f0
Oct 29 13:55:58 ws091 kernel: [201305.377053] ? ext4_writepages+0x533/0xe40
Oct 29 13:55:58 ws091 kernel: [201305.377060]
__ext4_journal_start_sb+0x6d/0x120
Oct 29 13:55:58 ws091 kernel: [201305.377067] ext4_writepages+0x533/0xe40
Oct 29 13:55:58 ws091 kernel: [201305.377080] do_writepages+0x1f/0x70
Oct 29 13:55:58 ws091 kernel: [201305.377087] ? do_writepages+0x1f/0x70
Oct 29 13:55:58 ws091 kernel: [201305.377114]
__writeback_single_inode+0x45/0x340
Oct 29 13:55:58 ws091 kernel: [201305.377116] writeback_sb_inodes+0x26a/0x590
Oct 29 13:55:58 ws091 kernel: [201305.377118] __writeback_inodes_wb+0x92/0xc0
Oct 29 13:55:58 ws091 kernel: [201305.377120] wb_writeback+0x274/0x330
Oct 29 13:55:58 ws091 kernel: [201305.377125] ? cpumask_next+0x1b/0x20
Oct 29 13:55:58 ws091 kernel: [201305.377126] wb_workfn+0x369/0x460
Oct 29 13:55:58 ws091 kernel: [201305.377128] ? wb_workfn+0x369/0x460
Oct 29 13:55:58 ws091 kernel: [201305.377131] process_one_work+0x14d/0x410
Oct 29 13:55:58 ws091 kernel: [201305.377133] worker_thread+0x4b/0x460
Oct 29 13:55:58 ws091 kernel: [201305.377135] kthread+0x105/0x140
Oct 29 13:55:58 ws091 kernel: [201305.377137] ? process_one_work+0x410/0x410
Oct 29 13:55:58 ws091 kernel: [201305.377139] ?
kthread_destroy_worker+0x50/0x50
Oct 29 13:55:58 ws091 kernel: [201305.377141] ret_from_fork+0x35/0x40
Oct 29 13:56:16 ws091 kernel: [201313.402918] INFO: rcu_sched detected stalls
on CPUs/tasks:
Oct 29 13:56:16 ws091 kernel: [201313.445035] 30-...0: (1 GPs behind)
idle=83e/140000000000000/0 softirq=294262/294263 fqs=4910
Oct 29 13:56:16 ws091 kernel: [201313.446659] (detected by 1, t=60098
jiffies, g=3090541, c=3090540, q=27661)
Oct 29 13:56:16 ws091 kernel: [201313.448004] Sending NMI from CPU 1 to CPUs 30:
Oct 29 13:56:16 ws091 kernel: [201323.370360] rcu_sched kthread starved for
51698 jiffies! g3090541 c3090540 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x200 ->cpu=45
Oct 29 13:56:16 ws091 kernel: [201323.374460] rcu_sched R 0 8
2 0x80000000
Oct 29 13:56:16 ws091 kernel: [201323.374463] Call Trace:
Oct 29 13:56:16 ws091 kernel: [201323.374472] __schedule+0x3d6/0x8b0
Oct 29 13:56:16 ws091 kernel: [201323.374474] schedule+0x36/0x80
Oct 29 13:56:16 ws091 kernel: [201323.374477] schedule_timeout+0x162/0x370
Oct 29 13:56:16 ws091 kernel: [201323.374480] ?
__next_timer_interrupt+0xe0/0xe0
Oct 29 13:56:16 ws091 kernel: [201323.374484] rcu_gp_kthread+0x5b4/0x960
Oct 29 13:56:16 ws091 kernel: [201323.374488] kthread+0x105/0x140
Oct 29 13:56:16 ws091 kernel: [201323.374490] ? rcu_barrier_sched+0x10/0x10
Oct 29 13:56:16 ws091 kernel: [201323.374491] ?
kthread_destroy_worker+0x50/0x50
Oct 29 13:56:16 ws091 kernel: [201323.374493] ret_from_fork+0x35/0x40
Oct 29 13:57:58 ws091 kernel: [201426.098277] INFO: task jbd2/vdc1-8:1412
blocked for more than 120 seconds.
Oct 29 13:57:58 ws091 kernel: [201426.107666] Not tainted
4.15.0-38-generic #41~16.04.1-Ubuntu
Oct 29 13:57:58 ws091 kernel: [201426.110161] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 13:57:58 ws091 kernel: [201426.113093] jbd2/vdc1-8 D 0 1412
2 0x80000000
Oct 29 13:57:58 ws091 kernel: [201426.113099] Call Trace:
Oct 29 13:57:58 ws091 kernel: [201426.113115] __schedule+0x3d6/0x8b0
Oct 29 13:57:58 ws091 kernel: [201426.113119] schedule+0x36/0x80
Oct 29 13:57:58 ws091 kernel: [201426.113126]
jbd2_journal_commit_transaction+0x241/0x1760
Oct 29 13:57:58 ws091 kernel: [201426.113133] ? __switch_to_asm+0x34/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113136] ? __switch_to_asm+0x40/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113140] ? __switch_to_asm+0x34/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113143] ? __switch_to_asm+0x40/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113146] ? __switch_to_asm+0x34/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113149] ? __switch_to_asm+0x40/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113153] ? __switch_to_asm+0x40/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113156] ? __switch_to_asm+0x34/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113159] ? __switch_to_asm+0x34/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113163] ? __switch_to_asm+0x40/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113166] ? __switch_to_asm+0x40/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113174] ? wait_woken+0x80/0x80
Oct 29 13:57:58 ws091 kernel: [201426.113191] ? __switch_to_asm+0x40/0x70
Oct 29 13:57:58 ws091 kernel: [201426.113202] ? lock_timer_base+0x75/0xa0
Oct 29 13:57:58 ws091 kernel: [201426.113209] kjournald2+0xc8/0x250
Oct 29 13:57:58 ws091 kernel: [201426.113212] ? kjournald2+0xc8/0x250
Oct 29 13:57:58 ws091 kernel: [201426.113215] ? wait_woken+0x80/0x80
Oct 29 13:57:58 ws091 kernel: [201426.113223] kthread+0x105/0x140
Oct 29 13:57:58 ws091 kernel: [201426.113226] ? commit_timeout+0x20/0x20
Oct 29 13:57:58 ws091 kernel: [201426.113230] ?
kthread_destroy_worker+0x50/0x50
Oct 29 13:57:58 ws091 kernel: [201426.113234] ret_from_fork+0x35/0x40
Oct 29 13:57:58 ws091 kernel: [201426.113343] INFO: task kworker/u112:1:45667
blocked for more than 120 seconds.
Oct 29 13:57:58 ws091 kernel: [201426.115925] Not tainted
4.15.0-38-generic #41~16.04.1-Ubuntu
Oct 29 13:57:58 ws091 kernel: [201426.118507] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 13:57:58 ws091 kernel: [201426.121226] kworker/u112:1 D 0 45667
2 0x80000000
Oct 29 13:57:58 ws091 kernel: [201426.121240] Workqueue: writeback wb_workfn
(flush-252:32)
Oct 29 13:57:58 ws091 kernel: [201426.121243] Call Trace:
Oct 29 13:57:58 ws091 kernel: [201426.121249] __schedule+0x3d6/0x8b0
Oct 29 13:57:58 ws091 kernel: [201426.121257] ?
blk_mq_sched_insert_requests+0x7a/0x80
Oct 29 13:57:58 ws091 kernel: [201426.121261] schedule+0x36/0x80
Oct 29 13:57:58 ws091 kernel: [201426.121270] wait_transaction_locked+0x8a/0xd0
Oct 29 13:57:58 ws091 kernel: [201426.121274] ? wait_woken+0x80/0x80
Oct 29 13:57:58 ws091 kernel: [201426.121278]
add_transaction_credits+0x1c1/0x2a0
Oct 29 13:57:58 ws091 kernel: [201426.121282] ? blk_mq_make_request+0x3ba/0x590
Oct 29 13:57:58 ws091 kernel: [201426.121286] start_this_handle+0x103/0x400
Oct 29 13:57:58 ws091 kernel: [201426.121290] ? _cond_resched+0x1a/0x50
Oct 29 13:57:58 ws091 kernel: [201426.121297] ? kmem_cache_alloc+0x10f/0x1c0
Oct 29 13:57:58 ws091 kernel: [201426.121301] jbd2__journal_start+0xdb/0x1f0
Oct 29 13:57:58 ws091 kernel: [201426.121308] ? ext4_writepages+0x533/0xe40
Oct 29 13:57:58 ws091 kernel: [201426.121315]
__ext4_journal_start_sb+0x6d/0x120
Oct 29 13:57:58 ws091 kernel: [201426.121319] ext4_writepages+0x533/0xe40
Oct 29 13:57:58 ws091 kernel: [201426.121329] do_writepages+0x1f/0x70
Oct 29 13:57:58 ws091 kernel: [201426.121332] ? do_writepages+0x1f/0x70
Oct 29 13:57:58 ws091 kernel: [201426.121336]
__writeback_single_inode+0x45/0x340
Oct 29 13:57:58 ws091 kernel: [201426.121339] writeback_sb_inodes+0x26a/0x590
Oct 29 13:57:58 ws091 kernel: [201426.121344] __writeback_inodes_wb+0x92/0xc0
Oct 29 13:57:58 ws091 kernel: [201426.121347] wb_writeback+0x274/0x330
Oct 29 13:57:58 ws091 kernel: [201426.121352] ? cpumask_next+0x1b/0x20
Oct 29 13:57:58 ws091 kernel: [201426.121355] wb_workfn+0x369/0x460
Oct 29 13:57:58 ws091 kernel: [201426.121358] ? wb_workfn+0x369/0x460
Oct 29 13:57:58 ws091 kernel: [201426.121363] process_one_work+0x14d/0x410
Oct 29 13:57:58 ws091 kernel: [201426.121366] worker_thread+0x4b/0x460
Oct 29 13:57:58 ws091 kernel: [201426.121370] kthread+0x105/0x140
Oct 29 13:57:58 ws091 kernel: [201426.121372] ? process_one_work+0x410/0x410
Oct 29 13:57:58 ws091 kernel: [201426.121376] ?
kthread_destroy_worker+0x50/0x50
Oct 29 13:57:58 ws091 kernel: [201426.121380] ret_from_fork+0x35/0x40
Oct 29 13:57:58 ws091 kernel: [201426.121390] INFO: task redis-server:57001
blocked for more than 120 seconds.
Oct 29 13:57:58 ws091 kernel: [201426.123886] Not tainted
4.15.0-38-generic #41~16.04.1-Ubuntu
Oct 29 13:57:58 ws091 kernel: [201426.125471] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 13:57:58 ws091 kernel: [201426.127067] redis-server D 0 57001
2236 0x00000000
Oct 29 13:57:58 ws091 kernel: [201426.127070] Call Trace:
Oct 29 13:57:58 ws091 kernel: [201426.127076] __schedule+0x3d6/0x8b0
Oct 29 13:57:58 ws091 kernel: [201426.127078] schedule+0x36/0x80
Oct 29 13:57:58 ws091 kernel: [201426.127082] wait_transaction_locked+0x8a/0xd0
Oct 29 13:57:58 ws091 kernel: [201426.127085] ? wait_woken+0x80/0x80
Oct 29 13:57:58 ws091 kernel: [201426.127089]
add_transaction_credits+0x1c1/0x2a0
Oct 29 13:57:58 ws091 kernel: [201426.127093] ?
radix_tree_lookup_slot+0x22/0x50
Oct 29 13:57:58 ws091 kernel: [201426.127099] ? find_get_entry+0x1e/0x110
Oct 29 13:57:58 ws091 kernel: [201426.127102] start_this_handle+0x103/0x400
Oct 29 13:57:58 ws091 kernel: [201426.127104] ? _cond_resched+0x1a/0x50
Oct 29 13:57:58 ws091 kernel: [201426.127107] ? kmem_cache_alloc+0x10f/0x1c0
Oct 29 13:57:58 ws091 kernel: [201426.127110] jbd2__journal_start+0xdb/0x1f0
Oct 29 13:57:58 ws091 kernel: [201426.127113] ? __ext4_new_inode+0x747/0x13e0
Oct 29 13:57:58 ws091 kernel: [201426.127116]
__ext4_journal_start_sb+0x6d/0x120
Oct 29 13:57:58 ws091 kernel: [201426.127119] __ext4_new_inode+0x747/0x13e0
Oct 29 13:57:58 ws091 kernel: [201426.127123] ext4_create+0x112/0x1b0
Oct 29 13:57:58 ws091 kernel: [201426.127130] path_openat+0x12e6/0x13f0
Oct 29 13:57:58 ws091 kernel: [201426.127142] do_filp_open+0x99/0x110
Oct 29 13:57:58 ws091 kernel: [201426.127149] ? __check_object_size+0xfc/0x1a0
Oct 29 13:57:58 ws091 kernel: [201426.127163] ? __alloc_fd+0x46/0x170
Oct 29 13:57:58 ws091 kernel: [201426.127172] do_sys_open+0x12d/0x290
Oct 29 13:57:58 ws091 kernel: [201426.127182] ? do_sys_open+0x12d/0x290
Oct 29 13:57:58 ws091 kernel: [201426.127198] SyS_open+0x1e/0x20
Oct 29 13:57:58 ws091 kernel: [201426.127212] do_syscall_64+0x73/0x130
Oct 29 13:57:58 ws091 kernel: [201426.127221]
entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Oct 29 13:57:58 ws091 kernel: [201426.127231] RIP: 0033:0x7f2c7a09f05d
Oct 29 13:57:58 ws091 kernel: [201426.127234] RSP: 002b:00007ffd4f460e60
EFLAGS: 00000293 ORIG_RAX: 0000000000000002
Oct 29 13:57:58 ws091 kernel: [201426.127246] RAX: ffffffffffffffda RBX:
00007f2b899ccc00 RCX: 00007f2c7a09f05d
Oct 29 13:57:58 ws091 kernel: [201426.127250] RDX: 00000000000001b6 RSI:
0000000000000241 RDI: 00007ffd4f460fb0
Oct 29 13:57:58 ws091 kernel: [201426.127256] RBP: 00007ffd4f460fb0 R08:
0000000000000004 R09: 0000000000000001
Oct 29 13:57:58 ws091 kernel: [201426.127259] R10: 0000000000000240 R11:
0000000000000293 R12: 0000559b5a3c52a2
Oct 29 13:57:58 ws091 kernel: [201426.127264] R13: 0000000000000001 R14:
00007f2c794101f0 R15: 0000000000000001
** Changed in: linux (Ubuntu)
Status: Incomplete => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1800509
Title:
unable to handle kernel NULL pointer dereference at 0000000000000007
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800509/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
