Hi Ralf, Thanks for the reply!
>Did you use jailhouse-config-create? I am using `jailhouse config create` to generate the sysconfig.c file. >You can use the --mem-hv option to increate the memory. Try, for example, 32MiB and see if it works. I tried with 32MiB. It worked. I am not getting -ENOMEM anymore. The driver prints "The Jailhouse is opening" on dmesg. However, right after that the CPUs get stuck, and I get rcu_sched detected stalls. The system is completely irresponsive. I attached a text file containing the full output from dmesg. Here is the initial part: [ 434.792008] The Jailhouse is opening. [ 455.787315] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 455.793303] rcu: 1-...0: (839 GPs behind) idle=c2a/1/0x4000000000000000 softirq=681/681 fqs=1827 [ 455.802292] rcu: 2-...0: (144 GPs behind) idle=812/1/0x4000000000000000 softirq=905/905 fqs=1827 [ 455.811276] rcu: 3-...0: (144 GPs behind) idle=eaa/1/0x4000000000000000 softirq=719/719 fqs=1827 [ 455.820266] rcu: 4-...0: (1 GPs behind) idle=c2e/1/0x4000000000000000 softirq=1324/1324 fqs=1827 [ 455.829252] rcu: 5-...0: (144 GPs behind) idle=41a/1/0x4000000000000000 softirq=556/556 fqs=1827 [ 455.838238] rcu: 6-...0: (144 GPs behind) idle=912/1/0x4000000000000000 softirq=777/777 fqs=1827 [ 455.847218] rcu: 7-...0: (144 GPs behind) idle=5e6/1/0x4000000000000000 softirq=2409/2410 fqs=1827 [ 455.856404] (detected by 87, t=5253 jiffies, g=48537, q=364) [ 455.862170] Sending NMI from CPU 87 to CPUs 1: [ 465.776884] Sending NMI from CPU 87 to CPUs 2: [ 467.182686] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:1:7] [ 467.189857] Modules linked in: jailhouse(O) nf_conntrack_netlink xfrm_user xt_addrtype br_netfilter xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_natp [ 467.189928] binfmt_misc configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 libcrc32c crc32c_generic async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq ] [ 467.320567] CPU: 0 PID: 7 Comm: kworker/0:1 Tainted: G O 5.10.0 #3 [ 467.328767] Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020 [ 467.337154] Workqueue: events drm_fb_helper_dirty_work [drm_kms_helper] [ 467.344501] RIP: 0010:smp_call_function_many_cond+0x289/0x2d0 [ 467.350979] Code: e8 1c 8a 39 00 3b 05 0a c1 74 01 89 c7 0f 83 0b fe ff ff 48 63 c7 49 8b 16 48 03 14 c5 00 d9 99 9c 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 20 cf 07 9d 4c 89 fe 44 7 [ 467.371232] RSP: 0018:ffffa7d78015fcd8 EFLAGS: 00000202 [ 467.377220] RAX: 0000000000000011 RBX: 0000000000031280 RCX: 0000000000000001 [ 467.385123] RDX: ffff964f1fa31280 RSI: 0000000000000000 RDI: 0000000000000001 [ 467.393024] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001 [ 467.400928] R10: 0000000000000002 R11: 0000000000000002 R12: 0000000000000000 [ 467.408836] R13: 000000000000007f R14: ffff962f1f42c9c0 R15: 0000000000000080 [ 467.416737] FS: 0000000000000000(0000) GS:ffff962f1f400000(0000) knlGS:0000000000000000 [ 467.425604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 467.432127] CR2: 0000000000000000 CR3: 00000010987ea000 CR4: 00000000003506f0 [ 467.440045] Call Trace: [ 467.443289] ? tlbflush_read_file+0x70/0x70 [ 467.448266] ? tlbflush_read_file+0x70/0x70 [ 467.453242] on_each_cpu+0x2b/0x60 [ 467.457437] __purge_vmap_area_lazy+0x5d/0x680 [ 467.462679] ? _cond_resched+0x16/0x40 [ 467.467224] ? unmap_kernel_range_noflush+0x2fa/0x380 [ 467.473072] free_vmap_area_noflush+0xe7/0x100 [ 467.478315] remove_vm_area+0x96/0xa0 [ 467.482770] __vunmap+0x8d/0x290 [ 467.486792] drm_gem_shmem_vunmap+0x8b/0xa0 [drm] [ 467.492299] drm_client_buffer_vunmap+0x16/0x20 [drm] [ 467.498144] drm_fb_helper_dirty_work+0x187/0x1b0 [drm_kms_helper] [ 467.505118] process_one_work+0x1b6/0x350 [ 467.509912] worker_thread+0x53/0x3e0 [ 467.514361] ? process_one_work+0x350/0x350 [ 467.519338] kthread+0x11b/0x140 [ 467.523342] ? __kthread_bind_mask+0x60/0x60 [ 467.528389] ret_from_fork+0x22/0x30 Cheers Karim ________________________________ From: Ralf Ramsauer <[email protected]> Sent: 12 November 2022 17:47 To: Karim Manaouil <[email protected]>; [email protected] <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: Jailhouse: enter_hypervisor returns -ENOMEM This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe. On 12/11/2022 18:15, Karim Manaouil wrote: > Hi Jan, > > I am trying to deploy Jailhouse on an AMD EPYC with 128 CPUs (8 NUMA > nodes), running Linux kernel v5.10 (same version used by jailhouse CI > with same patches applied). > > `jailhouse hardware check` return that everything is ok and that "Check > passed!". > > Memory was reserved via `memmap=0x5200000$0x3a000000` > > However, enter_hypervisor() [1] fails when entry() is called on every > cpu and return -ENOMEM as error_code. Try to reserve more memory. Maybe the default size of 6MiB for HV memory is insufficient for 128 CPUs. Did you use jailhouse-config-create? You can use the --mem-hv option to increate the memory. Try, for example, 32MiB and see if it works. Ralf > > Do you possibly know where could the issue come from? > > Best > Karim > > [1] > https://github.com/siemens/jailhouse/blob/c7a1b6971ac15e4be8a0918b9bef6e2cbd99f9fc/driver/main.c#L251 > > <https://github.com/siemens/jailhouse/blob/c7a1b6971ac15e4be8a0918b9bef6e2cbd99f9fc/driver/main.c#L251> > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. Is e buidheann carthannais > a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh > SC005336. > > -- > You received this message because you are subscribed to the Google > Groups "Jailhouse" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jailhouse-dev/AM0PR05MB6018F1663ABE61DA3C697CA4A9039%40AM0PR05MB6018.eurprd05.prod.outlook.com > > <https://groups.google.com/d/msgid/jailhouse-dev/AM0PR05MB6018F1663ABE61DA3C697CA4A9039%40AM0PR05MB6018.eurprd05.prod.outlook.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/AM0PR05MB601899CAC211F4C81A9757DFA9029%40AM0PR05MB6018.eurprd05.prod.outlook.com.
[ 434.792008] The Jailhouse is opening. [ 455.787315] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 455.793303] rcu: 1-...0: (839 GPs behind) idle=c2a/1/0x4000000000000000 softirq=681/681 fqs=1827 [ 455.802292] rcu: 2-...0: (144 GPs behind) idle=812/1/0x4000000000000000 softirq=905/905 fqs=1827 [ 455.811276] rcu: 3-...0: (144 GPs behind) idle=eaa/1/0x4000000000000000 softirq=719/719 fqs=1827 [ 455.820266] rcu: 4-...0: (1 GPs behind) idle=c2e/1/0x4000000000000000 softirq=1324/1324 fqs=1827 [ 455.829252] rcu: 5-...0: (144 GPs behind) idle=41a/1/0x4000000000000000 softirq=556/556 fqs=1827 [ 455.838238] rcu: 6-...0: (144 GPs behind) idle=912/1/0x4000000000000000 softirq=777/777 fqs=1827 [ 455.847218] rcu: 7-...0: (144 GPs behind) idle=5e6/1/0x4000000000000000 softirq=2409/2410 fqs=1827 [ 455.856404] (detected by 87, t=5253 jiffies, g=48537, q=364) [ 455.862170] Sending NMI from CPU 87 to CPUs 1: [ 465.776884] Sending NMI from CPU 87 to CPUs 2: [ 467.182686] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:1:7] [ 467.189857] Modules linked in: jailhouse(O) nf_conntrack_netlink xfrm_user xt_addrtype br_netfilter xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_natp [ 467.189928] binfmt_misc configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 libcrc32c crc32c_generic async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq ] [ 467.320567] CPU: 0 PID: 7 Comm: kworker/0:1 Tainted: G O 5.10.0 #3 [ 467.328767] Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020 [ 467.337154] Workqueue: events drm_fb_helper_dirty_work [drm_kms_helper] [ 467.344501] RIP: 0010:smp_call_function_many_cond+0x289/0x2d0 [ 467.350979] Code: e8 1c 8a 39 00 3b 05 0a c1 74 01 89 c7 0f 83 0b fe ff ff 48 63 c7 49 8b 16 48 03 14 c5 00 d9 99 9c 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 20 cf 07 9d 4c 89 fe 44 7 [ 467.371232] RSP: 0018:ffffa7d78015fcd8 EFLAGS: 00000202 [ 467.377220] RAX: 0000000000000011 RBX: 0000000000031280 RCX: 0000000000000001 [ 467.385123] RDX: ffff964f1fa31280 RSI: 0000000000000000 RDI: 0000000000000001 [ 467.393024] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001 [ 467.400928] R10: 0000000000000002 R11: 0000000000000002 R12: 0000000000000000 [ 467.408836] R13: 000000000000007f R14: ffff962f1f42c9c0 R15: 0000000000000080 [ 467.416737] FS: 0000000000000000(0000) GS:ffff962f1f400000(0000) knlGS:0000000000000000 [ 467.425604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 467.432127] CR2: 0000000000000000 CR3: 00000010987ea000 CR4: 00000000003506f0 [ 467.440045] Call Trace: [ 467.443289] ? tlbflush_read_file+0x70/0x70 [ 467.448266] ? tlbflush_read_file+0x70/0x70 [ 467.453242] on_each_cpu+0x2b/0x60 [ 467.457437] __purge_vmap_area_lazy+0x5d/0x680 [ 467.462679] ? _cond_resched+0x16/0x40 [ 467.467224] ? unmap_kernel_range_noflush+0x2fa/0x380 [ 467.473072] free_vmap_area_noflush+0xe7/0x100 [ 467.478315] remove_vm_area+0x96/0xa0 [ 467.482770] __vunmap+0x8d/0x290 [ 467.486792] drm_gem_shmem_vunmap+0x8b/0xa0 [drm] [ 467.492299] drm_client_buffer_vunmap+0x16/0x20 [drm] [ 467.498144] drm_fb_helper_dirty_work+0x187/0x1b0 [drm_kms_helper] [ 467.505118] process_one_work+0x1b6/0x350 [ 467.509912] worker_thread+0x53/0x3e0 [ 467.514361] ? process_one_work+0x350/0x350 [ 467.519338] kthread+0x11b/0x140 [ 467.523342] ? __kthread_bind_mask+0x60/0x60 [ 467.528389] ret_from_fork+0x22/0x30 [ 475.689990] Sending NMI from CPU 87 to CPUs 3: [ 485.604399] Sending NMI from CPU 87 to CPUs 4: [ 495.182798] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/0:1:7] [ 495.191179] Modules linked in: jailhouse(O) nf_conntrack_netlink xfrm_user xt_addrtype br_netfilter xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_natp [ 495.191239] binfmt_misc configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 libcrc32c crc32c_generic async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq ] [ 495.326354] CPU: 0 PID: 7 Comm: kworker/0:1 Tainted: G O L 5.10.0 #3 [ 495.334629] Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020 [ 495.343077] Workqueue: events drm_fb_helper_dirty_work [drm_kms_helper] [ 495.350485] RIP: 0010:smp_call_function_many_cond+0x28c/0x2d0 [ 495.357011] Code: 39 00 3b 05 0a c1 74 01 89 c7 0f 83 0b fe ff ff 48 63 c7 49 8b 16 48 03 14 c5 00 d9 99 9c 8b 42 08 a8 01 74 09 f3 90 8b 42 08 <a8> 01 75 f7 eb c9 48 c7 c2 20 cf 07 9d 4c 89 fe 44 89 f7 e8 a [ 495.377307] RSP: 0018:ffffa7d78015fcd8 EFLAGS: 00000202 [ 495.383307] RAX: 0000000000000011 RBX: 0000000000031280 RCX: 0000000000000001 [ 495.391258] RDX: ffff964f1fa31280 RSI: 0000000000000000 RDI: 0000000000000001 [ 495.399209] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001 [ 495.407156] R10: 0000000000000002 R11: 0000000000000002 R12: 0000000000000000 [ 495.415084] R13: 000000000000007f R14: ffff962f1f42c9c0 R15: 0000000000000080 [ 495.423003] FS: 0000000000000000(0000) GS:ffff962f1f400000(0000) knlGS:0000000000000000 [ 495.431870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 495.438381] CR2: 0000000000000000 CR3: 00000010987ea000 CR4: 00000000003506f0 [ 495.446287] Call Trace: [ 495.449532] ? tlbflush_read_file+0x70/0x70 [ 495.454525] ? tlbflush_read_file+0x70/0x70 [ 495.459506] on_each_cpu+0x2b/0x60 [ 495.463684] __purge_vmap_area_lazy+0x5d/0x680 [ 495.468906] ? _cond_resched+0x16/0x40 [ 495.473424] ? unmap_kernel_range_noflush+0x2fa/0x380 [ 495.479254] free_vmap_area_noflush+0xe7/0x100 [ 495.484475] remove_vm_area+0x96/0xa0 [ 495.488916] __vunmap+0x8d/0x290 [ 495.492930] drm_gem_shmem_vunmap+0x8b/0xa0 [drm] [ 495.498416] drm_client_buffer_vunmap+0x16/0x20 [drm] [ 495.504238] drm_fb_helper_dirty_work+0x187/0x1b0 [drm_kms_helper] [ 495.511192] process_one_work+0x1b6/0x350 [ 495.515976] worker_thread+0x53/0x3e0 [ 495.518410] Sending NMI from CPU 87 to CPUs 5: [ 495.520415] ? process_one_work+0x350/0x350 [ 495.520420] kthread+0x11b/0x140 [ 495.535956] ? __kthread_bind_mask+0x60/0x60 [ 495.541111] ret_from_fork+0x22/0x30 [ 505.434863] Sending NMI from CPU 87 to CPUs 6: [ 515.349197] Sending NMI from CPU 87 to CPUs 7: [ 518.847559] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 525.263176] rcu: rcu_sched kthread starved for 17167 jiffies! g48537 f0x0 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=48 [ 525.263181] rcu: 1-...0: (839 GPs behind) idle=c2a/1/0x4000000000000000 softirq=681/681 fqs=1828 [ 525.263187] rcu: 2-...0: (144 GPs behind) idle=812/1/0x4000000000000000 softirq=905/905 fqs=1828 [ 525.263190] watchdog: BUG: soft lockup - CPU#48 stuck for 66s! [rcu_sched:12] [ 525.263191] Modules linked in: jailhouse(O) nf_conntrack_netlink xfrm_user xt_addrtype br_netfilter xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_natp [ 525.263234] binfmt_misc configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 libcrc32c crc32c_generic async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq ] [ 525.263250] CPU: 48 PID: 12 Comm: rcu_sched Tainted: G O L 5.10.0 #3 [ 525.263252] Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020 [ 525.263255] RIP: 0010:_raw_spin_unlock_irqrestore+0x11/0x20 [ 525.263258] Code: e0 4c 39 f0 76 d8 80 4d 00 04 eb 92 cc cc cc cc cc cc cc cc cc cc cc cc cc 0f 1f 44 00 00 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 7 [ 525.263259] RSP: 0018:ffffa7d78020fe60 EFLAGS: 00000246 [ 525.263261] RAX: 0000000000000040 RBX: ffffffff9ced3280 RCX: 0000000000000040 [ 525.263262] RDX: 00000000000000fe RSI: 0000000000000246 RDI: 0000000000000246 [ 525.263264] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000040 [ 525.263264] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9ced32a0 [ 525.263265] R13: 0000000000000000 R14: 000000000002c780 R15: 0000000000000040 [ 525.263267] FS: 0000000000000000(0000) GS:ffff962f1f580000(0000) knlGS:0000000000000000 [ 525.263268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 525.263269] CR2: 0000000000000000 CR3: 000000010e312000 CR4: 00000000003506e0 [ 525.263269] Call Trace: [ 525.263274] force_qs_rnp+0x178/0x190 [ 525.263276] ? kfree_rcu_shrink_count+0x50/0x50 [ 525.263278] rcu_gp_kthread+0x59c/0xa70 [ 525.263280] ? rcu_cpu_kthread+0x190/0x190 [ 525.263282] kthread+0x11b/0x140 [ 525.263283] ? __kthread_bind_mask+0x60/0x60 [ 525.263285] ret_from_fork+0x22/0x30 [ 525.274791] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 525.274796] rcu: RCU grace-period kthread stack dump: [ 525.284962] rcu: 3-...0: (144 GPs behind) idle=eaa/1/0x4000000000000000 softirq=719/719 fqs=1829 [ 525.295128] task:rcu_sched state:R running task [ 525.303473] rcu: 4-...0: (1 GPs behind) idle=c2e/1/0x4000000000000000 softirq=1324/1324 fqs=1829 [ 525.303480] rcu: 5-...0: (144 GPs behind) idle=41a/1/0x4000000000000000 softirq=556/556 fqs=1829 [ 525.397853] stack: 0 pid: 12 ppid: 2 flags:0x00004000 [ 525.397858] Call Trace: [ 525.439766] rcu: 6-...0: (144 GPs behind) idle=912/1/0x4000000000000000 softirq=777/777 fqs=1829 [ 525.439771] rcu: 7-...0: (144 GPs behind) idle=5e6/1/0x4000000000000000 softirq=2409/2410 fqs=1829 [ 525.448572] ? lock_timer_base+0x61/0x80 [ 525.457520] (detected by 0, t=22650 jiffies, g=48537, q=1576) [ 525.464385] ? schedule+0x46/0xb0 [ 525.464389] ? schedule_timeout+0x8b/0x140 [ 525.485450] Sending NMI from CPU 0 to CPUs 1: [ 525.699665] ? force_qs_rnp+0x6c/0x190 [ 525.704206] ? kfree_rcu_shrink_count+0x50/0x50 [ 525.709513] ? rcu_gp_kthread+0x59c/0xa70 [ 525.714276] ? rcu_cpu_kthread+0x190/0x190 [ 525.719110] ? kthread+0x11b/0x140 [ 525.723228] ? __kthread_bind_mask+0x60/0x60 [ 525.728200] ? ret_from_fork+0x22/0x30 [ 535.411119] Sending NMI from CPU 0 to CPUs 2: [ 545.342328] Sending NMI from CPU 0 to CPUs 3: [ 555.273185] Sending NMI from CPU 0 to CPUs 4: [ 565.203956] Sending NMI from CPU 0 to CPUs 5: [ 575.134679] Sending NMI from CPU 0 to CPUs 6: [ 579.184057] watchdog: BUG: soft lockup - CPU#50 stuck for 22s! [dockerd:3423] [ 579.184204] watchdog: BUG: soft lockup - CPU#58 stuck for 22s! [dockerd:3430] [ 579.192094] Modules linked in: [ 579.200090] Modules linked in: [ 579.200095] jailhouse(O) [ 579.203821] jailhouse(O) [ 579.207665] nf_conntrack_netlink [ 579.211045] nf_conntrack_netlink [ 579.214421] xfrm_user [ 579.218501] xfrm_user [ 579.222572] xt_addrtype [ 579.225679] xt_addrtype [ 579.228769] br_netfilter [ 579.232007] br_netfilter [ 579.235211] xt_CHECKSUM [ 579.238500] xt_CHECKSUM [ 579.241773] xt_MASQUERADE xt_conntrack [ 579.244962] xt_MASQUERADE [ 579.248144] ipt_REJECT nf_reject_ipv4 [ 579.252637] xt_conntrack [ 579.255992] xt_tcpudp [ 579.260397] ipt_REJECT [ 579.263668] nft_compat nft_chain_nat [ 579.266643] nf_reject_ipv4 [ 579.269685] nf_nat [ 579.273934] xt_tcpudp [ 579.277307] nf_conntrack nf_defrag_ipv6 [ 579.279963] nft_compat nft_chain_nat [ 579.282873] nf_defrag_ipv4 [ 579.287329] nf_nat nf_conntrack [ 579.291517] nft_counter [ 579.294807] nf_defrag_ipv6 [ 579.298533] nf_tables nfnetlink [ 579.301540] nf_defrag_ipv4 [ 579.304792] bridge [ 579.308470] nft_counter [ 579.311713] stp llc [ 579.314265] nf_tables [ 579.317239] cpufreq_ondemand cpufreq_conservative [ 579.319868] nfnetlink [ 579.322657] cpufreq_powersave cpufreq_userspace [ 579.327890] bridge [ 579.330689] mlx5_ib [ 579.335745] stp llc [ 579.338290] overlay msr [ 579.340905] cpufreq_ondemand cpufreq_conservative [ 579.343522] esp6_offload [ 579.346486] cpufreq_powersave cpufreq_userspace [ 579.351726] esp6 ib_uverbs [ 579.354807] mlx5_ib [ 579.359849] esp4_offload [ 579.363072] overlay msr [ 579.365673] esp4 xfrm_algo [ 579.368708] esp6_offload [ 579.371651] ib_core [ 579.374850] esp6 ib_uverbs [ 579.377876] mlx5_core [ 579.380480] esp4_offload [ 579.383676] amd64_edac_mod edac_mce_amd [ 579.386442] esp4 xfrm_algo [ 579.389475] nls_ascii kvm_amd [ 579.393817] ib_core [ 579.397032] nls_cp437 [ 579.400509] mlx5_core amd64_edac_mod [ 579.403113] vfat fat [ 579.405898] edac_mce_amd nls_ascii [ 579.409986] kvm [ 579.412685] kvm_amd [ 579.416588] snd_pcm xhci_pci [ 579.418852] nls_cp437 vfat [ 579.421461] xhci_hcd [ 579.424855] fat [ 579.428062] tg3 [ 579.430746] kvm [ 579.432998] bnxt_en [ 579.435247] snd_pcm [ 579.437475] evdev usbcore [ 579.440062] xhci_pci [ 579.442635] snd_timer irqbypass [ 579.445732] xhci_hcd [ 579.448393] dcdbas [ 579.452017] tg3 [ 579.454677] crc32_pclmul nvme [ 579.457170] bnxt_en [ 579.459391] ahci [ 579.462835] evdev [ 579.465414] libphy snd [ 579.467730] usbcore [ 579.470114] ghash_clmulni_intel [ 579.472927] snd_timer irqbypass [ 579.475475] ipmi_ssif libahci [ 579.479065] dcdbas crc32_pclmul [ 579.482668] nvme_core [ 579.486091] nvme [ 579.489688] aesni_intel mlxfw [ 579.492407] ahci [ 579.494689] crypto_simd [ 579.498095] libphy [ 579.500372] sp5100_tco [ 579.503255] snd [ 579.505699] soundcore [ 579.508493] ghash_clmulni_intel [ 579.510683] ptp [ 579.513382] ipmi_ssif libahci [ 579.516960] libata [ 579.519150] nvme_core aesni_intel [ 579.522556] ccp [ 579.525002] mlxfw [ 579.528752] cryptd [ 579.530931] crypto_simd [ 579.533291] glue_helper [ 579.535734] sp5100_tco soundcore [ 579.538605] pps_core watchdog [ 579.541473] ptp [ 579.545127] sg [ 579.548526] libata [ 579.550704] pcspkr [ 579.552792] ccp [ 579.555223] pci_hyperv_intf acpi_ipmi [ 579.557658] cryptd [ 579.559831] efi_pstore [ 579.563917] glue_helper [ 579.566366] rapl [ 579.569157] pps_core watchdog [ 579.572041] i2c_piix4 [ 579.574313] sg [ 579.577717] rng_core k10temp [ 579.580426] pcspkr [ 579.582524] ipmi_si acpi_power_meter [ 579.585841] pci_hyperv_intf acpi_ipmi [ 579.588298] ipmi_devintf [ 579.592319] efi_pstore [ 579.596425] ipmi_msghandler button [ 579.599408] rapl [ 579.602218] nfsd [ 579.606077] i2c_piix4 [ 579.608370] auth_rpcgss [ 579.610667] rng_core k10temp [ 579.613390] nfs_acl [ 579.616284] ipmi_si acpi_power_meter [ 579.619616] lockd [ 579.622160] ipmi_devintf [ 579.626191] grace [ 579.628577] ipmi_msghandler button [ 579.631572] sunrpc [ 579.633956] nfsd [ 579.637818] sch_fq fuse [ 579.640296] auth_rpcgss [ 579.642600] tcp_htcp [ 579.645508] nfs_acl [ 579.648420] binfmt_misc configfs [ 579.651071] lockd [ 579.653627] efivarfs [ 579.657321] grace [ 579.659716] ip_tables [ 579.662357] sunrpc [ 579.664741] x_tables [ 579.667476] sch_fq fuse [ 579.669953] autofs4 [ 579.672592] tcp_htcp [ 579.675492] ext4 crc16 [ 579.678043] binfmt_misc configfs [ 579.680675] mbcache jbd2 [ 579.683485] efivarfs [ 579.687169] raid10 [ 579.690163] ip_tables [ 579.692803] raid456 [ 579.695279] x_tables [ 579.698006] libcrc32c [ 579.700552] autofs4 [ 579.703187] crc32c_generic [ 579.705908] ext4 crc16 [ 579.708458] async_raid6_recov [ 579.711610] mbcache jbd2 [ 579.714419] async_memcpy [ 579.717839] raid10 [ 579.720830] async_pq [ 579.723820] raid456 [ 579.726284] async_xor xor [ 579.728907] libcrc32c [ 579.731446] async_tx raid6_pq [ 579.734506] crc32c_generic [ 579.737219] raid1 [ 579.740627] async_raid6_recov [ 579.743772] raid0 [ 579.746137] async_memcpy [ 579.749548] multipath [ 579.751910] async_pq async_xor [ 579.754883] linear md_mod [ 579.757584] xor [ 579.761071] sd_mod [ 579.764126] async_tx raid6_pq [ 579.766315] t10_pi crc_t10dif [ 579.768760] raid1 [ 579.772155] crct10dif_generic [ 579.775554] raid0 [ 579.777908] mgag200 [ 579.781311] multipath [ 579.783665] drm_kms_helper [ 579.786194] linear md_mod [ 579.788898] cec [ 579.792031] sd_mod [ 579.795080] rc_core [ 579.797262] t10_pi crc_t10dif [ 579.799702] drm [ 579.802227] crct10dif_generic mgag200 [ 579.805629] megaraid_sas scsi_mod [ 579.807812] drm_kms_helper [ 579.811914] crct10dif_pclmul [ 579.815669] cec [ 579.818813] crct10dif_common crc32c_intel [ 579.822134] rc_core drm [ 579.824326] i2c_algo_bit [last unloaded: jailhouse] [ 579.828781] megaraid_sas scsi_mod [ 579.831680] [ 579.831684] CPU: 50 PID: 3423 Comm: dockerd Tainted: G O L 5.10.0 #3 [ 579.837013] crct10dif_pclmul [ 579.840790] Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020 [ 579.842655] crct10dif_common crc32c_intel [ 579.850547] RIP: 0010:smp_call_function_many_cond+0x28e/0x2d0 [ 579.853913] i2c_algo_bit [last unloaded: jailhouse] [ 579.862006] Code: 3b 05 0a c1 74 01 89 c7 0f 83 0b fe ff ff 48 63 c7 49 8b 16 48 03 14 c5 00 d9 99 9c 8b 42 08 a8 01 74 09 f3 90 8b 42 08 a8 01 <75> f7 eb c9 48 c7 c2 20 cf 07 9d 4c 89 fe 44 89 f7 e8 ac 8a 0 [ 579.866540] [ 579.866544] CPU: 58 PID: 3430 Comm: dockerd Tainted: G O L 5.10.0 #3 [ 579.872739] RSP: 0000:ffffa7d7a240bce0 EFLAGS: 00000202 [ 579.878164] Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020 [ 579.897890] RAX: 0000000000000011 RBX: 0000000000034060 RCX: 0000000000000009 [ 579.897893] RDX: ffff964f1fa74060 RSI: 0000000000000000 RDI: 0000000000000009 [ 579.899941] RIP: 0010:smp_call_function_many_cond+0x289/0x2d0 [ 579.907964] RBP: ffffffff9b8747a0 R08: 0000000000000000 R09: 0000000000000009 [ 579.907968] R10: 0000000000000002 R11: 0000000000000002 R12: ffff96371fbab3c0 [ 579.913757] Code: e8 1c 8a 39 00 3b 05 0a c1 74 01 89 c7 0f 83 0b fe ff ff 48 63 c7 49 8b 16 48 03 14 c5 00 d9 99 9c 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 20 cf 07 9d 4c 89 fe 44 7 [ 579.921998] R13: 000000000000007e R14: ffff96371fbac9c0 R15: 0000000000000080 [ 579.922001] FS: 00007fbd927fc700(0000) GS:ffff96371fb80000(0000) knlGS:0000000000000000 [ 579.929724] RSP: 0000:ffffa7d78f6dbce0 EFLAGS: 00000202 [ 579.937458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 579.937461] CR2: 000000c001a3a010 CR3: 00000010987ea000 CR4: 00000000003506e0 [ 579.943818] RAX: 0000000000000011 RBX: 0000000000034160 RCX: 0000000000000009 [ 579.943821] RDX: ffff964f1fa74160 RSI: 0000000000000000 RDI: 0000000000000009 [ 579.951581] Call Trace: [ 579.959344] RBP: ffffffff9b8747a0 R08: 0000000000000000 R09: 0000000000000009 [ 579.959348] R10: 0000000000000002 R11: 0000000000000002 R12: ffff96371fbeb3c0 [ 579.979396] ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0 [ 579.987239] R13: 000000000000007e R14: ffff96371fbec9c0 R15: 0000000000000080 [ 579.987243] FS: 00007fbd72ffd700(0000) GS:ffff96371fbc0000(0000) knlGS:0000000000000000 [ 579.996063] ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0 [ 580.002035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 580.002038] CR2: 000000c001d92010 CR3: 00000010987ea000 CR4: 00000000003506e0 [ 580.008535] ? x86_configure_nx+0x40/0x40 [ 580.008538] on_each_cpu_cond_mask+0x2f/0x80 [ 580.016437] Call Trace: [ 580.024346] flush_tlb_mm_range+0xbd/0xf0 [ 580.032264] ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0 [ 580.035510] ptep_clear_flush+0x55/0x60 [ 580.043457] ? flush_tlb_func_common.constprop.0+0x1e0/0x1e0 [ 580.051413] wp_page_copy+0x2e7/0x860 [ 580.057910] ? x86_configure_nx+0x40/0x40 [ 580.057914] on_each_cpu_cond_mask+0x2f/0x80 [ 580.065895] handle_mm_fault+0x11ec/0x1c30 [ 580.074847] flush_tlb_mm_range+0xbd/0xf0 [ 580.081390] do_user_addr_fault+0x1b8/0x3f0 [ 580.088040] ptep_clear_flush+0x55/0x60 [ 580.096085] exc_page_fault+0x78/0x160 [ 580.101010] wp_page_copy+0x2e7/0x860 [ 580.106211] ? asm_exc_page_fault+0x8/0x30 [ 580.109581] handle_mm_fault+0x11ec/0x1c30 [ 580.114515] asm_exc_page_fault+0x1e/0x30 [ 580.121107] do_user_addr_fault+0x1b8/0x3f0 [ 580.125886] RIP: 0033:0x562b35eb2c7f [ 580.132490] exc_page_fault+0x78/0x160 [ 580.137100] Code: fa 00 80 00 00 77 e1 48 89 4c 24 18 48 89 54 24 30 48 89 44 24 28 48 8b 54 24 10 48 8b 5a 18 48 8d 04 0b 48 89 44 24 38 84 02 <48> c7 40 10 00 00 00 00 e8 54 56 fe ff 48 8b 4c 24 18 48 85 5 [ 580.142063] ? asm_exc_page_fault+0x8/0x30 [ 580.147265] RSP: 002b:00007fbd927fbbd8 EFLAGS: 00010246 [ 580.152296] asm_exc_page_fault+0x1e/0x30 [ 580.157235] RAX: 000000c001a3a000 RBX: 000000c001a3a000 RCX: 0000000000000000 [ 580.157238] RDX: 00007fbeb47bbc20 RSI: 0000000000000081 RDI: 0000562b3913a230 [ 580.162346] RIP: 0033:0x562b35eb2c7f [ 580.167096] RBP: 00007fbd927fbc28 R08: 0000000000000000 R09: 0000000000000000 [ 580.167099] R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000004 [ 580.171753] Code: fa 00 80 00 00 77 e1 48 89 4c 24 18 48 89 54 24 30 48 89 44 24 28 48 8b 54 24 10 48 8b 5a 18 48 8d 04 0b 48 89 44 24 38 84 02 <48> c7 40 10 00 00 00 00 e8 54 56 fe ff 48 8b 4c 24 18 48 85 5 [ 580.176298] R13: 00000000000000c4 R14: 000000c000f6c4e0 R15: 0000562b390c0000 [ 580.302758] RSP: 002b:00007fbd72ffcbd8 EFLAGS: 00010246 [ 580.308625] RAX: 000000c001d92000 RBX: 000000c001d92000 RCX: 0000000000000000 [ 580.316410] RDX: 00007fbeb45fc6c0 RSI: 0000000000000004 RDI: 0000000000000001 [ 580.324189] RBP: 00007fbd72ffcc28 R08: 0000000000000004 R09: 000000c001d92000 [ 580.331957] R10: 0000000000000ecc R11: 0000000000000003 R12: 0000000000000004 [ 580.339725] R13: 0000000000000075 R14: 000000c001088820 R15: 0000000000802000 [ 581.863767] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 585.065362] Sending NMI from CPU 0 to CPUs 7: [ 594.996503] rcu: rcu_sched kthread starved for 17377 jiffies! g48537 f0x2 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=14 [ 594.996507] rcu: 1-...0: (839 GPs behind) idle=c2a/1/0x4000000000000000 softirq=681/681 fqs=1834 [ 594.996514] rcu: 2-...0: (144 GPs behind) idle=812/1/0x4000000000000000 softirq=905/905 fqs=1834 [ 595.007655] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 595.007659] rcu: RCU grace-period kthread stack dump: [ 595.017809] rcu: 3-...0: (144 GPs behind) idle=eaa/1/0x4000000000000000 softirq=719/719 fqs=1835 [ 595.017815] rcu: 4-...0: (1 GPs behind) idle=c2e/1/0x4000000000000000 softirq=1324/1324 fqs=1835 [ 595.027917] task:rcu_sched state:R [ 595.038073] rcu: 5-...0: (144 GPs behind) idle=41a/1/0x4000000000000000 softirq=556/556 fqs=1835 [ 595.038077] rcu: 6-...0: (144 GPs behind) idle=912/1/0x4000000000000000 softirq=777/777 fqs=1835 [ 595.044295] running task stack: 0 pid: 12 ppid: 2 flags:0x00004000 [ 595.054473] rcu: 7-...0: (144 GPs behind) idle=5e6/1/0x4000000000000000 softirq=2409/2410 fqs=1835 [ 595.064615] Call Trace: [ 595.064622] ? lock_timer_base+0x61/0x80 [ 595.069820] (detected by 87, t=40072 jiffies, g=48537, q=2933) [ 595.079978] ? schedule+0x46/0xb0 [ 595.090180] Sending NMI from CPU 87 to CPUs 1: [ 595.098894] ? _raw_spin_lock_irqsave+0x32/0x40 [ 595.098899] ? force_qs_rnp+0x6c/0x190 [ 595.143922] ? kfree_rcu_shrink_count+0x50/0x50 [ 595.149239] ? rcu_gp_kthread+0x59c/0xa70 [ 595.154007] ? rcu_cpu_kthread+0x190/0x190 [ 595.158845] ? kthread+0x11b/0x140 [ 595.162991] ? __kthread_bind_mask+0x60/0x60 [ 595.168008] ? ret_from_fork+0x22/0x30 [ 605.020389] Sending NMI from CPU 87 to CPUs 2: [ 614.937251] Sending NMI from CPU 87 to CPUs 3: [ 619.183203] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:1:7] [ 619.191624] Modules linked in: jailhouse(O) nf_conntrack_netlink xfrm_user xt_addrtype br_netfilter xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_natp [ 619.191683] binfmt_misc configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 libcrc32c crc32c_generic async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq ] [ 619.327296] CPU: 0 PID: 7 Comm: kworker/0:1 Tainted: G O L 5.10.0 #3 [ 619.335639] Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020 [ 619.344147] Workqueue: events drm_fb_helper_dirty_work [drm_kms_helper] [ 619.351619] RIP: 0010:smp_call_function_many_cond+0x289/0x2d0 [ 619.358226] Code: e8 1c 8a 39 00 3b 05 0a c1 74 01 89 c7 0f 83 0b fe ff ff 48 63 c7 49 8b 16 48 03 14 c5 00 d9 99 9c 8b 42 08 a8 01 74 09 f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 20 cf 07 9d 4c 89 fe 44 7 [ 619.378747] RSP: 0018:ffffa7d78015fcd8 EFLAGS: 00000202 [ 619.384860] RAX: 0000000000000011 RBX: 0000000000031280 RCX: 0000000000000001 [ 619.392872] RDX: ffff964f1fa31280 RSI: 0000000000000000 RDI: 0000000000000001 [ 619.400873] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001 [ 619.408877] R10: 0000000000000002 R11: 0000000000000002 R12: 0000000000000000 [ 619.416866] R13: 000000000000007f R14: ffff962f1f42c9c0 R15: 0000000000000080 [ 619.424837] FS: 0000000000000000(0000) GS:ffff962f1f400000(0000) knlGS:0000000000000000 [ 619.433755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 619.440337] CR2: 0000000000000000 CR3: 00000010987ea000 CR4: 00000000003506f0 [ 619.448306] Call Trace: [ 619.451574] ? tlbflush_read_file+0x70/0x70 [ 619.456582] ? tlbflush_read_file+0x70/0x70 [ 619.461563] on_each_cpu+0x2b/0x60 [ 619.465756] __purge_vmap_area_lazy+0x5d/0x680 [ 619.470968] ? _cond_resched+0x16/0x40 [ 619.475496] ? unmap_kernel_range_noflush+0x2fa/0x380 [ 619.481323] free_vmap_area_noflush+0xe7/0x100 [ 619.486545] remove_vm_area+0x96/0xa0 [ 619.490978] __vunmap+0x8d/0x290 [ 619.494986] drm_gem_shmem_vunmap+0x8b/0xa0 [drm] [ 619.500471] drm_client_buffer_vunmap+0x16/0x20 [drm] [ 619.506297] drm_fb_helper_dirty_work+0x187/0x1b0 [drm_kms_helper] [ 619.513250] process_one_work+0x1b6/0x350 [ 619.518033] worker_thread+0x53/0x3e0 [ 619.522468] ? process_one_work+0x350/0x350 [ 619.527427] kthread+0x11b/0x140 [ 619.531430] ? __kthread_bind_mask+0x60/0x60 [ 619.536477] ret_from_fork+0x22/0x30 [ 624.853616] Sending NMI from CPU 87 to CPUs 4: [ 634.770449] Sending NMI from CPU 87 to CPUs 5: [ 644.686890] Sending NMI from CPU 87 to CPUs 6: [ 654.603263] Sending NMI from CPU 87 to CPUs 7: [ 664.519595] rcu: rcu_sched kthread starved for 8788 jiffies! g48537 f0x2 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=14 [ 664.531228] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 664.541441] rcu: RCU grace-period kthread stack dump: [ 664.547539] task:rcu_sched state:R running task stack: 0 pid: 12 ppid: 2 flags:0x00004008 [ 664.558709] Call Trace: [ 664.561896] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 [ 664.567918] ? _raw_spin_unlock_irqrestore+0x11/0x20 [ 664.573576] ? force_qs_rnp+0x178/0x190 [ 664.578068] ? kfree_rcu_shrink_count+0x50/0x50 [ 664.583233] ? rcu_gp_kthread+0x59c/0xa70 [ 664.587874] ? rcu_cpu_kthread+0x190/0x190 [ 664.592586] ? kthread+0x11b/0x140 [ 664.596597] ? __kthread_bind_mask+0x60/0x60 [ 664.601467] ? ret_from_fork+0x22/0x30 [ 664.605803] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 664.612287] rcu: 1-...0: (839 GPs behind) idle=c2a/1/0x4000000000000000 softirq=681/681 fqs=1838 [ 664.621808] rcu: 2-...0: (144 GPs behind) idle=812/1/0x4000000000000000 softirq=905/905 fqs=1838 [ 664.631325] rcu: 3-...0: (144 GPs behind) idle=eaa/1/0x4000000000000000 softirq=719/719 fqs=1838 [ 664.640836] rcu: 4-...0: (1 GPs behind) idle=c2e/1/0x4000000000000000 softirq=1324/1324 fqs=1838 [ 664.650347] rcu: 5-...0: (144 GPs behind) idle=41a/1/0x4000000000000000 softirq=556/556 fqs=1838 [ 664.659869] rcu: 6-...0: (144 GPs behind) idle=912/1/0x4000000000000000 softirq=777/777 fqs=1838 [ 664.669373] rcu: 7-...0: (144 GPs behind) idle=5e6/1/0x4000000000000000 softirq=2409/2410 fqs=1838 [ 664.679071] (detected by 87, t=57456 jiffies, g=48537, q=3354) [ 664.685533] Sending NMI from CPU 87 to CPUs 1: [ 674.601594] Sending NMI from CPU 87 to CPUs 2: [ 684.517641] Sending NMI from CPU 87 to CPUs 3: [ 694.433683] Sending NMI from CPU 87 to CPUs 4: [ 704.349722] Sending NMI from CPU 87 to CPUs 5: [ 714.265763] Sending NMI from CPU 87 to CPUs 6: [ 724.181774] Sending NMI from CPU 87 to CPUs 7: [ 734.097777] rcu: rcu_sched kthread starved for 17378 jiffies! g48537 f0x2 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=14 [ 734.108637] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 734.118115] rcu: RCU grace-period kthread stack dump: [ 734.123703] task:rcu_sched state:R running task stack: 0 pid: 12 ppid: 2 flags:0x00004008 [ 734.134160] Call Trace: [ 734.137159] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 [ 734.143028] ? _raw_spin_unlock_irqrestore+0x11/0x20 [ 734.148546] ? force_qs_rnp+0x178/0x190 [ 734.152933] ? kfree_rcu_shrink_count+0x50/0x50 [ 734.158015] ? rcu_gp_kthread+0x59c/0xa70 [ 734.162572] ? rcu_cpu_kthread+0x190/0x190 [ 734.167218] ? kthread+0x11b/0x140 [ 734.171166] ? __kthread_bind_mask+0x60/0x60 [ 734.175986] ? ret_from_fork+0x22/0x30 [ 734.180294] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 734.186775] rcu: 1-...0: (839 GPs behind) idle=c2a/1/0x4000000000000000 softirq=681/681 fqs=1838 [ 734.196314] rcu: 2-...0: (144 GPs behind) idle=812/1/0x4000000000000000 softirq=905/905 fqs=1838 [ 734.205847] rcu: 3-...0: (144 GPs behind) idle=eaa/1/0x4000000000000000 softirq=719/719 fqs=1838 [ 734.215361] rcu: 4-...0: (1 GPs behind) idle=c2e/1/0x4000000000000000 softirq=1324/1324 fqs=1838 [ 734.224868] rcu: 5-...0: (144 GPs behind) idle=41a/1/0x4000000000000000 softirq=556/556 fqs=1838 [ 734.234367] rcu: 6-...0: (144 GPs behind) idle=912/1/0x4000000000000000 softirq=777/777 fqs=1838 [ 734.243851] rcu: 7-...0: (144 GPs behind) idle=5e6/1/0x4000000000000000 softirq=2409/2410 fqs=1838 [ 734.253514] (detected by 87, t=74850 jiffies, g=48537, q=3354) [ 734.259951] Sending NMI from CPU 87 to CPUs 1: [ 744.175988] Sending NMI from CPU 87 to CPUs 2: [ 754.092020] Sending NMI from CPU 87 to CPUs 3: [ 764.008042] Sending NMI from CPU 87 to CPUs 4: [ 773.924058] Sending NMI from CPU 87 to CPUs 5: [ 783.840082] Sending NMI from CPU 87 to CPUs 6: [ 793.756086] Sending NMI from CPU 87 to CPUs 7: [ 803.672087] rcu: rcu_sched kthread starved for 34772 jiffies! g48537 f0x2 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=14 [ 803.682940] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 803.692409] rcu: RCU grace-period kthread stack dump: [ 803.697974] task:rcu_sched state:R running task stack: 0 pid: 12 ppid: 2 flags:0x00004008 [ 803.708416] Call Trace: [ 803.711404] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 [ 803.717254] ? _raw_spin_unlock_irqrestore+0x11/0x20 [ 803.722752] ? force_qs_rnp+0x178/0x190 [ 803.727132] ? kfree_rcu_shrink_count+0x50/0x50 [ 803.732208] ? rcu_gp_kthread+0x59c/0xa70 [ 803.736751] ? rcu_cpu_kthread+0x190/0x190 [ 803.741371] ? kthread+0x11b/0x140 [ 803.745299] ? __kthread_bind_mask+0x60/0x60 [ 803.750102] ? ret_from_fork+0x22/0x30 [ 803.754390] hrtimer: interrupt took 221889980360 ns
