Hi Juhamatti,
The second is probably the one that was fixed by commit d3092b2efca1c ("tipc:
fix unsafe rcu locking when accessing publication list") which was delivered to
4.18 around a month ago. Not sure how far back this has been applied, but if
you look at the very latest 4.15 upstream you may be lucky.
Unfortunately I don't recognize the first one, -it seems to be a timer
interrupt deleting a subscription object that is still locked while sending an
event. What I can say, is that the whole topology server was redesigned in
Linux 4.17, among other things because this kind of problems never seemed to
end.
Do you have any possibility to upgrade your kernel to 4.19 and try again? There
has been a few bug fixes added during 4.18/4.19 that you may benefit from.
Otherwise I may have to look closer at this one, and see what can be done.
Regards
///jon
> -----Original Message-----
> From: [email protected] <[email protected]>
> Sent: 31-Oct-18 10:23
> To: [email protected]
> Cc: [email protected]
> Subject: Re: [tipc-discussion] TIPC scalability viewpoints
>
> Hello again,
>
> Here is another issue found in the same setup, looks a bit different.
> Please see below.
>
> BR,
> --
> Juhamatti
>
> [ 6276.191737] Unable to remove publication from failed node [ 6276.191737]
> (type=18888, lower=43818, node=0x100136c, ref=936542876,
> key=936542877) [ 6276.205086] BUG: unable to handle kernel NULL pointer
> dereference at 0000000000000088 [ 6276.213016] IP:
> __call_rcu.constprop.69+0x25/0x210
> [ 6276.217837] PGD 0 P4D 0
> [ 6276.220405] Oops: 0002 [#1] SMP NOPTI [ 6276.224103] Modules linked in:
> tipc ip6_udp_tunnel udp_tunnel xt_CHECKSUM iptable_mangle
> ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4
> nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc
> veth iptable_filter ipmi_ssif
> nls_iso8859_1 intel_rapl pnd2_edac x86_pkg_temp_thermal
> intel_powerclamp coretemp kvm_intel asix kvm usbnet input_leds irqbypass
> joydev intel_cstate mii intel_rapl_perf mac_hid ipmi_si ipmi_devintf shpchp
> ipmi_msghandler sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core
> iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables
> autofs4
> btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy
> async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath
> linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul [ 6276.295019]
> ghash_clmulni_intel pcbc ast ttm aesni_intel drm_kms_helper aes_x86_64
> syscopyarea crypto_simd glue_helper sysfillrect sysimgblt cryptd ixgbe
> fb_sys_fops igb i40e drm ahci dca i2c_algo_bit mdio libahci ptp i2c_ismt
> pps_core [ 6276.316114] CPU: 15 PID: 100 Comm: ksoftirqd/15 Not tainted
> 4.15.0-38-generic #41-Ubuntu [ 6276.324259] Hardware name: Supermicro
> Super Server/A2SDV-16C-TLN5F, BIOS 1.0b 12/12/2017 [ 6276.332413] RIP:
> 0010:__call_rcu.constprop.69+0x25/0x210
> [ 6276.337773] RSP: 0018:ffffb215c34cbbd0 EFLAGS: 00010246 [ 6276.343033]
> RAX: 0000000000000076 RBX: ffff9a5046cdd110 RCX: 0000000000000001 [
> 6276.350213] RDX: ffffffffbe4a9200 RSI: 0000000000000080 RDI:
> 0000000000000000 [ 6276.357402] RBP: ffffb215c34cbc08 R08:
> 0000000000000000 R09: 0000000000001550 [ 6276.364595] R10:
> 00000000000012df R11: 00000000ffffffff R12: ffff9a4fe3bbfc80 [ 6276.371778]
> R13: ffff9a5221a54740 R14: ffffffffbe4a9200 R15: 0000000000000080 [
> 6276.378963] FS: 0000000000000000(0000) GS:ffff9a561f5c0000(0000)
> knlGS:0000000000000000
> [ 6276.387096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> 6276.392878] CR2: 0000000000000088 CR3: 00000006f4818000 CR4:
> 00000000003406e0 [ 6276.400057] Call Trace:
> [ 6276.402544] kfree_call_rcu+0x1a/0x20 [ 6276.406256]
> tipc_publ_notify+0xae/0x190 [tipc] [ 6276.410840]
> tipc_node_write_unlock+0xf3/0x100 [tipc] [ 6276.415979]
> tipc_node_link_down+0x99/0x140 [tipc] [ 6276.420858]
> tipc_node_timeout+0x126/0x130 [tipc] [ 6276.425644] ?
> tipc_node_calculate_timer.isra.13+0x60/0x60 [tipc] [ 6276.431795]
> call_timer_fn+0x30/0x130 [ 6276.435487] run_timer_softirq+0x3fb/0x450 [
> 6276.439614] ? __switch_to_asm+0x34/0x70 [ 6276.443573] ?
> __switch_to+0xad/0x500 [ 6276.447262] ? __switch_to_asm+0x40/0x70 [
> 6276.451219] ? __switch_to_asm+0x34/0x70 [ 6276.455178] ?
> __switch_to_asm+0x40/0x70 [ 6276.459141] __do_softirq+0xe4/0x2bb [
> 6276.462750] run_ksoftirqd+0x22/0x60 [ 6276.466364]
> smpboot_thread_fn+0xfc/0x170 [ 6276.470397] kthread+0x121/0x140 [
> 6276.473645] ? sort_range+0x30/0x30 [ 6276.477153] ?
> kthread_create_worker_on_cpu+0x70/0x70
> [ 6276.482237] ret_from_fork+0x35/0x40
> [ 6276.485835] Code: b7 f8 ff 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41
> 57 41 56 41 55 41 54 49 89 ff 53 49 89 d6 48 83 ec 10 83 e7 07 0f 85 dd 00 00
> 00
> <49> 89 77 08 49 c7 07 00 00 00 00 9c 58 0f 1f 44 00 00 49
> 89 c5
> [ 6276.504818] RIP: __call_rcu.constprop.69+0x25/0x210 RSP:
> ffffb215c34cbbd0 [ 6276.511642] CR2: 0000000000000088 [ 6276.515000] ---[
> end trace f418a9afa145bb5d ]--- [ 6276.528130] Kernel panic - not syncing:
> Fatal exception in interrupt [ 6276.534610] Kernel Offset: 0x3c000000 from
> 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> [ 6276.553092] ---[ end Kernel panic - not syncing: Fatal exception in
> interrupt
> [ 6276.560285] WARNING: CPU: 15 PID: 100 at
> /build/linux-CuEUJv/linux-4.15.0/kernel/sched/core.c:1192
> set_task_cpu+0x158/0x160
> [ 6276.571459] Modules linked in: tipc ip6_udp_tunnel udp_tunnel
> xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> nf_conntrack xt_tcpudp bridge stp llc veth iptable_filter ipmi_ssif
> nls_iso8859_1 intel_rapl pnd2_edac x86_pkg_temp_thermal
> intel_powerclamp coretemp kvm_intel asix kvm usbnet input_leds irqbypass
> joydev intel_cstate mii intel_rapl_perf mac_hid ipmi_si ipmi_devintf shpchp
> ipmi_msghandler sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core
> iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables
> autofs4
> btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy
> async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath
> linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul [ 6276.642267]
> ghash_clmulni_intel pcbc ast ttm aesni_intel drm_kms_helper aes_x86_64
> syscopyarea crypto_simd glue_helper sysfillrect sysimgblt cryptd ixgbe
> fb_sys_fops igb i40e drm ahci dca i2c_algo_bit mdio libahci ptp i2c_ismt
> pps_core
> [ 6276.663306] CPU: 15 PID: 100 Comm: ksoftirqd/15 Tainted: G D
> 4.15.0-38-generic #41-Ubuntu
> [ 6276.672734] Hardware name: Supermicro Super Server/A2SDV-16C-TLN5F,
> BIOS 1.0b 12/12/2017 [ 6276.683335] RIP: 0010:set_task_cpu+0x158/0x160 [
> 6276.690288] RSP: 0018:ffff9a561f5c3dc8 EFLAGS: 00010006 [ 6276.698025]
> RAX: 0000000000000200 RBX: ffff9a50a7565900 RCX: 0000000000000000 [
> 6276.707673] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> ffff9a50a7565900 [ 6276.717305] RBP: ffff9a561f5c3de8 R08:
> 0000000000000000 R09: 000000000000ffff [ 6276.726926] R10: ffff9a561f5c3c90
> R11: 0000000000000000 R12: ffff9a50a756642c [ 6276.736527] R13:
> 0000000000000000 R14: 0000000000000000 R15: 0000000000022880 [
> 6276.746122] FS: 0000000000000000(0000) GS:ffff9a561f5c0000(0000)
> knlGS:0000000000000000
> [ 6276.756700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> 6276.764891] CR2: 0000000000000088 CR3: 00000006f4818000 CR4:
> 00000000003406e0 [ 6276.774418] Call Trace:
> [ 6276.779188] <IRQ>
> [ 6276.783446] try_to_wake_up+0x159/0x480 [ 6276.789458]
> default_wake_function+0x12/0x20 [ 6276.795834]
> autoremove_wake_function+0x12/0x40
> [ 6276.802444] __wake_up_common+0x73/0x130 [ 6276.808426]
> __wake_up_common_lock+0x80/0xc0 [ 6276.814730]
> __wake_up+0x13/0x20 [ 6276.819987]
> wake_up_klogd_work_func+0x40/0x60 [ 6276.826444]
> irq_work_run_list+0x52/0x80 [ 6276.832376] irq_work_run+0x2c/0x40 [
> 6276.837876] flush_smp_call_function_queue+0x66/0xf0
> [ 6276.844845] generic_smp_call_function_single_interrupt+0x13/0x30
> [ 6276.852952] smp_call_function_interrupt+0x3e/0xd0
> [ 6276.859743] call_function_interrupt+0x84/0x90 [ 6276.866171] </IRQ> [
> 6276.870234] RIP: 0010:panic+0x1fe/0x244 [ 6276.876021] RSP:
> 0018:ffffb215c34cb918 EFLAGS: 00000286 ORIG_RAX:
> ffffffffffffff03
> [ 6276.885572] RAX: 0000000000000041 RBX: 0000000000000000 RCX:
> 0000000000000006 [ 6276.894699] RDX: 0000000000000000 RSI:
> 0000000000000092 RDI: ffff9a561f5d6490 [ 6276.903823] RBP:
> ffffb215c34cb990 R08: 0000000000000000 R09: 000000000000157e [
> 6276.912957] R10: 0000000000000008 R11: 00000000ffffffff R12:
> 0000000000000000 [ 6276.922071] R13: 0000000000000000 R14:
> 0000000000000009 R15: 0000000000000002 [ 6276.931172] ?
> panic+0x1f7/0x244 [ 6276.936358] oops_end+0xb6/0xd0 [ 6276.941463]
> no_context+0x17c/0x440 [ 6276.946934] ? __const_udelay+0x41/0x50 [
> 6276.952753] __bad_area_nosemaphore+0xf3/0x1b0 [ 6276.959186]
> bad_area_nosemaphore+0x14/0x20 [ 6276.965378]
> __do_page_fault+0xd4/0x4d0 [ 6276.971226] ? up+0x32/0x50 [ 6276.976035]
> do_page_fault+0x2e/0xe0 [ 6276.981632] page_fault+0x25/0x50 [
> 6276.986954] RIP: 0010:__call_rcu.constprop.69+0x25/0x210
> [ 6276.994290] RSP: 0018:ffffb215c34cbbd0 EFLAGS: 00010246 [ 6277.001543]
> RAX: 0000000000000076 RBX: ffff9a5046cdd110 RCX: 0000000000000001 [
> 6277.010723] RDX: ffffffffbe4a9200 RSI: 0000000000000080 RDI:
> 0000000000000000 [ 6277.019901] RBP: ffffb215c34cbc08 R08:
> 0000000000000000 R09: 0000000000001550 [ 6277.029015] R10:
> 00000000000012df R11: 00000000ffffffff R12: ffff9a4fe3bbfc80 [ 6277.038070]
> R13: ffff9a5221a54740 R14: ffffffffbe4a9200 R15: 0000000000000080 [
> 6277.047129] kfree_call_rcu+0x1a/0x20 [ 6277.052708]
> tipc_publ_notify+0xae/0x190 [tipc] [ 6277.059161]
> tipc_node_write_unlock+0xf3/0x100 [tipc] [ 6277.066123]
> tipc_node_link_down+0x99/0x140 [tipc] [ 6277.072792]
> tipc_node_timeout+0x126/0x130 [tipc] [ 6277.079348] ?
> tipc_node_calculate_timer.isra.13+0x60/0x60 [tipc] [ 6277.087259]
> call_timer_fn+0x30/0x130 [ 6277.092670] run_timer_softirq+0x3fb/0x450 [
> 6277.098438] ? __switch_to_asm+0x34/0x70 [ 6277.103961] ?
> __switch_to+0xad/0x500 [ 6277.109137] ? __switch_to_asm+0x40/0x70 [
> 6277.114493] ? __switch_to_asm+0x34/0x70 [ 6277.119765] ?
> __switch_to_asm+0x40/0x70 [ 6277.125006] __do_softirq+0xe4/0x2bb [
> 6277.129861] run_ksoftirqd+0x22/0x60 [ 6277.134699]
> smpboot_thread_fn+0xfc/0x170 [ 6277.139965] kthread+0x121/0x140 [
> 6277.144439] ? sort_range+0x30/0x30 [ 6277.149163] ?
> kthread_create_worker_on_cpu+0x70/0x70
> [ 6277.155452] ret_from_fork+0x35/0x40
> [ 6277.160258] Code: 32 ff ff ff 80 8b 64 08 00 00 04 e9 4e ff ff ff 0f 0b e9
> f2 fe
> ff ff f7 43 5c fd ff ff ff 0f 84 fc fe ff ff 0f 0b e9
> f5 fe ff ff <0f> 0b e9 ff fe ff ff 90 0f 1f 44 00 00 55 48 89 e5 41 56
> 41 55
> [ 6277.181837] ---[ end trace f418a9afa145bb5e ]--- [ 6277.187806]
> ------------[
> cut here ]------------ [ 6277.193765] sched: Unexpected reschedule of offline
> CPU#0!
> [ 6277.200584] WARNING: CPU: 15 PID: 100 at
> /build/linux-CuEUJv/linux-4.15.0/arch/x86/kernel/smp.c:128
> native_smp_send_reschedule+0x3a/0x40
> [ 6277.214247] Modules linked in: tipc ip6_udp_tunnel udp_tunnel
> xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4
> iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> nf_conntrack xt_tcpudp bridge stp llc veth iptable_filter ipmi_ssif
> nls_iso8859_1 intel_rapl pnd2_edac x86_pkg_temp_thermal
> intel_powerclamp coretemp kvm_intel asix kvm usbnet input_leds irqbypass
> joydev intel_cstate mii intel_rapl_perf mac_hid ipmi_si ipmi_devintf shpchp
> ipmi_msghandler sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core
> iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables
> autofs4
> btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy
> async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath
> linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul [ 6277.291150]
> ghash_clmulni_intel pcbc ast ttm aesni_intel drm_kms_helper aes_x86_64
> syscopyarea crypto_simd glue_helper sysfillrect sysimgblt cryptd ixgbe
> fb_sys_fops igb i40e drm ahci dca i2c_algo_bit mdio libahci ptp i2c_ismt
> pps_core
> [ 6277.315672] CPU: 15 PID: 100 Comm: ksoftirqd/15 Tainted: G D W
> 4.15.0-38-generic #41-Ubuntu
> [ 6277.326902] Hardware name: Supermicro Super Server/A2SDV-16C-TLN5F,
> BIOS 1.0b 12/12/2017 [ 6277.336859] RIP:
> 0010:native_smp_send_reschedule+0x3a/0x40
> [ 6277.344224] RSP: 0018:ffff9a561f5c3de8 EFLAGS: 00010086 [ 6277.351330]
> RAX: 0000000000000000 RBX: ffff9a50a7565900 RCX: 0000000000000006 [
> 6277.360358] RDX: 0000000000000007 RSI: 0000000000000092 RDI:
> ffff9a561f5d6490 [ 6277.369378] RBP: ffff9a561f5c3de8 R08:
> 0000000000000000 R09: 00000000000015cd [ 6277.378405] R10:
> ffff9a561f5c3dc8 R11: 00000000ffffffff R12: ffff9a50a756642c [ 6277.387428]
> R13: 0000000000000004 R14: 0000000000000046 R15: 0000000000022880 [
> 6277.396489] FS: 0000000000000000(0000) GS:ffff9a561f5c0000(0000)
> knlGS:0000000000000000
> [ 6277.406505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> 6277.414171] CR2: 0000000000000088 CR3: 00000006f4818000 CR4:
> 00000000003406e0 [ 6277.423241] Call Trace:
> [ 6277.427619] <IRQ>
> [ 6277.431550] try_to_wake_up+0x3c6/0x480 [ 6277.437319]
> default_wake_function+0x12/0x20 [ 6277.443503]
> autoremove_wake_function+0x12/0x40
> [ 6277.449960] __wake_up_common+0x73/0x130 [ 6277.455811]
> __wake_up_common_lock+0x80/0xc0 [ 6277.462004]
> __wake_up+0x13/0x20 [ 6277.467146]
> wake_up_klogd_work_func+0x40/0x60 [ 6277.473510]
> irq_work_run_list+0x52/0x80 [ 6277.479353] irq_work_run+0x2c/0x40 [
> 6277.484753] flush_smp_call_function_queue+0x66/0xf0
> [ 6277.491642] generic_smp_call_function_single_interrupt+0x13/0x30
> [ 6277.499674] smp_call_function_interrupt+0x3e/0xd0
> [ 6277.506405] call_function_interrupt+0x84/0x90 [ 6277.512787] </IRQ> [
> 6277.516805] RIP: 0010:panic+0x1fe/0x244 [ 6277.522558] RSP:
> 0018:ffffb215c34cb918 EFLAGS: 00000286 ORIG_RAX:
> ffffffffffffff03
> [ 6277.532076] RAX: 0000000000000041 RBX: 0000000000000000 RCX:
> 0000000000000006 [ 6277.541160] RDX: 0000000000000000 RSI:
> 0000000000000092 RDI: ffff9a561f5d6490 [ 6277.550232] RBP:
> ffffb215c34cb990 R08: 0000000000000000 R09: 000000000000157e [
> 6277.559296] R10: 0000000000000008 R11: 00000000ffffffff R12:
> 0000000000000000 [ 6277.568356] R13: 0000000000000000 R14:
> 0000000000000009 R15: 0000000000000002 [ 6277.577422] ?
> panic+0x1f7/0x244 [ 6277.582578] oops_end+0xb6/0xd0 [ 6277.587644]
> no_context+0x17c/0x440 [ 6277.593041] ? __const_udelay+0x41/0x50 [
> 6277.598779] __bad_area_nosemaphore+0xf3/0x1b0 [ 6277.605124]
> bad_area_nosemaphore+0x14/0x20 [ 6277.611212]
> __do_page_fault+0xd4/0x4d0 [ 6277.616958] ? up+0x32/0x50 [ 6277.621667]
> do_page_fault+0x2e/0xe0 [ 6277.627168] page_fault+0x25/0x50 [
> 6277.632417] RIP: 0010:__call_rcu.constprop.69+0x25/0x210
> [ 6277.639666] RSP: 0018:ffffb215c34cbbd0 EFLAGS: 00010246 [ 6277.646845]
> RAX: 0000000000000076 RBX: ffff9a5046cdd110 RCX: 0000000000000001 [
> 6277.655951] RDX: ffffffffbe4a9200 RSI: 0000000000000080 RDI:
> 0000000000000000 [ 6277.665065] RBP: ffffb215c34cbc08 R08:
> 0000000000000000 R09: 0000000000001550 [ 6277.674176] R10:
> 00000000000012df R11: 00000000ffffffff R12: ffff9a4fe3bbfc80 [ 6277.683298]
> R13: ffff9a5221a54740 R14: ffffffffbe4a9200 R15: 0000000000000080 [
> 6277.692426] kfree_call_rcu+0x1a/0x20 [ 6277.698025]
> tipc_publ_notify+0xae/0x190 [tipc] [ 6277.704432]
> tipc_node_write_unlock+0xf3/0x100 [tipc] [ 6277.711361]
> tipc_node_link_down+0x99/0x140 [tipc] [ 6277.718027]
> tipc_node_timeout+0x126/0x130 [tipc] [ 6277.724591] ?
> tipc_node_calculate_timer.isra.13+0x60/0x60 [tipc] [ 6277.732516]
> call_timer_fn+0x30/0x130 [ 6277.737940] run_timer_softirq+0x3fb/0x450 [
> 6277.743701] ? __switch_to_asm+0x34/0x70 [ 6277.749216] ?
> __switch_to+0xad/0x500 [ 6277.754386] ? __switch_to_asm+0x40/0x70 [
> 6277.759732] ? __switch_to_asm+0x34/0x70 [ 6277.764993] ?
> __switch_to_asm+0x40/0x70 [ 6277.770220] __do_softirq+0xe4/0x2bb [
> 6277.775071] run_ksoftirqd+0x22/0x60 [ 6277.779899]
> smpboot_thread_fn+0xfc/0x170 [ 6277.785156] kthread+0x121/0x140 [
> 6277.789619] ? sort_range+0x30/0x30 [ 6277.794333] ?
> kthread_create_worker_on_cpu+0x70/0x70
> [ 6277.800617] ret_from_fork+0x35/0x40
> [ 6277.805419] Code: 2d 61 60 01 73 17 48 8b 05 84 21 16 01 be fd 00
> 00 00 48 8b 40 30 e8 36 b6 ba 00 5d c3 89 fe 48 c7 c7 60 fe 0b be e8
> 06 48 03 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec
> 20 65
> [ 6277.826959] ---[ end trace f418a9afa145bb5f ]--- ke 31. lokak. 2018 klo
> 14.39
> [email protected] kirjoitti:
> >
> > Hello,
> >
> > Thanks Ying! While running a server with 250 LXCs and 2500 TIPC
> > sockets running with modified connection demo, I am seeing the problem
> > described below. The kernel is stock Ubuntu1804 4.15.0-38-generic with
> > 16 cores. It is fairly easy to reproduce, after starting up containers
> > it will happen within an hour or so.
> >
> > Please let me know if there is already a patch for it, I am happy to
> > try. I can also try fix candidates quite easily.
> >
> > BR,
> > --
> > Juhamatti
> >
> > [ 3195.231191] BUG: unable to handle kernel paging request at
> > 0000000200001dc7 [ 3195.238220] IP: _raw_spin_lock_bh+0x1b/0x30 [
> > 3195.242432] PGD 0 P4D 0 [ 3195.244988] Oops: 0002 [#1] SMP NOPTI [
> > 3195.248668] Modules linked in: binfmt_misc tipc ip6_udp_tunnel
> > udp_tunnel xt_CHECKSUM iptable_mangle ipt_MASQUERADE
> > nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
> > nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc veth
> > iptable_filter ipmi_ssif nls_iso8859_1 intel_rapl pnd2_edac
> > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
> irqbypass
> > asix intel_cstate usbnet intel_rapl_perf joydev input_leds mii ipmi_si
> > ipmi_devintf ipmi_msghandler mac_hid shpchp sch_fq_codel ib_iser
> > rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi
> > scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress
> > raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
> > async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
> > hid_generic usbhid hid crct10dif_pclmul [ 3195.319457] crc32_pclmul
> > ghash_clmulni_intel pcbc ast aesni_intel ttm aes_x86_64 drm_kms_helper
> > crypto_simd glue_helper cryptd syscopyarea sysfillrect sysimgblt ixgbe
> > igb fb_sys_fops i40e drm ahci dca i2c_algo_bit libahci mdio i2c_ismt
> > ptp pps_core [ 3195.341632] CPU: 1 PID: 348 Comm: kworker/u32:7 Not
> > tainted 4.15.0-38-generic #41-Ubuntu [ 3195.349754] Hardware name:
> > Supermicro Super Server/A2SDV-16C-TLN5F, BIOS 1.0b 12/12/2017 [
> > 3195.357891] Workqueue: tipc_send tipc_send_work [tipc] [ 3195.363061]
> > RIP: 0010:_raw_spin_lock_bh+0x1b/0x30 [ 3195.367788] RSP:
> > 0018:ffff90d3df243e90 EFLAGS: 00010246 [ 3195.373032] RAX:
> > 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08 [
> > 3195.380203] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI:
> > 0000000200001dc7 [ 3195.387393] RBP: ffff90d3df243e90 R08:
> > ffff90d3df243f10 R09: ffff90d3df25a780 [ 3195.394561] R10:
> > ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7 [
> > 3195.401729] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15:
> > ffff90d1317a6c18 [ 3195.408897] FS: 0000000000000000(0000)
> > GS:ffff90d3df240000(0000)
> > knlGS:0000000000000000
> > [ 3195.417025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> > 3195.422798] CR2: 0000000200001dc7 CR3: 00000007711f8000 CR4:
> > 00000000003406e0 [ 3195.429965] Call Trace:
> > [ 3195.432429] <IRQ>
> > [ 3195.434467] tipc_subscrp_timeout+0x25/0x90 [tipc] [ 3195.439311]
> > call_timer_fn+0x30/0x130 [ 3195.442994] run_timer_softirq+0x3fb/0x450
> > [ 3195.447113] ? lapic_next_deadline+0x26/0x30 [ 3195.451404] ?
> > clockevents_program_event+0xd8/0x110
> > [ 3195.456304] __do_softirq+0xe4/0x2bb [ 3195.459903]
> > do_softirq_own_stack+0x2a/0x40 [ 3195.464107] </IRQ> [ 3195.466219]
> > do_softirq.part.17+0x50/0x60 [ 3195.470250]
> > __local_bh_enable_ip+0x50/0x60 [ 3195.474456]
> > _raw_spin_unlock_bh+0x1e/0x20 [ 3195.478576] tipc_sk_rcv+0x15a/0x440
> > [tipc] [ 3195.482782] tipc_node_xmit+0x147/0x190 [tipc] [
> > 3195.487254] __tipc_sendstream+0x1a9/0x2d0 [tipc] [ 3195.491987] ?
> > aa_sk_perm+0x48/0x1a0 [ 3195.495587] tipc_sendstream+0x35/0x50 [tipc]
> > [ 3195.499976] tipc_send_packet+0x17/0x20 [tipc] [ 3195.504454]
> > sock_sendmsg+0x3e/0x50 [ 3195.507967] kernel_sendmsg+0x2b/0x30 [
> > 3195.511663] tipc_send_to_sock+0x12f/0x290 [tipc] [ 3195.516395] ?
> > __switch_to_asm+0x40/0x70 [ 3195.520338] ?
> __switch_to_asm+0x34/0x70
> > [ 3195.524280] ? __switch_to_asm+0x34/0x70 [ 3195.528222] ?
> > __switch_to_asm+0x40/0x70 [ 3195.532174] ?
> __switch_to_asm+0x34/0x70
> > [ 3195.536133] tipc_send_work+0x23/0x40 [tipc] [ 3195.540433]
> > process_one_work+0x1de/0x410 [ 3195.544468]
> worker_thread+0x32/0x410
> > [ 3195.548147] kthread+0x121/0x140 [ 3195.551403] ?
> > process_one_work+0x410/0x410 [ 3195.555611] ?
> > kthread_create_worker_on_cpu+0x70/0x70
> > [ 3195.560686] ? do_syscall_64+0x73/0x130 [ 3195.564545] ?
> > SyS_exit_group+0x14/0x20 [ 3195.568404] ret_from_fork+0x35/0x40 [
> > 3195.571998] Code: 00 5d c3 31 c0 5d c3 66 0f 1f 84 00 00 00 00 00 0f
> > 1f 44 00 00 55 65 81 05 8f 8a c7 73 00 02 00 00 48 89 e5 31 c0 ba
> > 01 00 00 00 <f0> 0f b1 17 85 c0 75 02 5d c3 89 c6 e8 24 12 74 ff 66 90
> > 5d c3 [ 3195.590981] RIP: _raw_spin_lock_bh+0x1b/0x30 RSP:
> > ffff90d3df243e90 [ 3195.599011] CR2: 0000000200001dc7 [ 3195.604403]
> > ---[ end trace 4ce4054b7f724f81 ]--- [ 3195.619914] Kernel panic - not
> > syncing: Fatal exception in interrupt [ 3195.628264] Kernel Offset:
> > 0xaa00000 from 0xffffffff81000000 (relocation range:
> > 0xffffffff80000000-0xffffffffbfffffff)
> > [ 3195.649393] ---[ end Kernel panic - not syncing: Fatal exception in
> > interrupt [ 3195.658447] WARNING: CPU: 1 PID: 348 at
> > /build/linux-CuEUJv/linux-4.15.0/kernel/sched/core.c:1192
> > set_task_cpu+0x158/0x160
> > [ 3195.671428] Modules linked in: binfmt_misc tipc ip6_udp_tunnel
> > udp_tunnel xt_CHECKSUM iptable_mangle ipt_MASQUERADE
> > nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
> > nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc veth
> > iptable_filter ipmi_ssif nls_iso8859_1 intel_rapl pnd2_edac
> > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
> irqbypass
> > asix intel_cstate usbnet intel_rapl_perf joydev input_leds mii ipmi_si
> > ipmi_devintf ipmi_msghandler mac_hid shpchp sch_fq_codel ib_iser
> > rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi
> > scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress
> > raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
> > async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
> > hid_generic usbhid hid crct10dif_pclmul [ 3195.749731] crc32_pclmul
> > ghash_clmulni_intel pcbc ast aesni_intel ttm aes_x86_64 drm_kms_helper
> > crypto_simd glue_helper cryptd syscopyarea sysfillrect sysimgblt ixgbe
> > igb fb_sys_fops i40e drm ahci dca i2c_algo_bit libahci mdio i2c_ismt
> > ptp pps_core
> > [ 3195.775727] CPU: 1 PID: 348 Comm: kworker/u32:7 Tainted: G D
> > 4.15.0-38-generic #41-Ubuntu
> > [ 3195.787072] Hardware name: Supermicro Super Server/A2SDV-16C-
> TLN5F,
> > BIOS 1.0b 12/12/2017 [ 3195.797128] Workqueue: tipc_send
> > tipc_send_work [tipc] [ 3195.804212] RIP:
> > 0010:set_task_cpu+0x158/0x160 [ 3195.810603] RSP:
> > 0018:ffff90d3df243808 EFLAGS: 00010006 [ 3195.817773] RAX:
> > 0000000000000200 RBX: ffff90d2c5b11640 RCX: 0000000000000000 [
> > 3195.826856] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> > ffff90d2c5b11640 [ 3195.835936] RBP: ffff90d3df243828 R08:
> > 0000000000000000 R09: 000000000000ffff [ 3195.845016] R10:
> > 0000000000000010 R11: 0000000000000000 R12: ffff90d2c5b1216c [
> > 3195.854101] R13: 0000000000000000 R14: 0000000000000000 R15:
> > 0000000000022880 [ 3195.863174] FS: 0000000000000000(0000)
> > GS:ffff90d3df240000(0000)
> > knlGS:0000000000000000
> > [ 3195.873202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> > 3195.880878] CR2: 0000000200001dc7 CR3: 00000007711f8000 CR4:
> > 00000000003406e0 [ 3195.889943] Call Trace:
> > [ 3195.894313] <IRQ>
> > [ 3195.898250] try_to_wake_up+0x159/0x480 [ 3195.904009]
> > default_wake_function+0x12/0x20 [ 3195.910199]
> > autoremove_wake_function+0x12/0x40
> > [ 3195.916660] __wake_up_common+0x73/0x130 [ 3195.922508]
> > __wake_up_common_lock+0x80/0xc0 [ 3195.928695]
> __wake_up+0x13/0x20 [
> > 3195.933831] wake_up_klogd_work_func+0x40/0x60 [ 3195.940185]
> > irq_work_run_list+0x52/0x80 [ 3195.946026] ?
> > tick_sched_do_timer+0x50/0x50 [ 3195.952206] irq_work_tick+0x3b/0x50
> > [ 3195.957696] update_process_times+0x42/0x60 [ 3195.963797]
> > tick_sched_handle+0x26/0x70 [ 3195.969635] ?
> > tick_sched_do_timer+0x50/0x50 [ 3195.975827]
> > tick_sched_timer+0x39/0x80 [ 3195.981576]
> > __hrtimer_run_queues+0xdf/0x230 [ 3195.987756]
> > hrtimer_interrupt+0xa0/0x1d0 [ 3195.993676]
> > smp_apic_timer_interrupt+0x6f/0x130
> > [ 3196.000208] apic_timer_interrupt+0x84/0x90 [ 3196.006312] RIP:
> > 0010:panic+0x1fe/0x244 [ 3196.012062] RSP: 0018:ffff90d3df243bd8
> > EFLAGS: 00000286 ORIG_RAX:
> > ffffffffffffff11
> > [ 3196.021578] RAX: 0000000000000041 RBX: 0000000000000000 RCX:
> > 0000000000000006 [ 3196.030668] RDX: 0000000000000000 RSI:
> > 0000000000000092 RDI: ffff90d3df256490 [ 3196.039773] RBP:
> > ffff90d3df243c50 R08: 0000000000000000 R09: 000000000000113b [
> > 3196.048892] R10: 0000000000000008 R11: 00000000ffffffff R12:
> > 0000000000000000 [ 3196.058026] R13: 0000000000000000 R14:
> > 0000000000000009 R15: 0000000000000002 [ 3196.067174] ?
> > panic+0x1f7/0x244 [ 3196.072419] oops_end+0xb6/0xd0 [ 3196.077569]
> > no_context+0x17c/0x440 [ 3196.083061]
> > __bad_area_nosemaphore+0xf3/0x1b0 [ 3196.089520]
> > bad_area_nosemaphore+0x14/0x20 [ 3196.095714]
> > __do_page_fault+0xd4/0x4d0 [ 3196.101560] ?
> > update_load_avg+0x57f/0x6e0 [ 3196.107657] do_page_fault+0x2e/0xe0 [
> > 3196.113242] page_fault+0x25/0x50 [ 3196.118560] RIP:
> > 0010:_raw_spin_lock_bh+0x1b/0x30 [ 3196.125207] RSP:
> > 0018:ffff90d3df243e90 EFLAGS: 00010246 [ 3196.132299] RAX:
> > 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08 [
> > 3196.141292] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI:
> > 0000000200001dc7 [ 3196.150282] RBP: ffff90d3df243e90 R08:
> > ffff90d3df243f10 R09: ffff90d3df25a780 [ 3196.159277] R10:
> > ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7 [
> > 3196.168262] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15:
> > ffff90d1317a6c18 [ 3196.177238] ? tipc_subscrb_release_cb+0x30/0x30
> > [tipc] [ 3196.184271] ? tipc_subscrb_release_cb+0x30/0x30 [tipc] [
> > 3196.191262] tipc_subscrp_timeout+0x25/0x90 [tipc] [ 3196.197751]
> > call_timer_fn+0x30/0x130 [ 3196.203029] run_timer_softirq+0x3fb/0x450
> > [ 3196.208673] ? lapic_next_deadline+0x26/0x30 [ 3196.214423] ?
> > clockevents_program_event+0xd8/0x110
> > [ 3196.220707] __do_softirq+0xe4/0x2bb [ 3196.225616]
> > do_softirq_own_stack+0x2a/0x40 [ 3196.231114] </IRQ> [ 3196.234491]
> > do_softirq.part.17+0x50/0x60 [ 3196.239759]
> > __local_bh_enable_ip+0x50/0x60 [ 3196.245196]
> > _raw_spin_unlock_bh+0x1e/0x20 [ 3196.250555] tipc_sk_rcv+0x15a/0x440
> > [tipc] [ 3196.255986] tipc_node_xmit+0x147/0x190 [tipc] [
> > 3196.261677] __tipc_sendstream+0x1a9/0x2d0 [tipc] [ 3196.267615] ?
> > aa_sk_perm+0x48/0x1a0 [ 3196.272420] tipc_sendstream+0x35/0x50 [tipc]
> > [ 3196.278014] tipc_send_packet+0x17/0x20 [tipc] [ 3196.283673]
> > sock_sendmsg+0x3e/0x50 [ 3196.288359] kernel_sendmsg+0x2b/0x30 [
> > 3196.293212] tipc_send_to_sock+0x12f/0x290 [tipc] [ 3196.299094] ?
> > __switch_to_asm+0x40/0x70 [ 3196.304205] ?
> __switch_to_asm+0x34/0x70
> > [ 3196.309301] ? __switch_to_asm+0x34/0x70 [ 3196.314379] ?
> > __switch_to_asm+0x40/0x70 [ 3196.319446] ?
> __switch_to_asm+0x34/0x70
> > [ 3196.324510] tipc_send_work+0x23/0x40 [tipc] [ 3196.329931]
> > process_one_work+0x1de/0x410 [ 3196.335103]
> worker_thread+0x32/0x410
> > [ 3196.339919] kthread+0x121/0x140 [ 3196.344299] ?
> > process_one_work+0x410/0x410 [ 3196.349640] ?
> > kthread_create_worker_on_cpu+0x70/0x70
> > [ 3196.355853] ? do_syscall_64+0x73/0x130 [ 3196.360851] ?
> > SyS_exit_group+0x14/0x20 [ 3196.365841] ret_from_fork+0x35/0x40 [
> > 3196.370567] Code: 32 ff ff ff 80 8b 64 08 00 00 04 e9 4e ff ff ff 0f
> > 0b e9 f2 fe ff ff f7 43 5c fd ff ff ff 0f 84 fc fe ff ff 0f 0b e9
> > f5 fe ff ff <0f> 0b e9 ff fe ff ff 90 0f 1f 44 00 00 55 48 89 e5 41 56
> > 41 55
> > [ 3196.391999] ---[ end trace 4ce4054b7f724f82 ]--- [ 3196.397922]
> > ------------[ cut here ]------------ [ 3196.403838] sched: Unexpected
> > reschedule of offline CPU#0!
> > [ 3196.410627] WARNING: CPU: 1 PID: 348 at
> > /build/linux-CuEUJv/linux-4.15.0/arch/x86/kernel/smp.c:128
> > native_smp_send_reschedule+0x3a/0x40
> > [ 3196.424161] Modules linked in: binfmt_misc tipc ip6_udp_tunnel
> > udp_tunnel xt_CHECKSUM iptable_mangle ipt_MASQUERADE
> > nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
> > nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc veth
> > iptable_filter ipmi_ssif nls_iso8859_1 intel_rapl pnd2_edac
> > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
> irqbypass
> > asix intel_cstate usbnet intel_rapl_perf joydev input_leds mii ipmi_si
> > ipmi_devintf ipmi_msghandler mac_hid shpchp sch_fq_codel ib_iser
> > rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi
> > scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress
> > raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
> > async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
> > hid_generic usbhid hid crct10dif_pclmul [ 3196.500686] crc32_pclmul
> > ghash_clmulni_intel pcbc ast aesni_intel ttm aes_x86_64 drm_kms_helper
> > crypto_simd glue_helper cryptd syscopyarea sysfillrect sysimgblt ixgbe
> > igb fb_sys_fops i40e drm ahci dca i2c_algo_bit libahci mdio i2c_ismt
> > ptp pps_core
> > [ 3196.526181] CPU: 1 PID: 348 Comm: kworker/u32:7 Tainted: G D W
> > 4.15.0-38-generic #41-Ubuntu
> > [ 3196.537321] Hardware name: Supermicro Super Server/A2SDV-16C-
> TLN5F,
> > BIOS 1.0b 12/12/2017 [ 3196.547206] Workqueue: tipc_send
> > tipc_send_work [tipc] [ 3196.554142] RIP:
> > 0010:native_smp_send_reschedule+0x3a/0x40
> > [ 3196.561423] RSP: 0018:ffff90d3df243790 EFLAGS: 00010082 [
> > 3196.568445] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> > 0000000000000006 [ 3196.577395] RDX: 0000000000000007 RSI:
> > 0000000000000086 RDI: ffff90d3df256490 [ 3196.586344] RBP:
> > ffff90d3df243790 R08: 0000000000000000 R09: 000000000000119e [
> > 3196.595297] R10: 000000000000025f R11: 00000000ffffffff R12:
> > ffff90d3df222880 [ 3196.604262] R13: ffff90d2c5b11640 R14:
> > ffff90d3df243848 R15: ffff90d3df222880 [ 3196.613232] FS:
> > 0000000000000000(0000) GS:ffff90d3df240000(0000)
> > knlGS:0000000000000000
> > [ 3196.623178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [
> > 3196.630768] CR2: 0000000200001dc7 CR3: 00000007711f8000 CR4:
> > 00000000003406e0 [ 3196.639761] Call Trace:
> > [ 3196.644061] <IRQ>
> > [ 3196.647923] resched_curr+0x5d/0xc0 [ 3196.653276]
> > check_preempt_curr+0x7a/0x90 [ 3196.659130]
> ttwu_do_wakeup+0x1e/0x140
> > [ 3196.664745] ttwu_do_activate+0x77/0x80 [ 3196.670433]
> > try_to_wake_up+0x1d6/0x480 [ 3196.676117]
> > default_wake_function+0x12/0x20 [ 3196.682237]
> > autoremove_wake_function+0x12/0x40
> > [ 3196.688617] __wake_up_common+0x73/0x130 [ 3196.694389]
> > __wake_up_common_lock+0x80/0xc0 [ 3196.700494]
> __wake_up+0x13/0x20 [
> > 3196.705549] wake_up_klogd_work_func+0x40/0x60 [ 3196.711831]
> > irq_work_run_list+0x52/0x80 [ 3196.717598] ?
> > tick_sched_do_timer+0x50/0x50 [ 3196.723707] irq_work_tick+0x3b/0x50
> > [ 3196.729118] update_process_times+0x42/0x60 [ 3196.735141]
> > tick_sched_handle+0x26/0x70 [ 3196.740900] ?
> > tick_sched_do_timer+0x50/0x50 [ 3196.747014]
> > tick_sched_timer+0x39/0x80 [ 3196.752684]
> > __hrtimer_run_queues+0xdf/0x230 [ 3196.758789]
> > hrtimer_interrupt+0xa0/0x1d0 [ 3196.764640]
> > smp_apic_timer_interrupt+0x6f/0x130
> > [ 3196.771101] apic_timer_interrupt+0x84/0x90 [ 3196.777130] RIP:
> > 0010:panic+0x1fe/0x244 [ 3196.782802] RSP: 0018:ffff90d3df243bd8
> > EFLAGS: 00000286 ORIG_RAX:
> > ffffffffffffff11
> > [ 3196.792240] RAX: 0000000000000041 RBX: 0000000000000000 RCX:
> > 0000000000000006 [ 3196.801259] RDX: 0000000000000000 RSI:
> > 0000000000000092 RDI: ffff90d3df256490 [ 3196.810293] RBP:
> > ffff90d3df243c50 R08: 0000000000000000 R09: 000000000000113b [
> > 3196.819342] R10: 0000000000000008 R11: 00000000ffffffff R12:
> > 0000000000000000 [ 3196.828408] R13: 0000000000000000 R14:
> > 0000000000000009 R15: 0000000000000002 [ 3196.837484] ?
> > panic+0x1f7/0x244 [ 3196.842654] oops_end+0xb6/0xd0 [ 3196.847725]
> > no_context+0x17c/0x440 [ 3196.853142]
> > __bad_area_nosemaphore+0xf3/0x1b0 [ 3196.859519]
> > bad_area_nosemaphore+0x14/0x20 [ 3196.865635]
> > __do_page_fault+0xd4/0x4d0 [ 3196.871401] ?
> > update_load_avg+0x57f/0x6e0 [ 3196.877434] do_page_fault+0x2e/0xe0 [
> > 3196.882942] page_fault+0x25/0x50 [ 3196.888121] RIP:
> > 0010:_raw_spin_lock_bh+0x1b/0x30 [ 3196.894624] RSP:
> > 0018:ffff90d3df243e90 EFLAGS: 00010246 [ 3196.901641] RAX:
> > 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08 [
> > 3196.910595] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI:
> > 0000000200001dc7 [ 3196.919541] RBP: ffff90d3df243e90 R08:
> > ffff90d3df243f10 R09: ffff90d3df25a780 [ 3196.928459] R10:
> > ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7 [
> > 3196.937309] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15:
> > ffff90d1317a6c18 [ 3196.946097] ? tipc_subscrb_release_cb+0x30/0x30
> > [tipc] [ 3196.952912] ? tipc_subscrb_release_cb+0x30/0x30 [tipc] [
> > 3196.959636] tipc_subscrp_timeout+0x25/0x90 [tipc] [ 3196.965850]
> > call_timer_fn+0x30/0x130 [ 3196.970861] run_timer_softirq+0x3fb/0x450
> > [ 3196.976282] ? lapic_next_deadline+0x26/0x30 [ 3196.981856] ?
> > clockevents_program_event+0xd8/0x110
> > [ 3196.988004] __do_softirq+0xe4/0x2bb [ 3196.992839]
> > do_softirq_own_stack+0x2a/0x40 [ 3196.998277] </IRQ> [ 3197.001619]
> > do_softirq.part.17+0x50/0x60 [ 3197.006875]
> > __local_bh_enable_ip+0x50/0x60 [ 3197.012300]
> > _raw_spin_unlock_bh+0x1e/0x20 [ 3197.017639] tipc_sk_rcv+0x15a/0x440
> > [tipc] [ 3197.023057] tipc_node_xmit+0x147/0x190 [tipc] [
> > 3197.028715] __tipc_sendstream+0x1a9/0x2d0 [tipc] [ 3197.034618] ?
> > aa_sk_perm+0x48/0x1a0 [ 3197.039384] tipc_sendstream+0x35/0x50 [tipc]
> > [ 3197.044919] tipc_send_packet+0x17/0x20 [tipc] [ 3197.050539]
> > sock_sendmsg+0x3e/0x50 [ 3197.055207] kernel_sendmsg+0x2b/0x30 [
> > 3197.060044] tipc_send_to_sock+0x12f/0x290 [tipc] [ 3197.065927] ?
> > __switch_to_asm+0x40/0x70 [ 3197.071048] ?
> __switch_to_asm+0x34/0x70
> > [ 3197.076162] ? __switch_to_asm+0x34/0x70 [ 3197.081257] ?
> > __switch_to_asm+0x40/0x70 [ 3197.086345] ?
> __switch_to_asm+0x34/0x70
> > [ 3197.091425] tipc_send_work+0x23/0x40 [tipc] [ 3197.096854]
> > process_one_work+0x1de/0x410 [ 3197.102030]
> worker_thread+0x32/0x410
> > [ 3197.106854] kthread+0x121/0x140 [ 3197.111239] ?
> > process_one_work+0x410/0x410 [ 3197.116576] ?
> > kthread_create_worker_on_cpu+0x70/0x70
> > [ 3197.122788] ? do_syscall_64+0x73/0x130 [ 3197.127786] ?
> > SyS_exit_group+0x14/0x20 [ 3197.132784] ret_from_fork+0x35/0x40 [
> > 3197.137508] Code: 2d 61 60 01 73 17 48 8b 05 84 21 16 01 be fd 00
> > 00 00 48 8b 40 30 e8 36 b6 ba 00 5d c3 89 fe 48 c7 c7 60 fe ab 8c e8
> > 06 48 03 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec
> > 20 65
> > [ 3197.158946] ---[ end trace 4ce4054b7f724f83 ]--- [ 3197.164963]
> > unchecked MSR access error: WRMSR to 0x83f (tried to write
> > 0x00000000000000f6) at rIP: 0xffffffff8ba6c1a8
> > (native_write_msr+0x8/0x30)
> > [ 3197.179360] Call Trace:
> > [ 3197.183165] <IRQ>
> > [ 3197.186532] native_apic_msr_write+0x2b/0x40 [ 3197.192143]
> > x2apic_send_IPI_self+0x20/0x30 [ 3197.197660]
> > arch_irq_work_raise+0x2a/0x40 [ 3197.203080]
> irq_work_queue+0x8d/0xa0
> > [ 3197.208058] dbs_update_util_handler+0x80/0xa0 [ 3197.213822]
> > update_load_avg+0x5c5/0x6e0 [ 3197.219068]
> enqueue_entity+0x5e/0x6b0
> > [ 3197.224124] enqueue_task_fair+0x6c/0x7f0 [ 3197.229449] ?
> > __update_load_avg_blocked_se.isra.37+0xd1/0x150
> > [ 3197.236608] activate_task+0x54/0xc0 [ 3197.241510] ?
> > sched_clock+0x9/0x10 [ 3197.246338] ttwu_do_activate+0x49/0x80 [
> > 3197.251514] try_to_wake_up+0x1d6/0x480 [ 3197.256684]
> > default_wake_function+0x12/0x20 [ 3197.262288]
> > autoremove_wake_function+0x12/0x40
> > [ 3197.268162] __wake_up_common+0x73/0x130 [ 3197.273428]
> > __wake_up_common_lock+0x80/0xc0 [ 3197.279026]
> __wake_up+0x13/0x20 [
> > 3197.283562] wake_up_klogd_work_func+0x40/0x60 [ 3197.289321]
> > irq_work_run_list+0x52/0x80 [ 3197.294545] ?
> > tick_sched_do_timer+0x50/0x50 [ 3197.300116] irq_work_tick+0x3b/0x50
> > [ 3197.304987] update_process_times+0x42/0x60 [ 3197.310455]
> > tick_sched_handle+0x26/0x70 [ 3197.315672] ?
> > tick_sched_do_timer+0x50/0x50 [ 3197.321233]
> > tick_sched_timer+0x39/0x80 [ 3197.326351]
> > __hrtimer_run_queues+0xdf/0x230 [ 3197.331909]
> > hrtimer_interrupt+0xa0/0x1d0 [ 3197.337205]
> > smp_apic_timer_interrupt+0x6f/0x130
> > [ 3197.343108] apic_timer_interrupt+0x84/0x90 [ 3197.348574] RIP:
> > 0010:panic+0x1fe/0x244 [ 3197.353674] RSP: 0018:ffff90d3df243bd8
> > EFLAGS: 00000286 ORIG_RAX:
> > ffffffffffffff11
> > [ 3197.362537] RAX: 0000000000000041 RBX: 0000000000000000 RCX:
> > 0000000000000006 [ 3197.370972] RDX: 0000000000000000 RSI:
> > 0000000000000092 RDI: ffff90d3df256490 [ 3197.379405] RBP:
> > ffff90d3df243c50 R08: 0000000000000000 R09: 000000000000113b [
> > 3197.387843] R10: 0000000000000008 R11: 00000000ffffffff R12:
> > 0000000000000000 [ 3197.396291] R13: 0000000000000000 R14:
> > 0000000000000009 R15: 0000000000000002 [ 3197.404743] ?
> > panic+0x1f7/0x244 [ 3197.409283] oops_end+0xb6/0xd0 [ 3197.413730]
> > no_context+0x17c/0x440 [ 3197.418521]
> > __bad_area_nosemaphore+0xf3/0x1b0 [ 3197.424273]
> > bad_area_nosemaphore+0x14/0x20 [ 3197.429763]
> > __do_page_fault+0xd4/0x4d0 [ 3197.434912] ?
> > update_load_avg+0x57f/0x6e0 [ 3197.440318] do_page_fault+0x2e/0xe0 [
> > 3197.445203] page_fault+0x25/0x50 [ 3197.449827] RIP:
> > 0010:_raw_spin_lock_bh+0x1b/0x30 [ 3197.455862] RSP:
> > 0018:ffff90d3df243e90 EFLAGS: 00010246 [ 3197.462432] RAX:
> > 0000000000000000 RBX: ffff90d1317a6c18 RCX: ffff90d3df243f08 [
> > 3197.470940] RDX: 0000000000000001 RSI: ffffffffc07d8290 RDI:
> > 0000000200001dc7 [ 3197.479454] RBP: ffff90d3df243e90 R08:
> > ffff90d3df243f10 R09: ffff90d3df25a780 [ 3197.487973] R10:
> > ffff90d3df243f10 R11: ffff90d3df25a770 R12: 0000000200001dc7 [
> > 3197.496495] R13: ffff90d1317a6c00 R14: ffffffffc07d8290 R15:
> > ffff90d1317a6c18 [ 3197.505043] ? tipc_subscrb_release_cb+0x30/0x30
> > [tipc] [ 3197.511689] ? tipc_subscrb_release_cb+0x30/0x30 [tipc] [
> > 3197.518319] tipc_subscrp_timeout+0x25/0x90 [tipc] [ 3197.524511]
> > call_timer_fn+0x30/0x130 [ 3197.529573] run_timer_softirq+0x3fb/0x450
> > [ 3197.535073] ? lapic_next_deadline+0x26/0x30 [ 3197.540684] ?
> > clockevents_program_event+0xd8/0x110
> > [ 3197.546832] __do_softirq+0xe4/0x2bb [ 3197.551680]
> > do_softirq_own_stack+0x2a/0x40 [ 3197.557086] </IRQ> [ 3197.560362]
> > do_softirq.part.17+0x50/0x60 [ 3197.565561]
> > __local_bh_enable_ip+0x50/0x60 [ 3197.570937]
> > _raw_spin_unlock_bh+0x1e/0x20 [ 3197.576228] tipc_sk_rcv+0x15a/0x440
> > [tipc] [ 3197.581602] tipc_node_xmit+0x147/0x190 [tipc] [
> > 3197.587245] __tipc_sendstream+0x1a9/0x2d0 [tipc] [ 3197.593144] ?
> > aa_sk_perm+0x48/0x1a0 [ 3197.597909] tipc_sendstream+0x35/0x50 [tipc]
> > [ 3197.603461] tipc_send_packet+0x17/0x20 [tipc] [ 3197.609101]
> > sock_sendmsg+0x3e/0x50 [ 3197.613767] kernel_sendmsg+0x2b/0x30 [
> > 3197.618609] tipc_send_to_sock+0x12f/0x290 [tipc] [ 3197.624497] ?
> > __switch_to_asm+0x40/0x70 [ 3197.629601] ?
> __switch_to_asm+0x34/0x70
> > [ 3197.634697] ? __switch_to_asm+0x34/0x70 [ 3197.639781] ?
> > __switch_to_asm+0x40/0x70 [ 3197.644846] ?
> __switch_to_asm+0x34/0x70
> > [ 3197.649909] tipc_send_work+0x23/0x40 [tipc] [ 3197.655317]
> > process_one_work+0x1de/0x410 [ 3197.660470]
> worker_thread+0x32/0x410
> > [ 3197.665274] kthread+0x121/0x140 [ 3197.669636] ?
> > process_one_work+0x410/0x410 [ 3197.674955] ?
> > kthread_create_worker_on_cpu+0x70/0x70
> > [ 3197.681146] ? do_syscall_64+0x73/0x130 [ 3197.686118] ?
> > SyS_exit_group+0x14/0x20 [ 3197.691094] ret_from_fork+0x35/0x40 pe
> > 26. lokak. 2018 klo 6.38 Ying Xue ([email protected]) kirjoitti:
> > >
> > > Hi Juhamatti,
> > >
> > > This is a good test plan indeed. If you encounter any problem during
> > > the testing, please share it here.
> > >
> > > From my view, TIPC had fully supported networking nemespace at least
> > > three years ago and its sockets could be fully isolated between
> containers.
> > >
> > > Thanks,
> > > Ying
> > >
> > > On 10/26/2018 11:12 AM, [email protected] wrote:
> > > > Hello,
> > > >
> > > > I'm planning to test TIPC scalability in more detail, especially
> > > > regarding the binding table and topology (service tracking)
> > > > service with large number of nodes (~1k) and sockets (100-1k). Is
> > > > it enough to do it with LXC containers with isolated namespaces to
> > > > get realistic results? My concern here is that if TIPC
> > > > implementation would not provide full isolation to sockets e.g.
> > > > for performance reasons inside the same kernel, then this may not
> > > > provide results matching to real environments. VMs are the other
> > > > choice, but they tend to be heavy. Are the namespaces fully isolated
> inside the kernel?
> > > >
> > > > Thanks for the info,
> > > > --
> > > > Juhamatti
> > > >
> > > >
> > > > _______________________________________________
> > > > tipc-discussion mailing list
> > > > [email protected]
> > > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion
> > > >
>
>
> _______________________________________________
> tipc-discussion mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/tipc-discussion
_______________________________________________
tipc-discussion mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tipc-discussion
