Thanks Ying.

As you suggested, I will revert the "tipc: avoid packets leaking on
socket receive queue" patch. Since the issue is not reproducible on
demand, I may need to wait the issue is seen again or not with the new
driver.

We do experience in traffic throughput due to TIPC connections are
failing (not with this change). If anyone aware of the failures please
let me know.

========
Background ....
System is originally based on Kernel 3.4.2 on Fedora 16 and stable.
Recently, I updated the system with kernel 4.4.0. All stock kernel
drivers are being used and no customization except ported some latest
TIPC patches to fix some TIPC issues.

All 6 routing CPUs went down over the course of the weekend. The noted
output is from one CPU; others are unknown, but assumed to have gone
down with the same cause. Also note that in addition, one of the
routing cards (no output available) went down again on Monday.

The new kernel 4.4.0 is being used in the system Since April 1st, and
seen the issue 3 times so far. All these times, mostly heartbeat type
traffic, not heavy traffic.
=========



On Tue, May 3, 2016 at 6:26 AM, Xue, Ying <[email protected]> wrote:
> I agree with Erik too.
>
>
>
> The oops should be caused by socket was freed early. But
>
>
>
> GUNA, can you reproduce the issue? If so, please try to revert the commit
> f4195d1eac954a67adf112dd53404560cc55b942 (“tipc: avoid packets leaking on
> socket receive queue”), and verify whether the issue occurs or not.
>
>
>
> I suspect the commit bring some unknown side effect, leading to the panic.
>
>
>
> Thanks,
>
> Ying
>
> From: Erik Hugne [mailto:[email protected]]
> Sent: 2016年5月3日 13:09
> To: GUNA
> Cc: [email protected];
> [email protected]; Richard Alpe; Xue, Ying
> Subject: Re: [tipc-discussion] tipc: tipc_recv_stream with kernel panic
>
>
>
> (On mobile)
>
> At first glance, it seems that the socket was freed, but there was a pending
> wakeup signal for it. Which then causes the subsequent spin_lock_bh() to
> deref freed mem.
>
> //E
>
> On May 3, 2016 02:43, "GUNA" <[email protected]> wrote
> [...]
>>> [375832.498126] BUG: unable to handle kernel paging request at
>>> 000001a400015ff4
>> [375832.505300] IP: [<ffffffff810c3566>]
>> queued_spin_lock_slowpath+0xe6/0x160
>> [375832.512394] PGD 0
>> [375832.514657] Oops: 0002 [#1] SMP
>> [375832.518306] Modules linked in: nf_log_ipv6 nf_log_ipv4
>> nf_log_common xt_LOG sctp libcrc32c e1000e tipc udp_tunnel
>> ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev br_netfilter bridge stp
>> llc nf_conntrack_ipv4 nf_defrag_ipv4 ipmiq_drv(O) sio_mmc(O)
>> ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
>> nf_conntrack lockd ip6table_filter event_drv(O) ip6_tables grace
>> pt_timer_info(O) ddi(O) usb_storage ixgbe igb i2c_i801
>> iTCO_vendor_support i2c_algo_bit ioatdma intel_ips i2c_core pcspkr
>> sunrpc ptp mdio dca pps_core lpc_ich tpm_tis mfd_core tpm [last
>> unloaded: iTCO_wdt]
>> [375832.573693] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G           O
>>  4.4.0 #14
>> [375832.581385] Hardware name: PT AMC124/Base Board Product Name, BIOS
>> LGNAJFIP.PTI.0012.P15 01/15/2014
>> [375832.591028] task: ffff880351a89b40 ti: ffff880351a90000 task.ti:
>> ffff880351a90000
>> [375832.599026] RIP: 0010:[<ffffffff810c3566>]  [<ffffffff810c3566>]
>> queued_spin_lock_slowpath+0xe6/0x160
>> [375832.608964] RSP: 0018:ffff88035fc83d58  EFLAGS: 00010002
>> [375832.614825] RAX: 0000000000001447 RBX: 0000000000000292 RCX:
>> ffff88035fc95fc0
>> [375832.622743] RDX: 000001a400015ff4 RSI: 0000000000140000 RDI:
>> ffff880351232f80
>> [375832.630567] RBP: ffff88035fc83d58 R08: 0000000000000101 R09:
>> 0000000000000004
>> [375832.638348] R10: 0000000000000000 R11: 0000000000000000 R12:
>> 0000000001001002
>> [375832.645919] R13: 0000000000000001 R14: 0000000000000000 R15:
>> 0000000000000000
>> [375832.653610] FS:  0000000000000000(0000) GS:ffff88035fc80000(0000)
>> knlGS:0000000000000000
>> [375832.662317] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [375832.668483] CR2: 000001a400015ff4 CR3: 0000000001c0a000 CR4:
>> 00000000000006e0
>> [375832.676133] Stack:
>> [375832.678344]  ffff88035fc83d78 ffffffff816de2c1 ffff88034a8bba60
>> ffff880351232f80
>> [375832.686163]  ffff88035fc83db8 ffffffff810bc592 ffff88035fc83dc8
>> ffff880351758000
>> [375832.694139]  0000000001001002 0000000000000000 00000000b802f4bd
>> ffffffffa024e6f0
>> [375832.702154] Call Trace:
>> [375832.704844]  <IRQ>
>> [375832.707018]  [<ffffffff816de2c1>] rqsav_raw_spin_lock_ie+0x31/0x40
>
> rqsav_raw_spin_lock_ie??
> Is this some proprietary spinlock implementation?
>
>> [375832.713970]  [<ffffffff810bc592>] __wake_up+0x32/0x70
>> [375832.719444]  [<ffffffffa024e6f0>] ? tipc_recv_stream+0x370/0x370
>> [tipc]
>> [375832.726589]  [<ffffffff815bf890>] sock_def_wakeup+0x30/0x40
>> [375832.732566]  [<ffffffffa024e838>] tipc_sk_timeout+0x148/0x180 [tipc]
>> [375832.739388]  [<ffffffffa024e6f0>] ? tipc_recv_stream+0x370/0x370
>> [tipc]
>> [375832.746507]  [<ffffffff810e1644>] call_timer_fn+0x44/0x110
>> [375832.752378]  [<ffffffff810e0e6a>] ? cascade+0x4a/0x80
>> [375832.757848]  [<ffffffffa024e6f0>] ? tipc_recv_stream+0x370/0x370
>> [tipc]
>> [375832.764871]  [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280
>> [375832.771175]  [<ffffffff81083d88>] __do_softirq+0xc8/0x260
>> [375832.776958]  [<ffffffff81084123>] irq_exit+0x83/0xb0
>> [375832.782369]  [<ffffffff816e0c65>] do_IRQ+0x65/0xf0
>> [375832.787607]  [<ffffffff816df13f>] common_interrupt+0x7f/0x7f
>> [375832.793709]  <EOI>
>> [375832.795803]  [<ffffffff81591bdd>] ? cpuidle_enter_state+0xad/0x200
>> [375832.802765]  [<ffffffff81591bc1>] ? cpuidle_enter_state+0x91/0x200
>> [375832.809338]  [<ffffffff81591d67>] cpuidle_enter+0x17/0x20
>> [375832.815155]  [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60
>> [375832.821184]  [<ffffffff81591d43>] ? cpuidle_select+0x13/0x20
>> [375832.827249]  [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0
>> [375832.833535]  [<ffffffff8103b213>] start_secondary+0x103/0x130
>> [375832.839759] Code: 87 47 02 c1 e0 10 85 c0 74 38 48 89 c2 c1 e8 12
>> 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 c0 5f 01 00 48 03 14 c5
>> 00 b2 d1 81 <48> 89 0a 8b 41 08 85 c0 75 0d f3 90 8b 41 08 85 c0 74 f7
>> eb 02
>> [375832.861151] RIP  [<ffffffff810c3566>]
>> queued_spin_lock_slowpath+0xe6/0x160
>> [375832.868607]  RSP <ffff88035fc83d58>
>> [375832.872371] CR2: 000001a400015ff4
>> [375832.876408] ---[ end trace f12e0074b180a165 ]---
>> [375832.881433] Kernel panic - not syncing: Fatal exception in interrupt
>> [375832.888391] Kernel Offset: disabled
>> [375832.891968] ---[ end Kernel panic - not syncing: Fatal exception
>> in interrupt
>>
>>
>> ------------------------------------------------------------------------------
>> Find and fix application performance issues faster with Applications
>> Manager
>> Applications Manager provides deep performance insights into multiple
>> tiers of
>> your business applications. It resolves application problems quickly and
>> reduces your MTTR. Get your free trial!
>> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
>> _______________________________________________
>> tipc-discussion mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/tipc-discussion

------------------------------------------------------------------------------
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
_______________________________________________
tipc-discussion mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

Reply via email to