Ying, Just to clarify that I have not changed anything on tipc_sk_timeout(). The 4.4.0 version already has jiffies change. The issue was seen with this code. I will try to apply the patch once it is available on upstream.
thanks Guna On Thu, Jun 2, 2016 at 6:57 AM, Xue, Ying <ying....@windriver.com> wrote: > Hi Guna, > > Please see my comments below. > > Regards, > Ying > > -----Original Message----- > From: GUNA [mailto:gbala...@gmail.com] > Sent: 2016年6月1日 23:26 > To: Xue, Ying > Cc: Jon Maloy; Jon Maloy; tipc-discussion@lists.sourceforge.net; Erik Hugne; > Xue Ying (ying.x...@gmail.com) > Subject: Re: [tipc-discussion] tipc_sk_rcv: Kernel panic on one of the card > on 4.4.0 > > If the issue is reproducible then I could try with Erik's patch even > though the root cause is unknown. Currently, we are not clear yet on > root cause of the issue. I could add the patch on production release, > only if the patch will fix the issue. > > [Ying] Understood your meaning. I think Erik's patch should be merged into > upstream whatever it can fix your issue or not. But in my opinion, it should > be able to fix it. > > Otherwise, I may need to find > test stream. > > As per Erik's proposal, > ====== > if (sock_owned_by_user(sk)) > we can reschedule timer for a retry in a few jiffies > ====== > > I tried to call sk_reset_timer(sk, &sk->sk_timer, (HZ / 20)); > but the code or similar already is in place at tipc_sk_timeout() as > marked by "<<==" below" > > if (tsk->probing_state == TIPC_CONN_PROBING) { > if (!sock_owned_by_user(sk)) > ... > else > sk_reset_timer(sk, &sk->sk_timer, (HZ / 20)); <<== > } else { > sk_reset_timer(sk, &sk->sk_timer, jiffies + tsk->probing_intv); <<== > } > > Please let me know If I need to add/modify any. > > [Ying] your change above is right and it should be workable. But I still > suggest you should adopt Erik's patch("tipc: fix timer handling when socket > is owned ") as it's much better than above solution. > > thanks, > Guna > > On Wed, Jun 1, 2016 at 7:31 AM, Xue, Ying <ying....@windriver.com> wrote: >> Hi GUNA, >> >> Thanks for your confirmation, which is very important for us to look into >> what happened in 4.4.0 version. >> Yes, my mentioned Erik's patch is just as Erik said: "tipc: fix timer >> handling when socket is owned". >> >> I also agree to Erik's solution as its change is more common method to deal >> with the case when owner flag is not set in BH. >> >> But now we still need to know what root cause is the issue. >> >> If possible, please apply Erik's patch on your side to check whether the >> issue occurs or not. >> >> Regards, >> Ying >> >> -----Original Message----- >> From: GUNA [mailto:gbala...@gmail.com] >> Sent: 2016年5月31日 23:34 >> To: Xue, Ying >> Cc: Jon Maloy; Jon Maloy; tipc-discussion@lists.sourceforge.net; Erik Hugne; >> Xue Ying (ying.x...@gmail.com) >> Subject: Re: [tipc-discussion] tipc_sk_rcv: Kernel panic on one of the card >> on 4.4.0 >> >> Just want to clarify, system was upgraded only the kernel from 3.4.2 >> to 4.4.0 + some tipc patches on Fedora distribution. That said, the >> patch, "net: do not block BH while processing socket backlog" is not >> part of the 4.4.0. So, the issue is not due to this commit. >> >> If the patch, "tipc: block BH in TCP callbacks" could resolve the >> issue then I could try applying the patch. However the issue is not >> reproducible. So, we may not get the result right away. >> >> Which Erik's patch you are talking about? >> Is this one, "tipc: fix timer handling when socket is owned" ? >> >> >> /// Guna >> >> On Tue, May 31, 2016 at 3:49 AM, Xue, Ying <ying....@windriver.com> wrote: >>> Hi Jon, >>> >>> Today, I spent time further analyzing its potential root cause why the soft >>> lockup occurred regarding log provided by GUNA. But I don't find some >>> valuable hints. >>> >>> To be honest, even if CONN_MANAGER/CONN_PROBE message is sent through >>> tipc_node_xmit_skb() without holding "owner" flag in tipc_sk_timeout(), >>> deadlock should not happen in theory. Before the tipc_sk_rcv() is secondly >>> called, destination port and source port of CONN_MANAGER/CONN_PROBE message >>> created in tipc_sk_timeout() have been reversed. As a result, the tsk found >>> at (*1) is different with another tsk found at (*2) because we use >>> different destination number to look up tsk instances. >>> >>> tipc_sk_timeout() >>> create: CONN_MANAGER/CONN_PROBE msg (src port= tsk->portid, dst port = >>> peer_port) >>> tipc_node_xmit_skb() >>> tipc_node_xmit() >>> tipc_sk_rcv() >>> tsk = tipc_sk_lookup(net, dport); // use dst port(peer_port) to >>> look up tsk, and the tsk is called tsk1 ------------(*1) >>> if (likely(spin_trylock_bh(&sk->sk_lock.slock))) >>> tipc_sk_enqueue() >>> filter_rcv() >>> tipc_sk_proto_rcv() >>> tipc_sk_respond() >>> reverse ports: dport = tsk->portid; src port = >>> peer_port >>> tipc_node_xmit_skb() >>> tipc_node_xmit() >>> tipc_sk_rcv() >>> tsk = tipc_sk_lookup(net, dport); // use dst >>> port(portid) to look up tsk, and the tsk is supposed as tsk2 ------(*2) >>> if >>> (likely(spin_trylock_bh(&sk->sk_lock.slock))) >>> >>> So even if "owner" flag of tsk1 is not set, it's safe for us to operate >>> tsk2 in the same BH context. >>> >>> I also agree with you. Although Erik's patch might solve the issue, we >>> still need to further find its root cause. >>> >>> Additionally, I suspect there is a certain relationship between the issue >>> and 5413d1babe8f10de13d72496c12b862eef8ba613 (net: do not block BH while >>> processing socket backlog). >>> I don't know whether it's easy to reproduce the issue. But I suggest we can >>> revert above commit or apply the following patch to verify whether the >>> issue is related to the commit. >>> >>> http://www.spinics.net/lists/netdev/msg378109.html >>> >>> Regards, >>> Ying >>> >>> -----Original Message----- >>> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >>> Sent: 2016年5月30日 22:43 >>> To: Xue, Ying; GUNA; Jon Maloy; tipc-discussion@lists.sourceforge.net; Erik >>> Hugne; Xue Ying (ying.x...@gmail.com) >>> Subject: RE: [tipc-discussion] tipc_sk_rcv: Kernel panic on one of the card >>> on 4.4.0 >>> >>> >>> >>>> -----Original Message----- >>>> From: Xue, Ying [mailto:ying....@windriver.com] >>>> Sent: Monday, 30 May, 2016 14:15 >>>> To: Jon Maloy; GUNA; Jon Maloy; tipc-discussion@lists.sourceforge.net; Erik >>>> Hugne; Xue Ying (ying.x...@gmail.com) >>>> Subject: RE: [tipc-discussion] tipc_sk_rcv: Kernel panic on one of the >>>> card on 4.4.0 >>>> >>>> Hi Jon, >>>> >>> ... >>> >>>> In fact we ever proposed several possible solutions of how to well deal >>>> with the >>>> concurrent scenario in skb receive path. Especially, we need to stay BH >>>> mode to >>>> directly forward back a skb received in BH, which very easily causes the >>>> problem >>>> that a skb will be routed back and forth, leading deadlock occurs. >>> >>> We stay in BH, but we do *not* keep slock during the send/receive sequence, >>> so this cannot be the cause of the problem. On the receive path, this >>> should work just and any other message received in softirq. >>> >>> Anyway, I am not sure at all this is a deadlock concerning slock (see >>> earlier mail), but rather a message arriving to a socket that has been >>> corrupted by interference between the timer in softirq and tipc_recv_xxx() >>> in user context. Maybe a state change? >>> >>> I think a slightly improved version of what Erik suggested might solve the >>> issue, but I won't feel comfortable until we understand completely what is >>> happening. >>> >>> ///jon >>> >>>> Unfortunately, >>>> so far it seems that we are still unable to perfectly solve the issue. >>>> Therefore, >>>> maybe we have to radically change the current mechanism of receiving skb >>>> and >>>> forward skb on BH mode. >>>> >>>> Regards, >>>> Ying >>>> >>>> -----Original Message----- >>>> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >>>> Sent: 2016年5月30日 5:32 >>>> To: GUNA; Jon Maloy; tipc-discussion@lists.sourceforge.net; Erik Hugne; >>>> Xue, >>>> Ying; Xue Ying (ying.x...@gmail.com) >>>> Subject: RE: [tipc-discussion] tipc_sk_rcv: Kernel panic on one of the >>>> card on 4.4.0 >>>> >>>> Hi Guna, >>>> I am looking at it, but don't have much time to spend at it right now. >>>> A further study of your dump makes me believe this is a case of race >>>> between >>>> tipc_recv_stream(), which in user context is setting the "owned" flag but >>>> not >>>> grabbing slock, and tipc_sk_timeout(), which in interrupt mode is grabbing >>>> slock >>>> without checking the "owned" flag. This creates an obvious risk of a >>>> race, but I >>>> don't yet understand the exact implications or solution to it. It may be >>>> that it is >>>> sufficient to just replace the grabbing of slock with a call to >>>> lock_sock() in the >>>> timer function as I first suggested, but I rather believe we have to >>>> explicitly grab >>>> both slock and the flag and keep them for the duration of the timeout, >>>> before >>>> tipc_node_xmit_skb(). This will protect the timeout against interference >>>> from >>>> both user context (tipc_recv_stream() and other sw interrups >>>> (tipc_sk_rcv()) and >>>> vice versa. Or maybe not even this is enough? >>>> >>>> I will continue my analysis, but input from others would be appreciated. >>>> >>>> ///jon >>>> >>>> >>>> > -----Original Message----- >>>> > From: GUNA [mailto:gbala...@gmail.com] >>>> > Sent: Saturday, 28 May, 2016 06:00 >>>> > To: Jon Maloy; tipc-discussion@lists.sourceforge.net; Erik Hugne >>>> > Subject: Re: [tipc-discussion] tipc_sk_rcv: Kernel panic on one of the >>>> > card on >>>> 4.4.0 >>>> > >>>> > Any update on the issue? Any other thoughts or possible fix ? >>>> > >>>> > The issue was seen on slot12 (1.1.12) node only. The other slots were up. >>>> > >>>> > I got the full logs as listed here: >>>> > >>>> > >>>> > May 19 05:03:01 [SEQ 248049] dcsx5testslot13 /USR/SBIN/CROND[11359]: >>>> > (root) CMD (/opt/cpu_ss7gw/current/scripts/mgmt_apache_watchdog) >>>> > May 19 05:03:21 [SEQ 249182] dcsx5testslot12 kernel: [673637.606852] >>>> > NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0] >>>> > May 19 05:03:21 [SEQ 249183] dcsx5testslot12 kernel: [673637.607791] >>>> > NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0] >>>> > May 19 05:03:21 [SEQ 249184] dcsx5testslot12 kernel: [673637.607817] >>>> > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c >>>> > e1000e tip >>>> > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev >>>> > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) >>>> > nf_defrag_ipv4 ip6 >>>> > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state >>>> > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd >>>> > pt_timer_info(O >>>> > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit >>>> > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core >>>> > intel_ips >>>> > tpm mdio sunrpc [last unloaded: iTCO_wdt] >>>> > May 19 05:03:21 [SEQ 249185] dcsx5testslot12 kernel: [673637.607819] >>>> > CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O 4.4.0 #17 >>>> > May 19 05:03:21 [SEQ 249186] dcsx5testslot12 kernel: [673637.607820] >>>> > Hardware name: PT AMC124/Base Board Product Name, BIOS >>>> > LGNAJFIP.PTI.0012.P15 >>>> > 01/15/2014 >>>> > May 19 05:03:21 [SEQ 249187] dcsx5testslot12 kernel: [673637.607821] >>>> > task: ffff880351a88000 ti: ffff880351a84000 task.ti: ffff880351a84000 >>>> > May 19 05:03:21 [SEQ 249188] dcsx5testslot12 kernel: [673637.607827] >>>> > RIP: 0010:[<ffffffff81361750>] [<ffffffff81361750>] >>>> > rhashtable_jhash2+0x0/0x >>>> > f0 >>>> > May 19 05:03:21 [SEQ 249189] dcsx5testslot12 kernel: [673637.607828] >>>> > RSP: 0018:ffff88035fc63a60 EFLAGS: 00000206 >>>> > May 19 05:03:21 [SEQ 249190] dcsx5testslot12 kernel: [673637.607829] >>>> > RAX: ffff880351613a80 RBX: ffff880347948000 RCX: ffff880347949060 >>>> > May 19 05:03:21 [SEQ 249191] dcsx5testslot12 kernel: [673637.607830] >>>> > RDX: 0000000003ff6972 RSI: 0000000000000001 RDI: ffff88035fc63a84 >>>> > May 19 05:03:21 [SEQ 249192] dcsx5testslot12 kernel: [673637.607830] >>>> > RBP: ffff88035fc63ab8 R08: 0000000000000001 R09: 0000000000000004 >>>> > May 19 05:03:21 [SEQ 249193] dcsx5testslot12 kernel: [673637.607831] >>>> > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc63bd0 >>>> > May 19 05:03:21 [SEQ 249194] dcsx5testslot12 kernel: [673637.607832] >>>> > R13: ffff88009a4dc000 R14: ffff8803502f5640 R15: ffff88035fc63be4 >>>> > May 19 05:03:21 [SEQ 249195] dcsx5testslot12 kernel: [673637.607833] >>>> > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) >>>> > knlGS:0000000000000000 >>>> > May 19 05:03:21 [SEQ 249196] dcsx5testslot12 kernel: [673637.607834] >>>> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>>> > May 19 05:03:21 [SEQ 249197] dcsx5testslot12 kernel: [673637.607835] >>>> > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 >>>> > May 19 05:03:21 [SEQ 249198] dcsx5testslot12 kernel: [673637.607835] >>>> > Stack: >>>> > :6600 >>>> > May 19 05:04:45 [SEQ 252476] dcsx5testslot12 kernel: [673721.608818] >>>> > NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [swapper/3:0] >>>> > May 19 05:04:45 [SEQ 252477] dcsx5testslot12 kernel: [673721.608844] >>>> > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c >>>> > e1000e tip >>>> > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev >>>> > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) >>>> > nf_defrag_ipv4 ip6 >>>> > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state >>>> > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd >>>> > pt_timer_info(O >>>> > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit >>>> > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core >>>> > intel_ips >>>> > tpm mdio sunrpc [last unloaded: iTCO_wdt] >>>> > May 19 05:04:45 [SEQ 252478] dcsx5testslot12 kernel: [673721.608847] >>>> > CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O L 4.4.0 #17 >>>> > May 19 05:04:45 [SEQ 252479] dcsx5testslot12 kernel: [673721.608848] >>>> > Hardware name: PT AMC124/Base Board Product Name, BIOS >>>> > LGNAJFIP.PTI.0012.P15 >>>> > 01/15/2014 >>>> > May 19 05:04:45 [SEQ 252480] dcsx5testslot12 kernel: [673721.608849] >>>> > task: ffff880351a88000 ti: ffff880351a84000 task.ti: ffff880351a84000 >>>> > May 19 05:04:45 [SEQ 252481] dcsx5testslot12 kernel: [673721.608858] >>>> > RIP: 0010:[<ffffffffa01546a5>] [<ffffffffa01546a5>] >>>> > tipc_sk_rcv+0x95/0x490 [ >>>> > tipc] >>>> > May 19 05:04:45 [SEQ 252482] dcsx5testslot12 kernel: [673721.608859] >>>> > RSP: 0018:ffff88035fc63ac8 EFLAGS: 00000296 >>>> > May 19 05:04:45 [SEQ 252483] dcsx5testslot12 kernel: [673721.608860] >>>> > RAX: 0000000000000301 RBX: ffff88035fc63b70 RCX: 0000000000000001 >>>> > May 19 05:04:45 [SEQ 252484] dcsx5testslot12 kernel: [673721.608861] >>>> > RDX: 00000000b2db9944 RSI: 0000000000000200 RDI: ffffffff81ce6240 >>>> > May 19 05:04:45 [SEQ 252485] dcsx5testslot12 kernel: [673721.608862] >>>> > RBP: ffff88035fc63b38 R08: 0000000000000001 R09: 0000000000000004 >>>> > May 19 05:04:45 [SEQ 252486] dcsx5testslot12 kernel: [673721.608862] >>>> > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc63bd0 >>>> > May 19 05:04:45 [SEQ 252487] dcsx5testslot12 kernel: [673721.608863] >>>> > R13: 00000000b2db9944 R14: ffff8803502f5640 R15: ffff88035fc63be4 >>>> > May 19 05:04:45 [SEQ 252488] dcsx5testslot12 kernel: [673721.608864] >>>> > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) >>>> > knlGS:0000000000000000 >>>> > May 19 05:04:45 [SEQ 252489] dcsx5testslot12 kernel: [673721.608865] >>>> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>>> > May 19 05:04:45 [SEQ 252490] dcsx5testslot12 kernel: [673721.608866] >>>> > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 >>>> > May 19 05:04:45 [SEQ 252491] dcsx5testslot12 kernel: [673721.608867] >>>> > Stack: >>>> > May 19 05:04:45 [SEQ 252492] dcsx5testslot12 kernel: [673721.608868] >>>> > ffff880350b37000 ffffffffa03017eb ffff88035fc63b7c ffffffff81ce6240 >>>> > May 19 05:04:45 [SEQ 252493] dcsx5testslot12 kernel: [673721.608870] >>>> > ffff8803502f56c8 000000012821c002 ffff880300000000 02c14a9800000000 >>>> > May 19 05:04:45 [SEQ 252494] dcsx5testslot12 kernel: [673721.608871] >>>> > ffff88035fc63b48 ffff88035fc63b70 0000000000000000 000000000100100c >>>> > May 19 05:04:45 [SEQ 252495] dcsx5testslot12 kernel: [673721.608871] >>>> > Call Trace: >>>> > May 19 05:04:45 [SEQ 252496] dcsx5testslot12 kernel: [673721.608882] >>>> > <IRQ> >>>> > May 19 05:04:45 [SEQ 252497] dcsx5testslot12 kernel: [673721.608882] >>>> > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] >>>> > May 19 05:04:45 [SEQ 252498] dcsx5testslot12 kernel: [673721.608887] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:04:45 [SEQ 252499] dcsx5testslot12 kernel: [673721.608890] >>>> > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 >>>> > May 19 05:04:45 [SEQ 252500] dcsx5testslot12 kernel: [673721.608894] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:04:45 [SEQ 252501] dcsx5testslot12 kernel: [673721.608900] >>>> > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] >>>> > May 19 05:04:45 [SEQ 252502] dcsx5testslot12 kernel: [673721.608903] >>>> > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] >>>> > May 19 05:04:45 [SEQ 252503] dcsx5testslot12 kernel: [673721.608907] >>>> > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] >>>> > May 19 05:04:45 [SEQ 252504] dcsx5testslot12 kernel: [673721.608910] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:04:45 [SEQ 252505] dcsx5testslot12 kernel: [673721.608913] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:04:45 [SEQ 252506] dcsx5testslot12 kernel: [673721.608916] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:04:45 [SEQ 252507] dcsx5testslot12 kernel: [673721.608919] >>>> > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] >>>> > May 19 05:04:45 [SEQ 252508] dcsx5testslot12 kernel: [673721.608923] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:04:45 [SEQ 252509] dcsx5testslot12 kernel: [673721.608925] >>>> > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 >>>> > May 19 05:04:45 [SEQ 252510] dcsx5testslot12 kernel: [673721.608926] >>>> > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 >>>> > May 19 05:04:45 [SEQ 252511] dcsx5testslot12 kernel: [673721.608929] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:04:45 [SEQ 252512] dcsx5testslot12 kernel: [673721.608930] >>>> > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 >>>> > May 19 05:04:45 [SEQ 252513] dcsx5testslot12 kernel: [673721.608933] >>>> > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 >>>> > May 19 05:04:45 [SEQ 252514] dcsx5testslot12 kernel: [673721.608934] >>>> > [<ffffffff81084123>] irq_exit+0x83/0xb0 >>>> > May 19 05:04:45 [SEQ 252515] dcsx5testslot12 kernel: [673721.608937] >>>> > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 >>>> > :6800 >>>> > May 19 05:05:13 [SEQ 254646] dcsx5testslot12 kernel: [673749.609200] >>>> > RDX: 00000000b2db9944 RSI: 0000000000000200 RDI: ffffffff81ce6240 >>>> > May 19 05:05:13 [SEQ 254647] dcsx5testslot12 kernel: [673749.609200] >>>> > RBP: ffff88035fc63b38 R08: 0000000000000001 R09: 0000000000000004 >>>> > May 19 05:05:13 [SEQ 254648] dcsx5testslot12 kernel: [673749.609201] >>>> > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc63bd0 >>>> > May 19 05:05:13 [SEQ 254649] dcsx5testslot12 kernel: [673749.609202] >>>> > R13: 00000000b2db9944 R14: ffff8803502f5640 R15: ffff88035fc63be4 >>>> > May 19 05:05:13 [SEQ 254650] dcsx5testslot12 kernel: [673749.609203] >>>> > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) >>>> > knlGS:0000000000000000 >>>> > May 19 05:05:13 [SEQ 254651] dcsx5testslot12 kernel: [673749.609205] >>>> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>>> > May 19 05:05:13 [SEQ 254652] dcsx5testslot12 kernel: [673749.609206] >>>> > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 >>>> > May 19 05:05:13 [SEQ 254653] dcsx5testslot12 kernel: [673749.609207] >>>> > Stack: >>>> > May 19 05:05:13 [SEQ 254654] dcsx5testslot12 kernel: [673749.609209] >>>> > ffff880350b37000 ffffffffa03017eb ffff88035fc63b7c ffffffff81ce6240 >>>> > May 19 05:05:13 [SEQ 254655] dcsx5testslot12 kernel: [673749.609211] >>>> > ffff8803502f56c8 000000012821c002 ffff880300000000 02c14a9800000000 >>>> > May 19 05:05:13 [SEQ 254656] dcsx5testslot12 kernel: [673749.609212] >>>> > ffff88035fc63b48 ffff88035fc63b70 0000000000000000 000000000100100c >>>> > May 19 05:05:13 [SEQ 254657] dcsx5testslot12 kernel: [673749.609213] >>>> > Call Trace: >>>> > May 19 05:05:13 [SEQ 254658] dcsx5testslot12 kernel: [673749.609224] >>>> > <IRQ> >>>> > May 19 05:05:13 [SEQ 254659] dcsx5testslot12 kernel: [673749.609224] >>>> > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] >>>> > May 19 05:05:13 [SEQ 254660] dcsx5testslot12 kernel: [673749.609228] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:05:13 [SEQ 254661] dcsx5testslot12 kernel: [673749.609232] >>>> > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 >>>> > May 19 05:05:13 [SEQ 254662] dcsx5testslot12 kernel: [673749.609235] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:05:13 [SEQ 254663] dcsx5testslot12 kernel: [673749.609239] >>>> > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] >>>> > May 19 05:05:13 [SEQ 254664] dcsx5testslot12 kernel: [673749.609242] >>>> > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] >>>> > May 19 05:05:13 [SEQ 254665] dcsx5testslot12 kernel: [673749.609245] >>>> > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] >>>> > May 19 05:05:13 [SEQ 254666] dcsx5testslot12 kernel: [673749.609249] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:05:13 [SEQ 254667] dcsx5testslot12 kernel: [673749.609252] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:13 [SEQ 254668] dcsx5testslot12 kernel: [673749.609255] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:05:13 [SEQ 254669] dcsx5testslot12 kernel: [673749.609259] >>>> > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] >>>> > May 19 05:05:13 [SEQ 254670] dcsx5testslot12 kernel: [673749.609262] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:13 [SEQ 254671] dcsx5testslot12 kernel: [673749.609264] >>>> > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 >>>> > May 19 05:05:13 [SEQ 254672] dcsx5testslot12 kernel: [673749.609265] >>>> > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 >>>> > May 19 05:05:13 [SEQ 254673] dcsx5testslot12 kernel: [673749.609269] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:13 [SEQ 254674] dcsx5testslot12 kernel: [673749.609270] >>>> > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 >>>> > May 19 05:05:13 [SEQ 254675] dcsx5testslot12 kernel: [673749.609272] >>>> > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 >>>> > May 19 05:05:13 [SEQ 254676] dcsx5testslot12 kernel: [673749.609274] >>>> > [<ffffffff81084123>] irq_exit+0x83/0xb0 >>>> > May 19 05:05:13 [SEQ 254677] dcsx5testslot12 kernel: [673749.609277] >>>> > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 >>>> > May 19 05:05:13 [SEQ 254678] dcsx5testslot12 kernel: [673749.609279] >>>> > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f >>>> > May 19 05:05:13 [SEQ 254679] dcsx5testslot12 kernel: [673749.609282] >>>> > <EOI> >>>> > May 19 05:05:13 [SEQ 254680] dcsx5testslot12 kernel: [673749.609282] >>>> > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 >>>> > May 19 05:05:13 [SEQ 254681] dcsx5testslot12 kernel: [673749.609284] >>>> > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 >>>> > May 19 05:05:13 [SEQ 254682] dcsx5testslot12 kernel: [673749.609285] >>>> > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 >>>> > May 19 05:05:13 [SEQ 254683] dcsx5testslot12 kernel: [673749.609287] >>>> > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 >>>> > May 19 05:05:13 [SEQ 254684] dcsx5testslot12 kernel: [673749.609291] >>>> > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 >>>> > May 19 05:05:13 [SEQ 254685] dcsx5testslot12 kernel: [673749.609292] >>>> > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 >>>> > May 19 05:05:13 [SEQ 254686] dcsx5testslot12 kernel: [673749.609295] >>>> > [<ffffffff8103b213>] start_secondary+0x103/0x130 >>>> > May 19 05:05:13 [SEQ 254687] dcsx5testslot12 kernel: [673749.609309] >>>> > Code: 7d 02 00 00 84 c9 74 14 48 8b 90 d8 00 00 00 48 8b 00 8b 52 14 >>>> > 49 39 c4 >>>> > 0f ca 75 df 41 89 d5 4c 89 ff e8 ff 9a 58 e1 48 8b 7d a8 <44> 89 ee >>>> > e8 13 ec ff ff 48 85 c0 49 89 c6 0f 84 62 02 00 00 48 >>>> > May 19 05:05:13 [SEQ 254688] dcsx5testslot12 kernel: [673749.986573] >>>> > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c >>>> > e1000e tip >>>> > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev >>>> > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) >>>> > nf_defrag_ipv4 ip6 >>>> > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state >>>> > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd >>>> > pt_timer_info(O >>>> > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit >>>> > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core >>>> > intel_ips >>>> > tpm mdio sunrpc [last unloaded: iTCO_wdt] >>>> > May 19 05:05:13 [SEQ 254689] dcsx5testslot12 kernel: [673750.038404] >>>> > CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O L 4.4.0 #17 >>>> > May 19 05:05:13 [SEQ 254690] dcsx5testslot12 kernel: [673750.045635] >>>> > Hardware name: PT AMC124/Base Board Product Name, BIOS >>>> > LGNAJFIP.PTI.0012.P15 >>>> > 01/15/2014 >>>> > May 19 05:05:13 [SEQ 254691] dcsx5testslot12 kernel: [673750.054809] >>>> > task: ffff880351a751c0 ti: ffff880351a80000 task.ti: ffff880351a80000 >>>> > May 19 05:05:13 [SEQ 254692] dcsx5testslot12 kernel: [673750.062396] >>>> > RIP: 0010:[<ffffffff81083fda>] [<ffffffff81083fda>] >>>> > __local_bh_enable_ip+0x3 >>>> > a/0x90 >>>> > May 19 05:05:13 [SEQ 254693] dcsx5testslot12 kernel: [673750.071327] >>>> > RSP: 0018:ffff88035fc43a98 EFLAGS: 00000203 >>>> > May 19 05:05:13 [SEQ 254694] dcsx5testslot12 kernel: [673750.076881] >>>> > RAX: 0000000000000301 RBX: 00000000fffffe01 RCX: 0000000000000001 >>>> > May 19 05:05:13 [SEQ 254695] dcsx5testslot12 kernel: [673750.084226] >>>> > RDX: 0000000000000003 RSI: 0000000000000200 RDI: ffffffffa01546cc >>>> > May 19 05:05:13 [SEQ 254696] dcsx5testslot12 kernel: [673750.091590] >>>> > RBP: ffff88035fc43aa8 R08: 00000000000002d4 R09: 0000000000000004 >>>> > May 19 05:05:13 [SEQ 254697] dcsx5testslot12 kernel: [673750.099014] >>>> > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc43bd0 >>>> > May 19 05:05:13 [SEQ 254698] dcsx5testslot12 kernel: [673750.106368] >>>> > R13: 00000000ab7500c6 R14: ffff880350124ec0 R15: ffff88035fc43be4 >>>> > May 19 05:05:13 [SEQ 254699] dcsx5testslot12 kernel: [673750.113679] >>>> > FS: 0000000000000000(0000) GS:ffff88035fc40000(0000) >>>> > knlGS:0000000000000000 >>>> > May 19 05:05:13 [SEQ 254700] dcsx5testslot12 kernel: [673750.122083] >>>> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>>> > May 19 05:05:13 [SEQ 254701] dcsx5testslot12 kernel: [673750.127989] >>>> > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 >>>> > May 19 05:05:13 [SEQ 254702] dcsx5testslot12 kernel: [673750.135328] >>>> > Stack: >>>> > May 19 05:05:13 [SEQ 254703] dcsx5testslot12 kernel: [673750.137503] >>>> > ffff88035fc43bd0 ffff88035fc43b70 ffff88035fc43ab8 ffffffff816de340 >>>> > May 19 05:05:13 [SEQ 254704] dcsx5testslot12 kernel: [673750.145228] >>>> > ffff88035fc43b38 ffffffffa01546cc ffff88034a8fad10 0000000000000020 >>>> > May 19 05:05:13 [SEQ 254705] dcsx5testslot12 kernel: [673750.152927] >>>> > ffff88035fc43af8 ffffffff81ce6240 ffff880350124f48 000000012821c002 >>>> > May 19 05:05:13 [SEQ 254706] dcsx5testslot12 kernel: [673750.160645] >>>> > Call Trace: >>>> > May 19 05:05:13 [SEQ 254707] dcsx5testslot12 kernel: [673750.163228] >>>> > <IRQ> >>>> > May 19 05:05:13 [SEQ 254708] dcsx5testslot12 kernel: [673750.165257] >>>> > [<ffffffff816de340>] _raw_spin_trylock_bh+0x40/0x50 >>>> > May 19 05:05:13 [SEQ 254709] dcsx5testslot12 kernel: [673750.171642] >>>> > [<ffffffffa01546cc>] tipc_sk_rcv+0xbc/0x490 [tipc] >>>> > May 19 05:05:13 [SEQ 254710] dcsx5testslot12 kernel: [673750.177847] >>>> > [<ffffffff8162ecde>] ? tcp_rcv_established+0x40e/0x760 >>>> > May 19 05:05:13 [SEQ 254711] dcsx5testslot12 kernel: [673750.184297] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:05:13 [SEQ 254712] dcsx5testslot12 kernel: [673750.190730] >>>> > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 >>>> > May 19 05:05:13 [SEQ 254713] dcsx5testslot12 kernel: [673750.197159] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:05:13 [SEQ 254714] dcsx5testslot12 kernel: [673750.203846] >>>> > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] >>>> > May 19 05:05:13 [SEQ 254715] dcsx5testslot12 kernel: [673750.210235] >>>> > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] >>>> > May 19 05:05:13 [SEQ 254716] dcsx5testslot12 kernel: [673750.216391] >>>> > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] >>>> > May 19 05:05:13 [SEQ 254717] dcsx5testslot12 kernel: [673750.222523] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:05:13 [SEQ 254718] dcsx5testslot12 kernel: [673750.228950] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:13 [SEQ 254719] dcsx5testslot12 kernel: [673750.235862] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:05:13 [SEQ 254720] dcsx5testslot12 kernel: [673750.242516] >>>> > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] >>>> > May 19 05:05:13 [SEQ 254721] dcsx5testslot12 kernel: [673750.249037] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:13 [SEQ 254722] dcsx5testslot12 kernel: [673750.255863] >>>> > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 >>>> > May 19 05:05:13 [SEQ 254723] dcsx5testslot12 kernel: [673750.261477] >>>> > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 >>>> > May 19 05:05:13 [SEQ 254724] dcsx5testslot12 kernel: [673750.266684] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:13 [SEQ 254725] dcsx5testslot12 kernel: [673750.273508] >>>> > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 >>>> > May 19 05:05:13 [SEQ 254726] dcsx5testslot12 kernel: [673750.279572] >>>> > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 >>>> > May 19 05:05:13 [SEQ 254727] dcsx5testslot12 kernel: [673750.285152] >>>> > [<ffffffff81084123>] irq_exit+0x83/0xb0 >>>> > May 19 05:05:13 [SEQ 254728] dcsx5testslot12 kernel: [673750.290309] >>>> > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 >>>> > May 19 05:05:13 [SEQ 254729] dcsx5testslot12 kernel: [673750.295338] >>>> > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f >>>> > May 19 05:05:13 [SEQ 254730] dcsx5testslot12 kernel: [673750.301132] >>>> > <EOI> >>>> > May 19 05:05:13 [SEQ 254731] dcsx5testslot12 kernel: [673750.303207] >>>> > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 >>>> > May 19 05:05:13 [SEQ 254732] dcsx5testslot12 kernel: [673750.309734] >>>> > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 >>>> > May 19 05:05:13 [SEQ 254733] dcsx5testslot12 kernel: [673750.316040] >>>> > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 >>>> > May 19 05:05:13 [SEQ 254734] dcsx5testslot12 kernel: [673750.321603] >>>> > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 >>>> > May 19 05:05:13 [SEQ 254735] dcsx5testslot12 kernel: [673750.327128] >>>> > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 >>>> > May 19 05:05:13 [SEQ 254736] dcsx5testslot12 kernel: [673750.332915] >>>> > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 >>>> > May 19 05:05:13 [SEQ 254737] dcsx5testslot12 kernel: [673750.339033] >>>> > [<ffffffff8103b213>] start_secondary+0x103/0x130 >>>> > May 19 05:05:13 [SEQ 254738] dcsx5testslot12 kernel: [673750.345007] >>>> > Code: 08 65 8b 05 21 6e f8 7e a9 00 00 0f 00 74 31 80 3d 49 80 c7 00 >>>> > 01 75 31 >>>> > 83 eb 01 f7 db 65 01 1d 05 6e f8 7e 65 8b 05 fe 6d f8 7e <a9> 00 ff >>>> > 1f 00 74 31 65 ff 0d f0 6d f8 7e 48 83 c4 08 5b 5d c3 >>>> > May 19 05:05:17 [SEQ 254974] dcsx5testslot2 kernel: [673706.438920] >>>> > TCP: request_sock_TCP: Possible SYN flooding on port 4322. Sending >>>> > cookies. Ch >>>> > eck SNMP counters. >>>> > May 19 05:05:26 [SEQ 255213] dcsx5testslot12 /USR/SBIN/CROND[9851]: >>>> > (root) CMD (/sbin/corecleanup) >>>> > May 19 05:05:41 [SEQ 255727] dcsx5testslot12 kernel: [673777.608487] >>>> > NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [swapper/2:0] >>>> > May 19 05:05:41 [SEQ 255728] dcsx5testslot12 kernel: [673777.609460] >>>> > NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [swapper/3:0] >>>> > May 19 05:05:41 [SEQ 255729] dcsx5testslot12 kernel: [673777.609482] >>>> > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c >>>> > e1000e tip >>>> > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev >>>> > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) >>>> > nf_defrag_ipv4 ip6 >>>> > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state >>>> > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd >>>> > pt_timer_info(O >>>> > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit >>>> > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core >>>> > intel_ips >>>> > tpm mdio sunrpc [last unloaded: iTCO_wdt] >>>> > May 19 05:05:41 [SEQ 255730] dcsx5testslot12 kernel: [673777.609485] >>>> > CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O L 4.4.0 #17 >>>> > May 19 05:05:41 [SEQ 255731] dcsx5testslot12 kernel: [673777.609485] >>>> > Hardware name: PT AMC124/Base Board Product Name, BIOS >>>> > LGNAJFIP.PTI.0012.P15 >>>> > 01/15/2014 >>>> > May 19 05:05:41 [SEQ 255732] dcsx5testslot12 kernel: [673777.609486] >>>> > task: ffff880351a88000 ti: ffff880351a84000 task.ti: ffff880351a84000 >>>> > May 19 05:05:41 [SEQ 255733] dcsx5testslot12 kernel: [673777.609492] >>>> > RIP: 0010:[<ffffffff81352b4d>] [<ffffffff81352b4d>] memcmp+0x1d/0x50 >>>> > May 19 05:05:41 [SEQ 255734] dcsx5testslot12 kernel: [673777.609493] >>>> > RSP: 0018:ffff88035fc63a58 EFLAGS: 00000297 >>>> > May 19 05:05:41 [SEQ 255735] dcsx5testslot12 kernel: [673777.609494] >>>> > RAX: 0000000000000000 RBX: ffff880347948000 RCX: 0000000000000002 >>>> > May 19 05:05:41 [SEQ 255736] dcsx5testslot12 kernel: [673777.609494] >>>> > RDX: 0000000000000003 RSI: ffff88035fc63a84 RDI: ffff8803502f5914 >>>> > May 19 05:05:41 [SEQ 255737] dcsx5testslot12 kernel: [673777.609495] >>>> > RBP: ffff88035fc63a58 R08: 00000000000000db R09: 0000000000000000 >>>> > May 19 05:05:41 [SEQ 255738] dcsx5testslot12 kernel: [673777.609496] >>>> > R10: 0000000000000000 R11: 0000000000000000 R12: ffff8803502f5640 >>>> > May 19 05:05:41 [SEQ 255739] dcsx5testslot12 kernel: [673777.609496] >>>> > R13: ffff88009a4dc000 R14: ffff8803502f59b0 R15: fffffffffffffc90 >>>> > May 19 05:05:41 [SEQ 255740] dcsx5testslot12 kernel: [673777.609498] >>>> > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) >>>> > knlGS:0000000000000000 >>>> > May 19 05:05:41 [SEQ 255741] dcsx5testslot12 kernel: [673777.609498] >>>> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>>> > May 19 05:05:41 [SEQ 255742] dcsx5testslot12 kernel: [673777.609499] >>>> > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 >>>> > May 19 05:05:41 [SEQ 255743] dcsx5testslot12 kernel: [673777.609500] >>>> > Stack: >>>> > May 19 05:05:41 [SEQ 255744] dcsx5testslot12 kernel: [673777.609501] >>>> > ffff88035fc63ab8 ffffffffa0153372 00000000000002d4 0000000000000004 >>>> > May 19 05:05:41 [SEQ 255745] dcsx5testslot12 kernel: [673777.609503] >>>> > ffff880347949060 b2db99445fc63a90 0000000000000018 ffff88035fc63b70 >>>> > May 19 05:05:41 [SEQ 255746] dcsx5testslot12 kernel: [673777.609504] >>>> > ffff88035fc63bd0 00000000b2db9944 ffff8803502f5640 ffff88035fc63be4 >>>> > May 19 05:05:41 [SEQ 255747] dcsx5testslot12 kernel: [673777.609504] >>>> > Call Trace: >>>> > May 19 05:05:41 [SEQ 255748] dcsx5testslot12 kernel: [673777.609512] >>>> > <IRQ> >>>> > May 19 05:05:41 [SEQ 255749] dcsx5testslot12 kernel: [673777.609512] >>>> > [<ffffffffa0153372>] tipc_sk_lookup+0xb2/0x110 [tipc] >>>> > May 19 05:05:41 [SEQ 255750] dcsx5testslot12 kernel: [673777.609520] >>>> > [<ffffffffa01546ad>] tipc_sk_rcv+0x9d/0x490 [tipc] >>>> > May 19 05:05:41 [SEQ 255751] dcsx5testslot12 kernel: [673777.609529] >>>> > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] >>>> > May 19 05:05:41 [SEQ 255752] dcsx5testslot12 kernel: [673777.609534] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:05:41 [SEQ 255753] dcsx5testslot12 kernel: [673777.609537] >>>> > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 >>>> > May 19 05:05:41 [SEQ 255754] dcsx5testslot12 kernel: [673777.609540] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:05:41 [SEQ 255755] dcsx5testslot12 kernel: [673777.609544] >>>> > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] >>>> > May 19 05:05:41 [SEQ 255756] dcsx5testslot12 kernel: [673777.609547] >>>> > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] >>>> > May 19 05:05:41 [SEQ 255757] dcsx5testslot12 kernel: [673777.609550] >>>> > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] >>>> > May 19 05:05:41 [SEQ 255758] dcsx5testslot12 kernel: [673777.609554] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:05:41 [SEQ 255759] dcsx5testslot12 kernel: [673777.609557] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:41 [SEQ 255760] dcsx5testslot12 kernel: [673777.609561] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:05:41 [SEQ 255761] dcsx5testslot12 kernel: [673777.609564] >>>> > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] >>>> > May 19 05:05:41 [SEQ 255762] dcsx5testslot12 kernel: [673777.609567] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:41 [SEQ 255763] dcsx5testslot12 kernel: [673777.609570] >>>> > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 >>>> > May 19 05:05:41 [SEQ 255764] dcsx5testslot12 kernel: [673777.609571] >>>> > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 >>>> > May 19 05:05:41 [SEQ 255765] dcsx5testslot12 kernel: [673777.609574] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:41 [SEQ 255766] dcsx5testslot12 kernel: [673777.609575] >>>> > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 >>>> > May 19 05:05:41 [SEQ 255767] dcsx5testslot12 kernel: [673777.609578] >>>> > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 >>>> > May 19 05:05:41 [SEQ 255768] dcsx5testslot12 kernel: [673777.609582] >>>> > [<ffffffff81084123>] irq_exit+0x83/0xb0 >>>> > May 19 05:05:41 [SEQ 255769] dcsx5testslot12 kernel: [673777.609584] >>>> > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 >>>> > May 19 05:05:41 [SEQ 255770] dcsx5testslot12 kernel: [673777.609586] >>>> > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f >>>> > May 19 05:05:41 [SEQ 255771] dcsx5testslot12 kernel: [673777.609589] >>>> > <EOI> >>>> > May 19 05:05:41 [SEQ 255772] dcsx5testslot12 kernel: [673777.609589] >>>> > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 >>>> > May 19 05:05:41 [SEQ 255773] dcsx5testslot12 kernel: [673777.609591] >>>> > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 >>>> > May 19 05:05:41 [SEQ 255774] dcsx5testslot12 kernel: [673777.609592] >>>> > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 >>>> > May 19 05:05:41 [SEQ 255775] dcsx5testslot12 kernel: [673777.609594] >>>> > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 >>>> > May 19 05:05:41 [SEQ 255776] dcsx5testslot12 kernel: [673777.609595] >>>> > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 >>>> > May 19 05:05:41 [SEQ 255777] dcsx5testslot12 kernel: [673777.609596] >>>> > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 >>>> > May 19 05:05:41 [SEQ 255778] dcsx5testslot12 kernel: [673777.609599] >>>> > [<ffffffff8103b213>] start_secondary+0x103/0x130 >>>> > May 19 05:05:41 [SEQ 255779] dcsx5testslot12 kernel: [673777.609613] >>>> > Code: c0 c6 06 00 5d c3 0f 1f 84 00 00 00 00 00 55 31 c0 48 85 d2 48 >>>> > 89 e5 74 >>>> > 2c 0f b6 07 0f b6 0e 29 c8 75 22 48 83 ea 01 31 c9 eb 15 <44> 0f b6 >>>> > 4c 0f 01 44 0f b6 44 0e 01 48 83 c1 01 45 29 c1 75 07 >>>> > May 19 05:05:41 [SEQ 255780] dcsx5testslot12 kernel: [673777.998251] >>>> > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c >>>> > e1000e tip >>>> > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev >>>> > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) >>>> > nf_defrag_ipv4 ip6 >>>> > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state >>>> > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd >>>> > pt_timer_info(O >>>> > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit >>>> > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core >>>> > intel_ips >>>> > tpm mdio sunrpc [last unloaded: iTCO_wdt] >>>> > May 19 05:05:41 [SEQ 255781] dcsx5testslot12 kernel: [673778.049966] >>>> > CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O L 4.4.0 #17 >>>> > May 19 05:05:41 [SEQ 255782] dcsx5testslot12 kernel: [673778.057328] >>>> > Hardware name: PT AMC124/Base Board Product Name, BIOS >>>> > LGNAJFIP.PTI.0012.P15 >>>> > 01/15/2014 >>>> > May 19 05:05:41 [SEQ 255783] dcsx5testslot12 kernel: [673778.066505] >>>> > task: ffff880351a751c0 ti: ffff880351a80000 task.ti: ffff880351a80000 >>>> > May 19 05:05:41 [SEQ 255784] dcsx5testslot12 kernel: [673778.074176] >>>> > RIP: 0010:[<ffffffff81083fc7>] [<ffffffff81083fc7>] >>>> > __local_bh_enable_ip+0x2 >>>> > 7/0x90 >>>> > May 19 05:05:41 [SEQ 255785] dcsx5testslot12 kernel: [673778.083184] >>>> > RSP: 0018:ffff88035fc43a98 EFLAGS: 00000202 >>>> > May 19 05:05:41 [SEQ 255786] dcsx5testslot12 kernel: [673778.088616] >>>> > RAX: 0000000000000246 RBX: 0000000000000200 RCX: 0000000000000001 >>>> > May 19 05:05:41 [SEQ 255787] dcsx5testslot12 kernel: [673778.095909] >>>> > RDX: 00000000ab7500c6 RSI: 0000000000000200 RDI: ffffffffa01546a1 >>>> > May 19 05:05:41 [SEQ 255788] dcsx5testslot12 kernel: [673778.103187] >>>> > RBP: ffff88035fc43aa8 R08: 0000000000000001 R09: 0000000000000004 >>>> > May 19 05:05:41 [SEQ 255789] dcsx5testslot12 kernel: [673778.110537] >>>> > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc43bd0 >>>> > May 19 05:05:41 [SEQ 255790] dcsx5testslot12 kernel: [673778.117816] >>>> > R13: 00000000ab7500c6 R14: ffff880350124ec0 R15: ffff88035fc43be4 >>>> > May 19 05:05:41 [SEQ 255791] dcsx5testslot12 kernel: [673778.125094] >>>> > FS: 0000000000000000(0000) GS:ffff88035fc40000(0000) >>>> > knlGS:0000000000000000 >>>> > May 19 05:05:41 [SEQ 255792] dcsx5testslot12 kernel: [673778.133394] >>>> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>>> > May 19 05:05:41 [SEQ 255793] dcsx5testslot12 kernel: [673778.139293] >>>> > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 >>>> > May 19 05:05:41 [SEQ 255794] dcsx5testslot12 kernel: [673778.146549] >>>> > Stack: >>>> > May 19 05:05:41 [SEQ 255795] dcsx5testslot12 kernel: [673778.148733] >>>> > ffff88035fc43bd0 ffff88035fc43b70 ffff88035fc43ab8 ffffffff816de1ba >>>> > May 19 05:05:41 [SEQ 255796] dcsx5testslot12 kernel: [673778.156407] >>>> > ffff88035fc43b38 ffffffffa01546a1 ffff88034a8fad10 0000000000000020 >>>> > May 19 05:05:41 [SEQ 255797] dcsx5testslot12 kernel: [673778.164030] >>>> > ffff88035fc43af8 ffffffff81ce6240 ffff880350124f48 000000012821c002 >>>> > May 19 05:05:41 [SEQ 255798] dcsx5testslot12 kernel: [673778.171626] >>>> > Call Trace: >>>> > May 19 05:05:41 [SEQ 255799] dcsx5testslot12 kernel: [673778.174174] >>>> > <IRQ> >>>> > May 19 05:05:41 [SEQ 255800] dcsx5testslot12 kernel: [673778.176224] >>>> > [<ffffffff816de1ba>] _raw_spin_unlock_bh+0x1a/0x20 >>>> > May 19 05:05:41 [SEQ 255801] dcsx5testslot12 kernel: [673778.182545] >>>> > [<ffffffffa01546a1>] tipc_sk_rcv+0x91/0x490 [tipc] >>>> > May 19 05:05:41 [SEQ 255802] dcsx5testslot12 kernel: [673778.188574] >>>> > [<ffffffff8162ecde>] ? tcp_rcv_established+0x40e/0x760 >>>> > May 19 05:05:41 [SEQ 255803] dcsx5testslot12 kernel: [673778.195080] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:05:41 [SEQ 255804] dcsx5testslot12 kernel: [673778.201586] >>>> > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 >>>> > May 19 05:05:41 [SEQ 255805] dcsx5testslot12 kernel: [673778.207928] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:05:41 [SEQ 255806] dcsx5testslot12 kernel: [673778.214549] >>>> > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] >>>> > May 19 05:05:41 [SEQ 255807] dcsx5testslot12 kernel: [673778.220911] >>>> > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] >>>> > May 19 05:05:41 [SEQ 255808] dcsx5testslot12 kernel: [673778.227002] >>>> > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] >>>> > May 19 05:05:41 [SEQ 255809] dcsx5testslot12 kernel: [673778.233133] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:05:41 [SEQ 255810] dcsx5testslot12 kernel: [673778.239640] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:41 [SEQ 255811] dcsx5testslot12 kernel: [673778.246432] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:05:41 [SEQ 255812] dcsx5testslot12 kernel: [673778.252997] >>>> > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] >>>> > May 19 05:05:41 [SEQ 255813] dcsx5testslot12 kernel: [673778.259399] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:41 [SEQ 255814] dcsx5testslot12 kernel: [673778.266250] >>>> > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 >>>> > May 19 05:05:41 [SEQ 255815] dcsx5testslot12 kernel: [673778.271873] >>>> > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 >>>> > May 19 05:05:41 [SEQ 255816] dcsx5testslot12 kernel: [673778.277046] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:05:41 [SEQ 255817] dcsx5testslot12 kernel: [673778.283878] >>>> > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 >>>> > May 19 05:05:41 [SEQ 255818] dcsx5testslot12 kernel: [673778.289917] >>>> > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 >>>> > May 19 05:05:41 [SEQ 255819] dcsx5testslot12 kernel: [673778.295485] >>>> > [<ffffffff81084123>] irq_exit+0x83/0xb0 >>>> > May 19 05:05:41 [SEQ 255820] dcsx5testslot12 kernel: [673778.300616] >>>> > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 >>>> > May 19 05:05:41 [SEQ 255821] dcsx5testslot12 kernel: [673778.305518] >>>> > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f >>>> > May 19 05:05:41 [SEQ 255822] dcsx5testslot12 kernel: [673778.311278] >>>> > <EOI> >>>> > May 19 05:05:41 [SEQ 255823] dcsx5testslot12 kernel: [673778.313319] >>>> > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 >>>> > May 19 05:05:41 [SEQ 255824] dcsx5testslot12 kernel: [673778.319908] >>>> > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 >>>> > May 19 05:05:41 [SEQ 255825] dcsx5testslot12 kernel: [673778.326317] >>>> > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 >>>> > May 19 05:05:41 [SEQ 255826] dcsx5testslot12 kernel: [673778.332001] >>>> > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 >>>> > May 19 05:05:41 [SEQ 255827] dcsx5testslot12 kernel: [673778.337473] >>>> > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 >>>> > May 19 05:05:41 [SEQ 255828] dcsx5testslot12 kernel: [673778.343278] >>>> > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 >>>> > May 19 05:05:41 [SEQ 255829] dcsx5testslot12 kernel: [673778.349396] >>>> > [<ffffffff8103b213>] start_secondary+0x103/0x130 >>>> > May 19 05:05:41 [SEQ 255830] dcsx5testslot12 kernel: [673778.355259] >>>> > Code: 1f 44 00 00 66 66 66 66 90 55 48 89 e5 53 89 f3 48 83 ec 08 65 >>>> > 8b 05 21 >>>> > 6e f8 7e a9 00 00 0f 00 74 31 80 3d 49 80 c7 00 01 75 31 <83> eb 01 >>>> > f7 db 65 01 1d 05 6e f8 7e 65 8b 05 fe 6d f8 7e a9 00 >>>> > May 19 05:06:01 [SEQ 256552] dcsx5testslot13 /USR/SBIN/CROND[30520]: >>>> > (root) CMD (/opt/cpu_ss7gw/current/scripts/mgmt_apache_watchdog) >>>> > May 19 05:06:01 [SEQ 256570] dcsx5testslot2 /USR/SBIN/CROND[17386]: >>>> > (root) CMD (/opt/cpu_ss7gw/current/scripts/mgmt_apache_watchdog) >>>> > May 19 05:06:09 [SEQ 256861] dcsx5testslot12 kernel: [673805.608813] >>>> > NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0] >>>> > May 19 05:06:09 [SEQ 256862] dcsx5testslot12 kernel: [673805.609784] >>>> > NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0] >>>> > May 19 05:06:09 [SEQ 256863] dcsx5testslot12 kernel: [673805.609807] >>>> > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c >>>> > e1000e tip >>>> > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev >>>> > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) >>>> > nf_defrag_ipv4 ip6 >>>> > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state >>>> > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd >>>> > pt_timer_info(O >>>> > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit >>>> > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core >>>> > intel_ips >>>> > tpm mdio sunrpc [last unloaded: iTCO_wdt] >>>> > May 19 05:06:09 [SEQ 256864] dcsx5testslot12 kernel: [673805.609809] >>>> > CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O L 4.4.0 #17 >>>> > May 19 05:06:09 [SEQ 256865] dcsx5testslot12 kernel: [673805.609810] >>>> > Hardware name: PT AMC124/Base Board Product Name, BIOS >>>> > LGNAJFIP.PTI.0012.P15 >>>> > 01/15/2014 >>>> > May 19 05:06:09 [SEQ 256866] dcsx5testslot12 kernel: [673805.609811] >>>> > task: ffff880351a88000 ti: ffff880351a84000 task.ti: ffff880351a84000 >>>> > May 19 05:06:09 [SEQ 256867] dcsx5testslot12 kernel: [673805.609817] >>>> > RIP: 0010:[<ffffffff81083ff0>] [<ffffffff81083ff0>] >>>> > __local_bh_enable_ip+0x5 >>>> > 0/0x90 >>>> > May 19 05:06:09 [SEQ 256868] dcsx5testslot12 kernel: [673805.609818] >>>> > RSP: 0018:ffff88035fc63a90 EFLAGS: 00000246 >>>> > May 19 05:06:09 [SEQ 256869] dcsx5testslot12 kernel: [673805.609819] >>>> > RAX: 0000000000000500 RBX: 0000000000000200 RCX: 0000000000000001 >>>> > May 19 05:06:09 [SEQ 256870] dcsx5testslot12 kernel: [673805.609819] >>>> > RDX: 00000000b2db9944 RSI: 0000000000000200 RDI: ffffffffa01546a1 >>>> > May 19 05:06:09 [SEQ 256871] dcsx5testslot12 kernel: [673805.609820] >>>> > RBP: ffff88035fc63aa8 R08: 0000000000000001 R09: 0000000000000004 >>>> > May 19 05:06:09 [SEQ 256872] dcsx5testslot12 kernel: [673805.609821] >>>> > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc63bd0 >>>> > May 19 05:06:09 [SEQ 256873] dcsx5testslot12 kernel: [673805.609821] >>>> > R13: 00000000b2db9944 R14: ffff8803502f5640 R15: ffff88035fc63be4 >>>> > May 19 05:06:09 [SEQ 256874] dcsx5testslot12 kernel: [673805.609823] >>>> > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) >>>> > knlGS:0000000000000000 >>>> > May 19 05:06:09 [SEQ 256875] dcsx5testslot12 kernel: [673805.609823] >>>> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>>> > May 19 05:06:09 [SEQ 256876] dcsx5testslot12 kernel: [673805.609824] >>>> > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 >>>> > May 19 05:06:09 [SEQ 256877] dcsx5testslot12 kernel: [673805.609825] >>>> > Stack: >>>> > May 19 05:06:09 [SEQ 256878] dcsx5testslot12 kernel: [673805.609826] >>>> > 0000000000000246 ffff88035fc63bd0 ffff88035fc63b70 ffff88035fc63ab8 >>>> > May 19 05:06:09 [SEQ 256879] dcsx5testslot12 kernel: [673805.609828] >>>> > ffffffff816de1ba ffff88035fc63b38 ffffffffa01546a1 ffff880350b37000 >>>> > May 19 05:06:09 [SEQ 256880] dcsx5testslot12 kernel: [673805.609829] >>>> > ffffffffa03017eb ffff88035fc63b7c ffffffff81ce6240 ffff8803502f56c8 >>>> > May 19 05:06:09 [SEQ 256881] dcsx5testslot12 kernel: [673805.609829] >>>> > Call Trace: >>>> > May 19 05:06:09 [SEQ 256882] dcsx5testslot12 kernel: [673805.609839] >>>> > <IRQ> >>>> > May 19 05:06:09 [SEQ 256883] dcsx5testslot12 kernel: [673805.609839] >>>> > [<ffffffff816de1ba>] _raw_spin_unlock_bh+0x1a/0x20 >>>> > May 19 05:06:09 [SEQ 256884] dcsx5testslot12 kernel: [673805.609846] >>>> > [<ffffffffa01546a1>] tipc_sk_rcv+0x91/0x490 [tipc] >>>> > May 19 05:06:09 [SEQ 256885] dcsx5testslot12 kernel: [673805.609855] >>>> > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] >>>> > May 19 05:06:09 [SEQ 256886] dcsx5testslot12 kernel: [673805.609861] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:06:09 [SEQ 256887] dcsx5testslot12 kernel: [673805.609864] >>>> > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 >>>> > May 19 05:06:09 [SEQ 256888] dcsx5testslot12 kernel: [673805.609867] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:06:09 [SEQ 256889] dcsx5testslot12 kernel: [673805.609871] >>>> > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] >>>> > May 19 05:06:09 [SEQ 256890] dcsx5testslot12 kernel: [673805.609874] >>>> > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] >>>> > May 19 05:06:09 [SEQ 256891] dcsx5testslot12 kernel: [673805.609877] >>>> > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] >>>> > May 19 05:06:09 [SEQ 256892] dcsx5testslot12 kernel: [673805.609881] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:06:09 [SEQ 256893] dcsx5testslot12 kernel: [673805.609884] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:06:09 [SEQ 256894] dcsx5testslot12 kernel: [673805.609887] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:06:09 [SEQ 256895] dcsx5testslot12 kernel: [673805.609891] >>>> > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] >>>> > May 19 05:06:09 [SEQ 256896] dcsx5testslot12 kernel: [673805.609894] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:06:09 [SEQ 256897] dcsx5testslot12 kernel: [673805.609896] >>>> > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 >>>> > May 19 05:06:09 [SEQ 256898] dcsx5testslot12 kernel: [673805.609897] >>>> > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 >>>> > May 19 05:06:09 [SEQ 256899] dcsx5testslot12 kernel: [673805.609901] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:06:09 [SEQ 256900] dcsx5testslot12 kernel: [673805.609902] >>>> > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 >>>> > May 19 05:06:09 [SEQ 256901] dcsx5testslot12 kernel: [673805.609904] >>>> > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 >>>> > May 19 05:06:09 [SEQ 256902] dcsx5testslot12 kernel: [673805.609905] >>>> > [<ffffffff81084123>] irq_exit+0x83/0xb0 >>>> > May 19 05:06:09 [SEQ 256903] dcsx5testslot12 kernel: [673805.609908] >>>> > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 >>>> > May 19 05:06:09 [SEQ 256904] dcsx5testslot12 kernel: [673805.609909] >>>> > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f >>>> > May 19 05:06:09 [SEQ 256905] dcsx5testslot12 kernel: [673805.609912] >>>> > <EOI> >>>> > May 19 05:06:09 [SEQ 256906] dcsx5testslot12 kernel: [673805.609912] >>>> > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 >>>> > May 19 05:06:09 [SEQ 256907] dcsx5testslot12 kernel: [673805.609914] >>>> > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 >>>> > May 19 05:06:09 [SEQ 256908] dcsx5testslot12 kernel: [673805.609915] >>>> > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 >>>> > May 19 05:06:09 [SEQ 256909] dcsx5testslot12 kernel: [673805.609917] >>>> > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 >>>> > May 19 05:06:09 [SEQ 256910] dcsx5testslot12 kernel: [673805.609918] >>>> > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 >>>> > May 19 05:06:09 [SEQ 256911] dcsx5testslot12 kernel: [673805.609919] >>>> > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 >>>> > May 19 05:06:09 [SEQ 256912] dcsx5testslot12 kernel: [673805.609922] >>>> > [<ffffffff8103b213>] start_secondary+0x103/0x130 >>>> > May 19 05:06:09 [SEQ 256913] dcsx5testslot12 kernel: [673805.609937] >>>> > Code: 75 31 83 eb 01 f7 db 65 01 1d 05 6e f8 7e 65 8b 05 fe 6d f8 7e >>>> > a9 00 ff >>>> > 1f 00 74 31 65 ff 0d f0 6d f8 7e 48 83 c4 08 5b 5d c3 9c <58> f6 c4 >>>> > 02 75 d1 eb c6 be 96 00 00 00 48 c7 c7 82 27 a3 81 e8 >>>> > May 19 05:06:09 [SEQ 256914] dcsx5testslot12 kernel: [673806.000394] >>>> > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c >>>> > e1000e tip >>>> > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev >>>> > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) >>>> > nf_defrag_ipv4 ip6 >>>> > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state >>>> > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd >>>> > pt_timer_info(O >>>> > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit >>>> > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core >>>> > intel_ips >>>> > tpm mdio sunrpc [last unloaded: iTCO_wdt] >>>> > May 19 05:06:09 [SEQ 256915] dcsx5testslot12 kernel: [673806.052443] >>>> > CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O L 4.4.0 #17 >>>> > May 19 05:06:09 [SEQ 256916] dcsx5testslot12 kernel: [673806.059943] >>>> > Hardware name: PT AMC124/Base Board Product Name, BIOS >>>> > LGNAJFIP.PTI.0012.P15 >>>> > 01/15/2014 >>>> > May 19 05:06:09 [SEQ 256917] dcsx5testslot12 kernel: [673806.069246] >>>> > task: ffff880351a751c0 ti: ffff880351a80000 task.ti: ffff880351a80000 >>>> > May 19 05:06:09 [SEQ 256918] dcsx5testslot12 kernel: [673806.076929] >>>> > RIP: 0010:[<ffffffffa01546a5>] [<ffffffffa01546a5>] >>>> > tipc_sk_rcv+0x95/0x490 [ >>>> > tipc] >>>> > May 19 05:06:09 [SEQ 256919] dcsx5testslot12 kernel: [673806.085716] >>>> > RSP: 0018:ffff88035fc43ac8 EFLAGS: 00000296 >>>> > May 19 05:06:09 [SEQ 256920] dcsx5testslot12 kernel: [673806.091252] >>>> > RAX: 0000000000000301 RBX: ffff88035fc43b70 RCX: 0000000000000001 >>>> > May 19 05:06:09 [SEQ 256921] dcsx5testslot12 kernel: [673806.098623] >>>> > RDX: 00000000ab7500c6 RSI: 0000000000000200 RDI: ffffffff81ce6240 >>>> > May 19 05:06:09 [SEQ 256922] dcsx5testslot12 kernel: [673806.106021] >>>> > RBP: ffff88035fc43b38 R08: 0000000000000001 R09: 0000000000000004 >>>> > May 19 05:06:09 [SEQ 256923] dcsx5testslot12 kernel: [673806.113331] >>>> > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc43bd0 >>>> > May 19 05:06:09 [SEQ 256924] dcsx5testslot12 kernel: [673806.120634] >>>> > R13: 00000000ab7500c6 R14: ffff880350124ec0 R15: ffff88035fc43be4 >>>> > May 19 05:06:09 [SEQ 256925] dcsx5testslot12 kernel: [673806.127940] >>>> > FS: 0000000000000000(0000) GS:ffff88035fc40000(0000) >>>> > knlGS:0000000000000000 >>>> > May 19 05:06:09 [SEQ 256926] dcsx5testslot12 kernel: [673806.136201] >>>> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>>> > May 19 05:06:09 [SEQ 256927] dcsx5testslot12 kernel: [673806.142110] >>>> > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 >>>> > May 19 05:06:09 [SEQ 256928] dcsx5testslot12 kernel: [673806.149540] >>>> > Stack: >>>> > May 19 05:06:09 [SEQ 256929] dcsx5testslot12 kernel: [673806.151715] >>>> > ffff88034a8fad10 0000000000000020 ffff88035fc43af8 ffffffff81ce6240 >>>> > May 19 05:06:09 [SEQ 256930] dcsx5testslot12 kernel: [673806.159346] >>>> > ffff880350124f48 000000012821c002 ffff88035fc43b58 ffffffff8162ecde >>>> > May 19 05:06:09 [SEQ 256931] dcsx5testslot12 kernel: [673806.167047] >>>> > ffff88035fc43b48 ffff88035fc43b70 0000000000000000 000000000100100c >>>> > May 19 05:06:09 [SEQ 256932] dcsx5testslot12 kernel: [673806.174661] >>>> > Call Trace: >>>> > May 19 05:06:09 [SEQ 256933] dcsx5testslot12 kernel: [673806.177199] >>>> > <IRQ> >>>> > May 19 05:06:09 [SEQ 256934] dcsx5testslot12 kernel: [673806.179238] >>>> > [<ffffffff8162ecde>] ? tcp_rcv_established+0x40e/0x760 >>>> > May 19 05:06:09 [SEQ 256935] dcsx5testslot12 kernel: [673806.185854] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:06:09 [SEQ 256936] dcsx5testslot12 kernel: [673806.192362] >>>> > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 >>>> > :6400 >>>> > May 19 05:03:49 [SEQ 250358] dcsx5testslot12 kernel: [673665.608194] >>>> > [<ffffffff816de332>] ? _raw_spin_trylock_bh+0x32/0x50 >>>> > May 19 05:03:49 [SEQ 250359] dcsx5testslot12 kernel: [673665.608198] >>>> > [<ffffffffa01546ad>] tipc_sk_rcv+0x9d/0x490 [tipc] >>>> > May 19 05:03:49 [SEQ 250360] dcsx5testslot12 kernel: [673665.608206] >>>> > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] >>>> > May 19 05:03:49 [SEQ 250361] dcsx5testslot12 kernel: [673665.608210] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:03:49 [SEQ 250362] dcsx5testslot12 kernel: [673665.608213] >>>> > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 >>>> > May 19 05:03:49 [SEQ 250363] dcsx5testslot12 kernel: [673665.608217] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:03:49 [SEQ 250364] dcsx5testslot12 kernel: [673665.608220] >>>> > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] >>>> > May 19 05:03:49 [SEQ 250365] dcsx5testslot12 kernel: [673665.608223] >>>> > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] >>>> > May 19 05:03:49 [SEQ 250366] dcsx5testslot12 kernel: [673665.608226] >>>> > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] >>>> > May 19 05:03:49 [SEQ 250367] dcsx5testslot12 kernel: [673665.608233] >>>> > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] >>>> > May 19 05:03:49 [SEQ 250368] dcsx5testslot12 kernel: [673665.608236] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:03:49 [SEQ 250369] dcsx5testslot12 kernel: [673665.608239] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:03:49 [SEQ 250370] dcsx5testslot12 kernel: [673665.608243] >>>> > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] >>>> > May 19 05:03:49 [SEQ 250371] dcsx5testslot12 kernel: [673665.608246] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:03:49 [SEQ 250372] dcsx5testslot12 kernel: [673665.608249] >>>> > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 >>>> > May 19 05:03:49 [SEQ 250373] dcsx5testslot12 kernel: [673665.608251] >>>> > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 >>>> > May 19 05:03:49 [SEQ 250374] dcsx5testslot12 kernel: [673665.608254] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:03:49 [SEQ 250375] dcsx5testslot12 kernel: [673665.608256] >>>> > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 >>>> > May 19 05:03:49 [SEQ 250376] dcsx5testslot12 kernel: [673665.608259] >>>> > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 >>>> > May 19 05:03:49 [SEQ 250377] dcsx5testslot12 kernel: [673665.608260] >>>> > [<ffffffff81084123>] irq_exit+0x83/0xb0 >>>> > May 19 05:03:49 [SEQ 250378] dcsx5testslot12 kernel: [673665.608263] >>>> > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 >>>> > May 19 05:03:49 [SEQ 250379] dcsx5testslot12 kernel: [673665.608265] >>>> > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f >>>> > May 19 05:03:49 [SEQ 250380] dcsx5testslot12 kernel: [673665.608268] >>>> > <EOI> >>>> > May 19 05:03:49 [SEQ 250381] dcsx5testslot12 kernel: [673665.608268] >>>> > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 >>>> > May 19 05:03:49 [SEQ 250382] dcsx5testslot12 kernel: [673665.608269] >>>> > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 >>>> > May 19 05:03:49 [SEQ 250383] dcsx5testslot12 kernel: [673665.608270] >>>> > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 >>>> > May 19 05:03:49 [SEQ 250384] dcsx5testslot12 kernel: [673665.608273] >>>> > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 >>>> > May 19 05:03:49 [SEQ 250385] dcsx5testslot12 kernel: [673665.608274] >>>> > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 >>>> > May 19 05:03:49 [SEQ 250386] dcsx5testslot12 kernel: [673665.608275] >>>> > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 >>>> > May 19 05:03:49 [SEQ 250387] dcsx5testslot12 kernel: [673665.608278] >>>> > [<ffffffff8103b213>] start_secondary+0x103/0x130 >>>> > May 19 05:03:49 [SEQ 250388] dcsx5testslot12 kernel: [673665.608292] >>>> > Code: 41 31 c8 c1 c2 0e 41 29 d0 44 89 c2 44 31 c0 c1 c2 0b 29 d0 89 >>>> > c2 31 c1 >>>> > c1 c2 19 29 d1 89 ca 41 31 c8 c1 c2 10 41 29 d0 44 89 c2 <44> 31 c0 >>>> > c1 c2 04 29 d0 31 c1 c1 c0 0e 29 c1 41 31 c8 c1 c1 18 >>>> > May 19 05:03:49 [SEQ 250389] dcsx5testslot12 kernel: [673666.007702] >>>> > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c >>>> > e1000e tip >>>> > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev >>>> > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) >>>> > nf_defrag_ipv4 ip6 >>>> > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state >>>> > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd >>>> > pt_timer_info(O >>>> > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit >>>> > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core >>>> > intel_ips >>>> > tpm mdio sunrpc [last unloaded: iTCO_wdt] >>>> > May 19 05:03:49 [SEQ 250390] dcsx5testslot12 kernel: [673666.060133] >>>> > CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O L 4.4.0 #17 >>>> > May 19 05:03:49 [SEQ 250391] dcsx5testslot12 kernel: [673666.067566] >>>> > Hardware name: PT AMC124/Base Board Product Name, BIOS >>>> > LGNAJFIP.PTI.0012.P15 >>>> > 01/15/2014 >>>> > May 19 05:03:49 [SEQ 250392] dcsx5testslot12 kernel: [673666.076908] >>>> > task: ffff880351a751c0 ti: ffff880351a80000 task.ti: ffff880351a80000 >>>> > May 19 05:03:49 [SEQ 250393] dcsx5testslot12 kernel: [673666.084770] >>>> > RIP: 0010:[<ffffffff816de29f>] [<ffffffff816de29f>] >>>> > _raw_spin_lock_bh+0x1f/0 >>>> > x40 >>>> > May 19 05:03:49 [SEQ 250394] dcsx5testslot12 kernel: [673666.093585] >>>> > RSP: 0018:ffff88035fc43ab8 EFLAGS: 00000246 >>>> > May 19 05:03:49 [SEQ 250395] dcsx5testslot12 kernel: [673666.099084] >>>> > RAX: 0000000000000000 RBX: ffff88035fc43b70 RCX: 0000000000000001 >>>> > May 19 05:03:49 [SEQ 250396] dcsx5testslot12 kernel: [673666.106567] >>>> > RDX: 0000000000000001 RSI: 0000000000000200 RDI: ffff88035fc43be4 >>>> > /1.1.12 >>>> > May 19 05:04:45 [SEQ 252557] dcsx5testslot12 kernel: [673722.241945] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:04:45 [SEQ 252558] dcsx5testslot12 kernel: [673722.248734] >>>> > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] >>>> > May 19 05:04:45 [SEQ 252559] dcsx5testslot12 kernel: [673722.255656] >>>> > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] >>>> > May 19 05:04:45 [SEQ 252560] dcsx5testslot12 kernel: [673722.262126] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:04:45 [SEQ 252561] dcsx5testslot12 kernel: [673722.269038] >>>> > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 >>>> > May 19 05:04:45 [SEQ 252562] dcsx5testslot12 kernel: [673722.274789] >>>> > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 >>>> > May 19 05:04:45 [SEQ 252563] dcsx5testslot12 kernel: [673722.280087] >>>> > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] >>>> > May 19 05:04:45 [SEQ 252564] dcsx5testslot12 kernel: [673722.286846] >>>> > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 >>>> > May 19 05:04:45 [SEQ 252565] dcsx5testslot12 kernel: [673722.292978] >>>> > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 >>>> > May 19 05:04:45 [SEQ 252566] dcsx5testslot12 kernel: [673722.298687] >>>> > [<ffffffff81084123>] irq_exit+0x83/0xb0 >>>> > May 19 05:04:45 [SEQ 252567] dcsx5testslot12 kernel: [673722.303788] >>>> > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 >>>> > May 19 05:04:45 [SEQ 252568] dcsx5testslot12 kernel: [673722.308718] >>>> > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f >>>> > May 19 05:04:45 [SEQ 252569] dcsx5testslot12 kernel: [673722.314666] >>>> > <EOI> >>>> > May 19 05:04:45 [SEQ 252570] dcsx5testslot12 kernel: [673722.316729] >>>> > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 >>>> > May 19 05:04:45 [SEQ 252571] dcsx5testslot12 kernel: [673722.323330] >>>> > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 >>>> > May 19 05:04:45 [SEQ 252572] dcsx5testslot12 kernel: [673722.329738] >>>> > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 >>>> > May 19 05:04:45 [SEQ 252573] dcsx5testslot12 kernel: [673722.335346] >>>> > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 >>>> > May 19 05:04:45 [SEQ 252574] dcsx5testslot12 kernel: [673722.340955] >>>> > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 >>>> > May 19 05:04:45 [SEQ 252575] dcsx5testslot12 kernel: [673722.346777] >>>> > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 >>>> > May 19 05:04:45 [SEQ 252576] dcsx5testslot12 kernel: [673722.352887] >>>> > [<ffffffff8103b213>] start_secondary+0x103/0x130 >>>> > May 19 05:04:45 [SEQ 252577] dcsx5testslot12 kernel: [673722.358765] >>>> > Code: 44 89 c0 5d c3 44 03 47 08 03 4f 04 89 ca 03 07 41 31 c8 c1 c2 >>>> > 0e 41 29 >>>> > d0 44 89 c2 44 31 c0 c1 c2 0b 29 d0 89 c2 31 c1 c1 c2 19 <29> d1 89 >>>> > ca 41 31 c8 c1 c2 10 41 29 d0 44 89 c2 44 31 c0 c1 c2 >>>> > May 19 05:04:57 [SEQ 253019] slot6_0 kernel: TIPC: Resetting link >>>> > <1.1.6:eth1-1.1.12:p19p2>, changeover initiated by peer >>>> > May 19 05:04:57 [SEQ 253020] slot7_0 kernel: TIPC: Resetting link >>>> > <1.1.7:eth1-1.1.12:p19p2>, changeover initiated by peer >>>> > May 19 05:04:57 [SEQ 253021] slot8_0 kernel: TIPC: Resetting link >>>> > <1.1.8:eth1-1.1.12:p19p2>, changeover initiated by peer >>>> > May 19 05:04:57 [SEQ 253022] slot9_0 kernel: TIPC: Resetting link >>>> > <1.1.9:eth1-1.1.12:p19p2>, changeover initiated by peer >>>> > May 19 05:04:57 [SEQ 253023] slot6_0 kernel: TIPC: Lost link >>>> > <1.1.6:eth1-1.1.12:p19p2> on network plane B >>>> > May 19 05:04:57 [SEQ 253024] slot7_0 kernel: TIPC: Lost link >>>> > <1.1.7:eth1-1.1.12:p19p2> on network plane B >>>> > May 19 05:04:57 [SEQ 253025] slot8_0 kernel: TIPC: Lost link >>>> > <1.1.8:eth1-1.1.12:p19p2> on network plane B >>>> > May 19 05:04:57 [SEQ 253026] slot9_0 kernel: TIPC: Lost link >>>> > <1.1.9:eth1-1.1.12:p19p2> on network plane B >>>> > May 19 05:05:01 [SEQ 253150] dcsx5testslot11 /USR/SBIN/CROND[4582]: >>>> > (root) CMD (/sbin/corecleanup) >>>> > May 19 05:05:01 [SEQ 253151] slot9_0 /USR/SBIN/CROND[23109]: (root) >>>> > CMD (/sbin/corecleanup) >>>> > May 19 05:05:01 [SEQ 253179] dcsx5testslot4 /USR/SBIN/CROND[10667]: >>>> > (root) CMD (/sbin/corecleanup) >>>> > May 19 05:05:01 [SEQ 253193] dcsx5testslot3 /USR/SBIN/CROND[10640]: >>>> > (root) CMD (/sbin/corecleanup) >>>> > May 19 05:05:01 [SEQ 253194] slot8_0 /USR/SBIN/CROND[19631]: (root) >>>> > CMD (/sbin/corecleanup) >>>> > >>>> > >>>> > >>>> > On Tue, May 24, 2016 at 7:29 PM, Jon Maloy <ma...@donjonn.com> wrote: >>>> > > >>>> > > >>>> > > On 05/24/2016 12:16 PM, GUNA wrote: >>>> > >> >>>> > >> I suspect there could be glitch on switch may cause lost the probe or >>>> > >> abort message. However, even if the messages are lost for what ever >>>> > >> reason, is not TIPC stack should handle the graceful shutdown of the >>>> > >> TIPC connection by releasing all the resources instead of panic or >>>> > >> dead itself ? >>>> > > >>>> > > I don't think this is about lost packets or shutdown at all, as far as >>>> > > I can >>>> > > see form the dump.This is a node local socket-to-socket call that >>>> > > bypasses >>>> > > any interfaces and switches. >>>> > >> >>>> > >> >>>> > >> Does lock_sock/release_sock use in tipc_sk_timeout() fix the issue ? >>>> > > >>>> > > >>>> > > No, it won't. At first I thought this obvious, as can be deferred from >>>> > > my >>>> > > first comment: >>>> > > tipc_sk_timeout() grabs sk.slock, sends a message to the receiver >>>> > > socket, >>>> > > which for some reason chooses to return it to sender. In tipc_sk_rcv >>>> > > we try >>>> > > to grab sk.slock again, and hence we end up in a deadlock. >>>> > > >>>> > > Unfortunately this theory is not correct. >>>> > > tipc_sk_timeout() actually releases sk.slock before it sends out the >>>> > > buffer, >>>> > > so there is no way a deadlock can happen this way. Besides, given >>>> > > that we >>>> > > don't actually spin on a spin_trylock_bh() call, I believe we are on >>>> > > the >>>> > > wrong track when suspecting a slock deadlock. >>>> > > >>>> > > A message produced by tipc_sk_timeout can only be a >>>> > CONN_MANAGER/CONN_PROBE >>>> > > message, but I don't see filter_rcv() calling tipc_sk_proto_rcv() >>>> > > anywhere. >>>> > > The explanation seems to be that tipc_sk_proto_rcv() is inlined. So, >>>> > > we can >>>> > > probably assume that the message causing the deadlock is a regular >>>> > > CONN_MANAGER/CONN_PROBE_REPLY. >>>> > > >>>> > > So, my theory right now is that the reply message for some reason >>>> > > causes >>>> > > tipc_sk_rv() to enter a loop where 'inputq' never is emptied. >>>> > > >>>> > > This is as far as I get in my analysis right now. Does anybody else >>>> > > have any >>>> > > good ideas? >>>> > > >>>> > > ///jon >>>> > > >>>> > > >>>> > > >>>> > >> >>>> > >> Thanks, >>>> > >> Guna >>>> > >> >>>> > >> >>>> > >> ------------------------------------------------------------------------------ >>>> > >> Mobile security can be enabling, not merely restricting. Employees who >>>> > >> bring their own devices (BYOD) to work are irked by the imposition of >>>> > >> MDM >>>> > >> restrictions. Mobile Device Manager Plus allows you to control only >>>> > >> the >>>> > >> apps on BYO-devices by containerizing them, leaving personal data >>>> > >> untouched! >>>> > >> https://ad.doubleclick.net/ddm/clk/304595813;131938128;j >>>> > >> _______________________________________________ >>>> > >> tipc-discussion mailing list >>>> > >> tipc-discussion@lists.sourceforge.net >>>> > >> https://lists.sourceforge.net/lists/listinfo/tipc-discussion >>>> > > >>>> > > >>>> > >>>> > ------------------------------------------------------------------------------ >>>> > What NetFlow Analyzer can do for you? Monitors network bandwidth and >>>> traffic >>>> > patterns at an interface-level. Reveals which users, apps, and protocols >>>> > are >>>> > consuming the most bandwidth. Provides multi-vendor support for NetFlow, >>>> > J-Flow, sFlow and other flows. Make informed decisions using capacity >>>> > planning reports. >>>> > https://ad.doubleclick.net/ddm/clk/305295220;132659582;e >>>> > _______________________________________________ >>>> > tipc-discussion mailing list >>>> > tipc-discussion@lists.sourceforge.net >>>> > https://lists.sourceforge.net/lists/listinfo/tipc-discussion ------------------------------------------------------------------------------ What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e _______________________________________________ tipc-discussion mailing list tipc-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tipc-discussion
