Quoting Peter Zijlstra <[EMAIL PROTECTED]>: Subject: Re: lockdep question (was Re: IPoIB caused a kernel: BUG: softlockup detected on CPU#0!)
> On Sun, 2007-03-11 at 15:50 +0200, Michael S. Tsirkin wrote: > > > Quoting Roland Dreier <[EMAIL PROTECTED]>: > > > Subject: Re: IPoIB caused a kernel: BUG: soft lockup detected on CPU#0! > > > > > > >Feb 27 17:47:52 sw169 kernel: [<ffffffff8053aaf1>] > > > >_spin_lock_irqsave+0x15/0x24 > > > >Feb 27 17:47:52 sw169 kernel: [<ffffffff88067a23>] > > > >:ib_ipoib:ipoib_neigh_destructor+0xc2/0x139 > > > > > > It looks like this is deadlocking trying to take priv->lock in > > > ipoib_neigh_destructor(). > > > One idea I just had would be to build a kernel with CONFIG_PROVE_LOCKING > > > turned on, and then rerun this test. There's a good chance that this > > > would > > > diagnose the deadlock. (I don't have good access to my test machines > > > right now, or > > > else I would do it myself) > > > > OK, I did that. But I get > > [13440.761857] INFO: trying to register non-static key. > > [13440.766903] the code is fine but needs lockdep annotation. > > [13440.772455] turning off the locking correctness validator. > > and I am not sure what triggers this, or how to fix it to have the > > validator actually do its job. > > It usually indicates a spinlock is not properly initialized. Like > __SPIN_LOCK_UNLOCKED() used in a non-static context, use > spin_lock_init() in these cases. > > However looking at the code, ipoib_neight_destructor only uses > &priv->lock, and that seems to get properly initialized in ipoib_setup() > using spin_lock_init(). > > So either there are other sites that instanciate those objects and > forget about the lock init, or the object is corrupted (use after free?) OK, thanks for the hint. So I added this: diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index f9dbc6f..2eea467 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -821,8 +821,15 @@ static void ipoib_neigh_destructor(struct neighbour *n) unsigned long flags; struct ipoib_ah *ah = NULL; + if (n->dev->type != ARPHRD_INFINIBAND) { + printk(KERN_ERR "ipoib_neigh_destructor lock %p wrong type %d !!!!!!!!!!\n", + &priv->lock, n->dev->type); + BUG_ON(n->dev->type != ARPHRD_INFINIBAND); + return; + } + ipoib_dbg(priv, "neigh_destructor for %06x " IPOIB_GID_FMT "\n", IPOIB_QPN(n->ha), IPOIB_GID_RAW_ARG(n->ha + 4)); And sure enough it triggers: [ 858.503010] ipoib_neigh_destructor lock c0687880 wrong type 772 !!!!!!!!!! [ 858.510036] ------------[ cut here ]------------ [ 858.514723] kernel BUG at drivers/infiniband/ulp/ipoib/ipoib_main.c:827! [ 858.521486] invalid opcode: 0000 [#1] [ 858.525212] SMP [ 858.527173] Modules linked in: rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_sa ib_uverbs ibv [ 858.538736] CPU: 0 [ 858.538737] EIP: 0060:[<f899bfa5>] Not tainted VLI [ 858.538738] EFLAGS: 00010206 (2.6.21-rc3-i686-dbg #4) [ 858.551755] EIP is at ipoib_neigh_destructor+0x40/0x178 [ib_ipoib] [ 858.557996] eax: c0687300 ebx: f240e880 ecx: c0223114 edx: c064f280 [ 858.564851] esi: f240e880 edi: f240e880 ebp: c0687880 esp: c06c7e9c [ 858.571702] ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068 [ 858.577602] Process swapper (pid: 0, ti=c06c6000 task=c064f280 task.ti=c06c6000) [ 858.584883] Stack: f89a37be c0687880 00000304 c022af6e c064f280 00000000 00000000 0000 [ 858.593573] 00000000 c06a2554 00000000 c064f280 00000001 00000000 c064f280 0000 [ 858.602259] c0860be0 c2a1fba0 00000246 c06a2554 f240e880 00000000 f240e880 c04a [ 858.610946] Call Trace: [ 858.613723] [<c022af6e>] run_timer_softirq+0x37/0x16b [ 858.618959] [<c04a1c0f>] dst_run_gc+0x0/0x118 [ 858.623498] [<c04a3eab>] neigh_destroy+0xbe/0x104 [ 858.628382] [<c04a1bb1>] dst_destroy+0x4d/0xab [ 858.632998] [<c04a1c64>] dst_run_gc+0x55/0x118 [ 858.637620] [<c022b03f>] run_timer_softirq+0x108/0x16b [ 858.642934] [<c0227634>] __do_softirq+0x5a/0xd5 [ 858.647648] [<c023b435>] trace_hardirqs_on+0x106/0x141 [ 858.652970] [<c0227643>] __do_softirq+0x69/0xd5 [ 858.657677] [<c02276e6>] do_softirq+0x37/0x4d [ 858.662210] [<c02167b0>] smp_apic_timer_interrupt+0x6b/0x77 [ 858.667965] [<c02029ef>] default_idle+0x3b/0x54 [ 858.672681] [<c02029ef>] default_idle+0x3b/0x54 [ 858.677391] [<c0204c33>] apic_timer_interrupt+0x33/0x38 [ 858.682796] [<c02029ef>] default_idle+0x3b/0x54 [ 858.687505] [<c02029f1>] default_idle+0x3d/0x54 [ 858.692211] [<c0202aaa>] cpu_idle+0xa2/0xbb [ 858.696569] [<c06cd7c3>] start_kernel+0x40b/0x413 [ 858.701453] [<c06cd1b3>] unknown_bootoption+0x0/0x205 [ 858.706678] ======================= [ 858.710321] Code: 66 83 f8 20 74 29 0f b7 c0 89 44 24 08 89 6c 24 04 c7 04 24 be 37 9a [ 858.730997] EIP: [<f899bfa5>] ipoib_neigh_destructor+0x40/0x178 [ib_ipoib] SS:ESP 0068c [ 858.740271] Kernel panic - not syncing: Fatal exception in interrupt Either something is corrupting neighbour dev pointer, or somehow a neighbour not related to infiniband is assigned ipoib_neigh_destructor. -- MST - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/