On 11/27/2012 03:20 AM, Dave Pigott wrote:
Hi all,
One overnight failure:
----------------
panda-es04
----------------
http://validation.linaro.org/lava-server/scheduler/job/40109
Failed to get into test image. Some sort of corruption. Went onto the
board and booted test image again. It came up, and we got a command
prompt, but it was still pretty flakey: https://pastebin.linaro.org/1129/
Although a reboot of the test image *might* have fixed this, it looks
like either the test image was dodgy, or the board is genuinely failing.
I've put back online to test.
This looks like a kernel issue we've seen before with IRQ unlock issue.
Maybe this can be addressed with a newer test image?
[ 16.606079] [ BUG: bad unlock balance detected! ]
[ 16.606079] -------------------------------------
[ 16.606109] syslink_daemon/1151 is trying to release lock
(gatemp_module->gate_mutex) at:
[ 16.606109] [<c0743b40>] mutex_unlock+0x18/0x1c
[ 16.606109] but there are no more locks to release!
[ 16.606109]
[ 16.606170] other info that might help us debug this:
[ 16.606170] no locks held by syslink_daemon/1151.
[ 16.606170]
[ 16.606170] stack backtrace:
[ 16.606201] [<c001b5f0>] (unwind_backtrace+0x0/0xec) from
[<c0720fc8>] (dump_stack+0x20/0x24)
[ 16.606201] [<c0720fc8>] (dump_stack+0x20/0x24) from [<c072452c>]
(print_unlock_inbalance_bug.part.19+0x84/0xac)
[ 16.606201] [<c072452c>]
(print_unlock_inbalance_bug.part.19+0x84/0xac) from [<c0094840>]
(print_unlock_inbalance_bug+0x44/0x50)
[ 16.606201] [<c0094840>] (print_unlock_inbalance_bug+0x44/0x50) from
[<c0097388>] (__lock_release+0x9c/0xd8)
[ 16.606201] [<c0097388>] (__lock_release+0x9c/0xd8) from [<c00974a0>]
(lock_release+0xdc/0x100)
[ 16.606201] [<c00974a0>] (lock_release+0xdc/0x100) from [<c0743a94>]
(__mutex_unlock_slowpath+0xf8/0x18c)
[ 16.606384] [<c0743a94>] (__mutex_unlock_slowpath+0xf8/0x18c) from
[<c0743b40>] (mutex_unlock+0x18/0x1c)
[ 16.606414] [<c0743b40>] (mutex_unlock+0x18/0x1c) from [<c0537f0c>]
(gatehwspinlock_leave+0xc0/0xe0)
[ 16.606414] [<c0537f0c>] (gatehwspinlock_leave+0xc0/0xe0) from
[<c0538ea8>] (gatemp_leave+0x28/0x2c)
[ 16.606414] [<c0538ea8>] (gatemp_leave+0x28/0x2c) from [<c053b21c>]
(heapmemmp_alloc+0x1b8/0x200)
[ 16.606414] [<c053b21c>] (heapmemmp_alloc+0x1b8/0x200) from
[<c053c6e4>] (heapmemmp_ioctl+0x13c/0x4f0)
[ 16.606506] [<c053c6e4>] (heapmemmp_ioctl+0x13c/0x4f0) from
[<c0544dcc>] (ipc_ioc_router+0x164/0x194)
[ 16.606506] [<c0544dcc>] (ipc_ioc_router+0x164/0x194) from
[<c0544e64>] (ipc_ioctl+0x68/0x74)
[ 16.606506] [<c0544e64>] (ipc_ioctl+0x68/0x74) from [<c0127af4>]
(do_vfs_ioctl+0x2a8/0x2e4)
[ 16.606597] [<c0127af4>] (do_vfs_ioctl+0x2a8/0x2e4) from [<c0127b90>]
(sys_ioctl+0x60/0x84)
[ 16.606628] [<c0127b90>] (sys_ioctl+0x60/0x84) from [<c00130e0>]
(ret_fast_syscall+0x0/0x3c)
[ 16.610687] gatehwspinlock_enter failed! status = 0xffffffed
[ 16.612670] Registered user-space process for DEV_SYS_ERROR event in
AppM3
[ 16.612670] Registered user-space process for DEV_WATCHDOG_ERROR
event in AppM3
[ 16.612670] Registered user-space process for DEV_SYS_ERROR event in
SysM3
[ 16.612670] Registered user-space process for DEV_WATCHDOG_ERROR
event in SysM3
[ 17.077484] Unable to handle kernel NULL pointer dereference at
virtual address 00000004
[ 17.080047] pgd = ea0e0000
[ 17.095672] [00000004] *pgd=00000000
[ 17.095672] Internal error: Oops: 805 [#1] PREEMPT SMP
[ 17.119110] Modules linked in: wl12xx_sdio wl12xx
[ 17.139221] CPU: 0 Tainted: G W (3.1.1-19-linaro-lt-omap
#19~lt~ci~20120216183525+1329716076~4f41e9c2-Ubuntu)
[ 17.158416] PC is at skb_dequeue+0x5c/0x74
[ 17.170257] LR is at _raw_spin_lock_irqsave+0x6c/0x78
[ 17.183166] pc : [<c0599324>] lr : [<c07449dc>] psr: a0000193
[ 17.183166] sp : ebb35cc8 ip : ebb35c98 fp : ebb35ce4
[ 17.197235] r10: 00000000 r9 : 0000076b r8 : ebb35df0
[ 17.223693] r7 : ee5b1ae0 r6 : ee5b1b84 r5 : ee5b1b78 r4 : ebca0200
[ 17.238250] r3 : 00000000 r2 : 00000000 r1 : 60000113 r0 : 00000000
[ 17.251831] Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM
Segment user
[ 17.267364] Control: 10c5387d Table: aa0e004a DAC: 00000015
[ 17.275360] Process upstart-socket- (pid: 1407, stack limit = 0xebb342f8)
[ 17.297088] Stack: (0xebb35cc8 to 0xebb36000)
[ 17.300384] 5cc0: eea901e0 ebca0200 ebb35f64
00000095 ebb35d6c ebb35ce8
[ 17.326080] 5ce0: c0627764 c05992d4 eea90690 501520aa 00000009
ee5b1ddc 00000001 ebb34000
[ 17.326080] 5d00: ed2d10e0 ee5b1d7c ee5b1b78 00000001 00000000
00000000 00000000 ee08e4e0
[ 17.359191] 5d20: ee192ac0 00000000 00000001 00000000 00000000
00000000 60000193 c10b76bc
[ 17.361938] 5d40: ebb35d5c ed2d10e0 40000040 ebb35f64 00000800
00000000 00000000 ebb35e64
[ 17.384552] 5d60: ebb35e54 ebb35d70 c059086c c062764c 40000040
ee5ad854 00000000 00000000
[ 17.409057] 5d80: 00000000 00000001 ffffffff 00000000 00000000
00000000 00000000 00000000
[ 17.415985] 5da0: eea901e0 c004ac64 00000000 00000000 a0000113
c0063898 ebb35df0 2e20b361
[ 17.415985] 5dc0: ebb35dfc ebb35e10 ee28bc00 ebb35de0 ebb35e04
ebb35de0 c0063898 c006348c
[ 17.454864] 5de0: 50b99e70 2e20b361 c0098020 ee5ad840 00000001
00000001 40000040 00000800
[ 17.470703] 5e00: ed2d10e0 c0063868 ebb35d1c ebb35f64 50b99e70
ebb35e64 ebb35ea4 40000000
[ 17.470703] 5e20: bea319c0 ebb35e64 00000008 00000000 00000800
ebb35f64 ed2d10e0 40000040
[ 17.509063] 5e40: bea319c0 00000000 ebb35f54 ebb35e58 c0591b6c
c05907b8 bea309a0 00000008
[ 17.509735] 5e60: ed01c710 b819e6ad 0000076b 00000000 60000113
00000002 ebb35ed4 ebb35e88
[ 17.533172] 5e80: c00975fc c0096858 00000002 ed01c710 c014ed30
c10bb6d0 ebb35ed4 ebb35ea8
[ 17.548797] 5ea0: c00974a0 c00972f8 00000000 00000002 ed01c710
00000000 00000000 00000000
[ 17.564422] 5ec0: 00000001 c014ed4c ebb35f3c ebb35ed8 c014ed4c
c0085444 00000001 00000000
[ 17.580047] 5ee0: c014eb68 00000000 ebb35f3c ea1b94e0 00000000
00000001 00000000 ea1b94e8
[ 17.580047] 5f00: c011637c 00000000 00000000 c0590a0c c00974a0
00000001 ea1b94e8 00000002
[ 17.580047] 5f20: ebb35f84 271aec19 bea319c0 ed2d10e0 40000000
bea319c0 00000129 c0013368
[ 17.637512] 5f40: ebb34000 00000000 ebb35fa4 ebb35f58 c0592ea4
c0591a58 00000000 00000000
[ 17.651184] 5f60: ebb35fa4 00000000 00000000 ebb35e64 00000001
bea309a0 0000100c 40000000
[ 17.658172] 5f80: 00000000 fffffff7 00000000 00000004 b81995dc
00000800 00000000 ebb35fa8
[ 17.673797] 5fa0: c00130e0 c0592e64 00000004 b81995dc 00000004
bea319c0 40000000 bea319ac
[ 17.689422] 5fc0: 00000004 b81995dc 00000800 00000129 00000000
bea31a2c 00000000 bea319dc
[ 17.689422] 5fe0: 00000000 bea3099c b6ee1e39 b6ebbf6c 40000110
00000004 00000000 00000000
[ 17.733093] [<c0599324>] (skb_dequeue+0x5c/0x74) from [<c0627764>]
(unix_stream_recvmsg+0x124/0x544)
[ 17.744262] [<c0627764>] (unix_stream_recvmsg+0x124/0x544) from
[<c059086c>] (sock_recvmsg+0xc0/0xdc)
[ 17.763671] [<c059086c>] (sock_recvmsg+0xc0/0xdc) from [<c0591b6c>]
(__sys_recvmsg+0x120/0x1e4)
[ 17.783172] [<c0591b6c>] (__sys_recvmsg+0x120/0x1e4) from
[<c0592ea4>] (sys_recvmsg+0x4c/0x70)
[ 17.783172] [<c0592ea4>] (sys_recvmsg+0x4c/0x70) from [<c00130e0>]
(ret_fast_syscall+0x0/0x3c)
[ 17.806610] Code: e5853008 e894000c e5840004 e5840000 (e5823004)
[ 17.830749] ---[ end trace da227214a82491b9 ]---
[ 17.830749] note: upstart-socket-[1407] exited with preempt_count 2
[ 17.853454] BUG: scheduling while atomic: upstart-socket-/1407/0x40000003
[ 17.872467] INFO: lockdep is turned off.
[ 17.884063] Modules linked in: wl12xx_sdio wl12xx
[ 17.896423] [<c001b5f0>] (unwind_backtrace+0x0/0xec) from
[<c0720fc8>] (dump_stack+0x20/0x24)
[ 17.896423] [<c0720fc8>] (dump_stack+0x20/0x24) from [<c072291c>]
(__schedule_bug+0x84/0x94)
[ 17.929504] [<c072291c>] (__schedule_bug+0x84/0x94) from [<c074170c>]
(__schedule+0x88/0x5c8)
[ 17.946105] [<c074170c>] (__schedule+0x88/0x5c8) from [<c00563dc>]
(__cond_resched+0x30/0x40)
[ 17.962493] [<c00563dc>] (__cond_resched+0x30/0x40) from [<c0741cf0>]
(_cond_resched+0x40/0x50)
[ 17.979675] [<c0741cf0>] (_cond_resched+0x40/0x50) from [<c00fe494>]
(unmap_page_range+0x170/0x178)
[ 17.997039] [<c00fe494>] (unmap_page_range+0x170/0x178) from
[<c00fea30>] (unmap_vmas+0x88/0xb8)
[ 18.014251] [<c00fea30>] (unmap_vmas+0x88/0xb8) from [<c0102c44>]
(exit_mmap+0x138/0x244)
[ 18.030914] [<c0102c44>] (exit_mmap+0x138/0x244) from [<c005c1e8>]
(mmput.part.36+0x24/0xe0)
[ 18.047698] [<c005c1e8>] (mmput.part.36+0x24/0xe0) from [<c005c2e4>]
(mmput+0x40/0x44)
[ 18.063964] [<c005c2e4>] (mmput+0x40/0x44) from [<c00606ec>]
(exit_mm+0x158/0x160)
[ 18.079833] [<c00606ec>] (exit_mm+0x158/0x160) from [<c00624b0>]
(do_exit+0x200/0x39c)
[ 18.096038] [<c00624b0>] (do_exit+0x200/0x39c) from [<c00176c4>]
(die+0xf8/0x10c)
[ 18.111816] [<c00176c4>] (die+0xf8/0x10c) from [<c0721190>]
(__do_kernel_fault.part.7+0x64/0x84)
[ 18.129119] [<c0721190>] (__do_kernel_fault.part.7+0x64/0x84) from
[<c0747758>] (do_page_fault+0x334/0x358)
[ 18.142852] [<c0747758>] (do_page_fault+0x334/0x358) from
[<c07477a8>] (do_translation_fault+0x2c/0xb0)
[ 18.165985] [<c07477a8>] (do_translation_fault+0x2c/0xb0) from
[<c0008518>] (do_DataAbort+0x44/0xa8)
[ 18.183898] [<c0008518>] (do_DataAbort+0x44/0xa8) from [<c074571c>]
(__dabt_svc+0x3c/0x60)
[ 18.200927] Exception stack(0xebb35c80 to 0xebb35cc8)
[ 18.214416] 5c80: 00000000 60000113 00000000 00000000 ebca0200
ee5b1b78 ee5b1b84 ee5b1ae0
[ 18.227050] 5ca0: ebb35df0 0000076b 00000000 ebb35ce4 ebb35c98
ebb35cc8 c07449dc c0599324
[ 18.248596] 5cc0: a0000193 ffffffff
[ 18.260498] [<c074571c>] (__dabt_svc+0x3c/0x60) from [<c0599324>]
(skb_dequeue+0x5c/0x74)
[ 18.277526] [<c0599324>] (skb_dequeue+0x5c/0x74) from [<c0627764>]
(unix_stream_recvmsg+0x124/0x544)
[ 18.295684] [<c0627764>] (unix_stream_recvmsg+0x124/0x544) from
[<c059086c>] (sock_recvmsg+0xc0/0xdc)
[ 18.306427] [<c059086c>] (sock_recvmsg+0xc0/0xdc) from [<c0591b6c>]
(__sys_recvmsg+0x120/0x1e4)
[ 18.331939] [<c0591b6c>] (__sys_recvmsg+0x120/0x1e4) from
[<c0592ea4>] (sys_recvmsg+0x4c/0x70)
[ 18.349853] [<c0592ea4>] (sys_recvmsg+0x4c/0x70) from [<c00130e0>]
(ret_fast_syscall+0x0/0x3c)
[ 18.367767] hci0: type 1 len 4
[ 18.551147] (stc): remove_channel_from_table: id 3
[ 18.564880] (stc): remove_channel_from_table: id 2
[ 18.578430] (stc): remove_channel_from_table: id 4
[ 18.591888] (stc): all chnl_ids unregistered
[ 18.596588] (stk) :ldisc_install = 0
[ 18.609069] (stc): st_tty_close
[ 36.869201] BUG: spinlock lockup on CPU#1, init/1, ee5b1ddc
[ 37.578979] [<c001b5f0>] (unwind_backtrace+0x0/0xec) from
[<c0720fc8>] (dump_stack+0x20/0x24)
[ 37.596435] [<c0720fc8>] (dump_stack+0x20/0x24) from [<c07369dc>]
(__spin_lock_debug+0xc4/0xd0)
[ 37.613983] [<c07369dc>] (__spin_lock_debug+0xc4/0xd0) from
[<c03784c8>] (do_raw_spin_lock+0x94/0xc4)
[ 37.632141] [<c03784c8>] (do_raw_spin_lock+0x94/0xc4) from
[<c07448a4>] (_raw_spin_lock+0x5c/0x64)
[ 37.649597] [<c07448a4>] (_raw_spin_lock+0x5c/0x64) from [<c0627db0>]
(unix_stream_sendmsg+0x22c/0x3fc)
[ 37.668395] [<c0627db0>] (unix_stream_sendmsg+0x22c/0x3fc) from
[<c0590980>] (sock_sendmsg+0xb4/0xd0)
[ 37.686614] [<c0590980>] (sock_sendmsg+0xb4/0xd0) from [<c0590c40>]
(__sys_sendmsg+0x1e4/0x29c)
[ 37.704284] [<c0590c40>] (__sys_sendmsg+0x1e4/0x29c) from
[<c0592d20>] (sys_sendmsg+0x4c/0x70)
[ 37.721923] [<c0592d20>] (sys_sendmsg+0x4c/0x70) from [<c00130e0>]
(ret_fast_syscall+0x0/0x3c)
[ 38.345764] BUG: spinlock lockup on CPU#0, upstart-socket-/1407, ee5b1ddc
[ 38.345764] [<c001b5f0>] (unwind_backtrace+0x0/0xec) from
[<c0720fc8>] (dump_stack+0x20/0x24)
[ 39.153442] [<c0720fc8>] (dump_stack+0x20/0x24) from [<c07369dc>]
(__spin_lock_debug+0xc4/0xd0)
[ 39.171325] [<c07369dc>] (__spin_lock_debug+0xc4/0xd0) from
[<c03784c8>] (do_raw_spin_lock+0x94/0xc4)
[ 39.189819] [<c03784c8>] (do_raw_spin_lock+0x94/0xc4) from
[<c07448a4>] (_raw_spin_lock+0x5c/0x64)
[ 39.208099] [<c07448a4>] (_raw_spin_lock+0x5c/0x64) from [<c0628828>]
(unix_release_sock+0x44/0x220)
[ 39.226593] [<c0628828>] (unix_release_sock+0x44/0x220) from
[<c0628a34>] (unix_release+0x30/0x34)
[ 39.244873] [<c0628a34>] (unix_release+0x30/0x34) from [<c05912f8>]
(sock_release+0x30/0xd4)
[ 39.262603] [<c05912f8>] (sock_release+0x30/0xd4) from [<c0591da8>]
(sock_close+0x30/0x3c)
[ 39.280090] [<c0591da8>] (sock_close+0x30/0x3c) from [<c0117bd8>]
(__fput+0xf8/0x1bc)
[ 39.296997] [<c0117bd8>] (__fput+0xf8/0x1bc) from [<c0117cdc>]
(fput+0x40/0x44)
[ 39.313354] [<c0117cdc>] (fput+0x40/0x44) from [<c0114dcc>]
(filp_close+0x80/0x8c)
[ 39.329956] [<c0114dcc>] (filp_close+0x80/0x8c) from [<c0060944>]
(close_files+0xbc/0xec)
[ 39.340393] [<c0060944>] (close_files+0xbc/0xec) from [<c0062190>]
(put_files_struct+0x44/0xc8)
[ 39.365173] [<c0062190>] (put_files_struct+0x44/0xc8) from
[<c00622ac>] (exit_files+0x48/0x4c)
[ 39.382995] [<c00622ac>] (exit_files+0x48/0x4c) from [<c0062560>]
(do_exit+0x2b0/0x39c)
[ 39.400146] [<c0062560>] (do_exit+0x2b0/0x39c) from [<c00176c4>]
(die+0xf8/0x10c)
[ 39.416687] [<c00176c4>] (die+0xf8/0x10c) from [<c0721190>]
(__do_kernel_fault.part.7+0x64/0x84)
[ 39.434600] [<c0721190>] (__do_kernel_fault.part.7+0x64/0x84) from
[<c0747758>] (do_page_fault+0x334/0x358)
[ 39.453582] [<c0747758>] (do_page_fault+0x334/0x358) from
[<c07477a8>] (do_translation_fault+0x2c/0xb0)
[ 39.472412] [<c07477a8>] (do_translation_fault+0x2c/0xb0) from
[<c0008518>] (do_DataAbort+0x44/0xa8)
[ 39.490844] [<c0008518>] (do_DataAbort+0x44/0xa8) from [<c074571c>]
(__dabt_svc+0x3c/0x60)
[ 39.508239] Exception stack(0xebb35c80 to 0xebb35cc8)
[ 39.522155] 5c80: 00000000 60000113 00000000 00000000 ebca0200
ee5b1b78 ee5b1b84 ee5b1ae0
[ 39.539550] 5ca0: ebb35df0 0000076b 00000000 ebb35ce4 ebb35c98
ebb35cc8 c07449dc c0599324
[ 39.556884] 5cc0: a0000193 ffffffff
[ 39.569091] [<c074571c>] (__dabt_svc+0x3c/0x60) from [<c0599324>]
(skb_dequeue+0x5c/0x74)
[ 39.586486] [<c0599324>] (skb_dequeue+0x5c/0x74) from [<c0627764>]
(unix_stream_recvmsg+0x124/0x544)
[ 39.604827] [<c0627764>] (unix_stream_recvmsg+0x124/0x544) from
[<c059086c>] (sock_recvmsg+0xc0/0xdc)
[ 39.623321] [<c059086c>] (sock_recvmsg+0xc0/0xdc) from [<c0591b6c>]
(__sys_recvmsg+0x120/0x1e4)
[ 39.641326] [<c0591b6c>] (__sys_recvmsg+0x120/0x1e4) from
[<c0592ea4>] (sys_recvmsg+0x4c/0x70)
[ 39.659271] [<c0592ea4>] (sys_recvmsg+0x4c/0x70) from [<c00130e0>]
(ret_fast_syscall+0x0/0x3c)
_______________________________________________
linaro-validation mailing list
[email protected]
http://lists.linaro.org/mailman/listinfo/linaro-validation
