Hi Brian
Thanks, here's a full stack trace.
Ciao
Zak
On 18 September 2016 at 21:58, Brian Bennett <[email protected]> wrote:
> Zak,
>
> Considering that illumos #3917 is three years old, you've probably hit a
> different bug involving mutexes. It would be best if you can give the full
> stack trace, not just the top two frames. Having the full stack trace, I may
> be able to identify the particular crash you encountered.
>
> Thanks.
>
> --
> Brian Bennett
> Systems Engineer, Cloud Operations
> Joyent, Inc. | www.joyent.com
>
>> On Sep 16, 2016, at 4:04 AM, Zak McGregor <[email protected]> wrote:
>>
>> Hi
>>
>> This issue here:
>> https://illumos.org/issues/3917
>>
>> seems to have hit one of our production boxes today. I took a look at
>> the dump and it seems to tally precisely with this issue.
>>
>> Here's a snippet:
>>
>> mdb -k unix.0 vmcore.0
>> Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc
>> pcplusmp scsi_vhci ufs ip hook neti sockfs arp usba uhci mm stmf_sbd
>> stmf zfs lofs idm crypto random cpc logindmux ptm kvm sd sppp nsmb
>> smbsrv nfs ipc ]
>>> ::status
>> debugging crash dump vmcore.0 (64-bit) from c7.jhb.cloudafrica.net
>> operating system: 5.11 joyent_20160330T234717Z (i86pc)
>> image uuid: (not set)
>> panic message: turnstile_block(ffffff3d8b4db508): unowned mutex
>> dump content: kernel pages only
>>> ::stack
>> vpanic()
>> turnstile_block+0x78a(0, 0, ffffff3e13c64d00, fffffffffbc07ac0, 0, 0)
>> mutex_vector_enter+0x3a3(ffffff3e13c64d00)
>>
>> If there is any further information you'd like please let me know.
>>
>> Thanks
>>
>> Cheers
>>
>> Zak
>>
>
>
::status:
debugging crash dump vmcore.0 (64-bit) from c1b.jhb.cloudafrica.net
operating system: 5.11 joyent_20160330T234717Z (i86pc)
image uuid: (not set)
panic message: turnstile_block(ffffff1b3466b168): unowned mutex
dump content: kernel pages only
::stack:
vpanic()
turnstile_block+0x78a(0, 0, ffffff1addd4a1b0, fffffffffbc07ac0, 0, 0)
mutex_vector_enter+0x3a3(ffffff1addd4a1b0)
vnd_mac_input+0x12a(ffffff1addd4a0f8, 0, ffffff19e8531ce0, ffffff00bb5764e0)
dls_rx_promisc+0x119(ffffff19ea49b608, 0, ffffff19e8531ce0, 0)
mac_promisc_dispatch_one+0x81(ffffff19ea02a620, ffffff1a9ac55120, 0, 1)
mac_promisc_dispatch+0xb2(ffffff19ad5f26c0, ffffff1a9ac55120, ffffff19ce6cf1d0,
1)
mac_tx_send+0x33f(ffffff19ce6cf1d0, ffffff19ac91ad88, ffffff1a9ac55120,
ffffff00bb576770)
mac_tx_single_ring_mode+0x6e(ffffff1a48ac6340, ffffff1a9ac55120, 0, 1, 0)
mac_tx+0xda(ffffff19ce6cf1d0, ffffff1a9ac55120, 0, 1, 0)
str_mdata_raw_fastpath_put+0x85(ffffff19ea49b018, ffffff1a9ac55120, 0, 1)
vnd_squeue_tx_one+0x6a(ffffff19f7ea2c28, ffffff1a9ac55120)
vnd_squeue_tx_drain+0x112(ffffff19f7ea2c28, 0, 0, 0)
vnd_squeue_tx_append+0x103(ffffff19f7ea2c28, ffffff1a9ac55120, ffffff19ac390740
, 0)
squeue_enter+0x41c(ffffff19ac390740, ffffff1a9ac55120, ffffff1a9ac55120, 1, 0, 4
)
gsqueue_enter_one+0x43(ffffff19ac390740, ffffff1a9ac55120, fffffffff80da2d0,
ffffff19f7ea2c28, 4, 44)
vnd_frameio_write+0x10e(ffffff19ea029cc8, f9c8c0, 200083)
vnd_ioctl+0x270(11b00000003, 766e6431, f9c8c0, 200083, ffffff19dca5bef0,
ffffff00bb576ea8)
cdev_ioctl+0x39(11b00000003, 766e6431, f9c8c0, 200083, ffffff19dca5bef0,
ffffff00bb576ea8)
spec_ioctl+0x60(ffffff1a1d60ea40, 766e6431, f9c8c0, 200083, ffffff19dca5bef0,
ffffff00bb576ea8)
fop_ioctl+0x55(ffffff1a1d60ea40, 766e6431, f9c8c0, 200083, ffffff19dca5bef0,
ffffff00bb576ea8)
ioctl+0x9b(a, 766e6431, f9c8c0)
sys_syscall+0x1bd()
::panicinfo:
cpu 13
thread ffffff19ef7fc140
message turnstile_block(ffffff1b3466b168): unowned mutex
rdi fffffffffbb6dce8
rsi ffffff00bb576280
rdx 6
rcx ffffff1a060aeae0
r8 ad
r9 2
rax ffffff00bb5762a0
rbx ffffff1b3466b168
rbp ffffff00bb5762e0
r10 ffffff1b3466b198
r11 ffffffff
r12 fffffffffbc07ac0
r13 ffffff1addd4a1b0
r14 ffffff19ef7fc140
r15 ffffff19fd3edec0
fsbase fffffd7fff102a40
gsbase ffffff1992c28580
ds 4b
es 4b
fs 0
gs 0
trapno 0
err 0
rip fffffffffb8612f0
cs 30
rflags 282
rsp ffffff00bb576278
ss 38
gdt_hi 0
gdt_lo 2000ffff
idt_hi 0
idt_lo 1000ffff
ldt 0
task 70
cr0 8005003b
cr2 7f442b395314
cr3 16307c1000
cr4 26f8
::cpuinfo -v:
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 fffffffffbc327c0 1f 0 0 109 no no t-1 ffffff00b80e6c40 sched
| |
RUNNING <--+ +--> PIL THREAD
READY 10 ffffff00b80e6c40
QUIESCED - ffffff19e9ae9b00 qemu-system-x86_
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
1 ffffff1992a4eb00 1f 1 0 -1 no no t-1 ffffff00b842bc40
(idle)
| |
RUNNING <--+ +--> PRI THREAD PROC
READY 60 ffffff00bc070c40 sched
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
2 ffffff1992c44500 1f 0 0 -1 no no t-1 ffffff00b84b6c40
(idle)
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
3 ffffff1992c43000 1f 0 0 59 no no t-22 ffffff19eef51420
child_setup
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
4 ffffff1992c3a580 1f 0 0 1 no no t-0 ffffff19ed715780
qemu-system-x86_
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
5 ffffff1992c39080 1f 0 0 -1 no no t-1 ffffff00b86f7c40
(idle)
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
6 ffffff1992c37a80 1f 0 0 -1 no no t-1 ffffff00b877dc40
(idle)
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
7 ffffff1992c32540 1f 0 0 2 no no t-1 ffffff19e0cfb4e0 cnc.js
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
8 ffffff1992c31040 1f 0 0 -1 no no t-1 ffffff00b8889c40
(idle)
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
9 ffffff1992c2fb00 1f 0 0 -1 no no t-1 ffffff00b890fc40
(idle)
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
10 ffffff1992c2e500 1f 0 0 -1 no no t-1 ffffff00b8995c40
(idle)
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
11 ffffff1992c2d000 1f 0 0 -1 no no t-3 ffffff00b8a1bc40
(idle)
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
12 ffffff1992c2bac0 1f 0 0 -1 no no t-0 ffffff00b8aa1c40
(idle)
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
13 fffffffffbc3d0c0 1b 0 0 1 no no t-5 ffffff19ef7fc140
qemu-system-x86_
|
RUNNING <--+
READY
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
14 ffffff1992c27080 1f 0 0 -1 no no t-1 ffffff00b8badc40
(idle)
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
15 ffffff1992e54a80 1f 0 0 1 no no t-5 ffffff19ceca70c0 fmd
|
RUNNING <--+
READY
QUIESCED
EXISTS
ENABLE
::msgbuf:
MESSAGE
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ac9cc6000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x80fe6e50 data fffffd7fffdfe770
unhandled wrmsr: 0x80fe6e50 data fffffd7fffdfe770
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ac9cc6000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
WARNING: ip_interface_cleanup: cannot open /devices/pseudo/udp6@0:udp6: error 13
WARNING: ip_interface_cleanup: cannot open /devices/pseudo/udp@0:udp: error 13
NOTICE: vnic1094 unregistered
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
/pseudo/zconsnex@1/zcons@2 (zcons2) online
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
/pseudo/zconsnex@1/zcons@2 (zcons2) online
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: vnic1095 registered
kvm_lapic_reset: vcpu=ffffff1a19d6f000, id=0, base_msr= fee00100 PRIx64 base_add
ress=fee00000
vmcs revision_id = f
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00000 PRIx64 base_add
ress=fee00000
vmcs revision_id = f
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
unhandled wrmsr: 0x1010101 data fffffd7fffdfe860
unhandled wrmsr: 0x1010101 data fffffd7fffdfe860
unhandled wrmsr: 0xff30ccec data fffffd7fffdfe830
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
WARNING: ip_interface_cleanup: cannot open /devices/pseudo/udp6@0:udp6: error 13
WARNING: ip_interface_cleanup: cannot open /devices/pseudo/udp@0:udp: error 13
NOTICE: vnic1095 unregistered
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
/pseudo/zconsnex@1/zcons@2 (zcons2) online
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
/pseudo/zconsnex@1/zcons@2 (zcons2) online
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: vnic1096 registered
kvm_lapic_reset: vcpu=ffffff355a62e000, id=0, base_msr= fee00100 PRIx64 base_add
ress=fee00000
vmcs revision_id = f
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00000 PRIx64 base_add
ress=fee00000
vmcs revision_id = f
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
unhandled wrmsr: 0x1010101 data fffffd7fffdfe860
unhandled wrmsr: 0x1010101 data fffffd7fffdfe860
unhandled wrmsr: 0xff30ccec data fffffd7fffdfe830
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x82954490 data fffffd7fffdfe770
unhandled wrmsr: 0x82954490 data fffffd7fffdfe770
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x810463f0 data fffffd7fffdfe770
unhandled wrmsr: 0x810463f0 data fffffd7fffdfe770
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
panic[cpu13]/thread=ffffff19ef7fc140:
turnstile_block(ffffff1b3466b168): unowned mutex
ffffff00bb576380 genunix:turnstile_block+78a ()
ffffff00bb5763f0 unix:mutex_vector_enter+3a3 ()
ffffff00bb5764c0 vnd:vnd_mac_input+12a ()
ffffff00bb576580 dls:dls_rx_promisc+119 ()
ffffff00bb5765e0 mac:mac_promisc_dispatch_one+81 ()
ffffff00bb576660 mac:mac_promisc_dispatch+b2 ()
ffffff00bb576750 mac:mac_tx_send+33f ()
ffffff00bb5767f0 mac:mac_tx_single_ring_mode+6e ()
ffffff00bb5768a0 mac:mac_tx+da ()
ffffff00bb576950 dld:str_mdata_raw_fastpath_put+85 ()
ffffff00bb576990 vnd:vnd_squeue_tx_one+6a ()
ffffff00bb576a20 vnd:vnd_squeue_tx_drain+112 ()
ffffff00bb576ac0 vnd:vnd_squeue_tx_append+103 ()
ffffff00bb576b50 ip:squeue_enter+41c ()
ffffff00bb576ba0 gsqueue:gsqueue_enter_one+43 ()
ffffff00bb576c40 vnd:vnd_frameio_write+10e ()
ffffff00bb576cc0 vnd:vnd_ioctl+270 ()
ffffff00bb576d00 genunix:cdev_ioctl+39 ()
ffffff00bb576d50 specfs:spec_ioctl+60 ()
ffffff00bb576de0 genunix:fop_ioctl+55 ()
ffffff00bb576f00 genunix:ioctl+9b ()
ffffff00bb576f10 unix:brand_sys_syscall+238 ()
syncing file systems...
done
-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription:
https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com
