Hi Brian

Thanks, here's a full stack trace.

Ciao

Zak

On 18 September 2016 at 21:58, Brian Bennett <brian.benn...@joyent.com> wrote:
> Zak,
>
> Considering that illumos #3917 is three years old, you've probably hit a 
> different bug involving mutexes. It would be best if you can give the full 
> stack trace, not just the top two frames. Having the full stack trace, I may 
> be able to identify the particular crash you encountered.
>
> Thanks.
>
> --
> Brian Bennett
> Systems Engineer, Cloud Operations
> Joyent, Inc. | www.joyent.com
>
>> On Sep 16, 2016, at 4:04 AM, Zak McGregor <z...@cloudafrica.net> wrote:
>>
>> Hi
>>
>> This issue here:
>> https://illumos.org/issues/3917
>>
>> seems to have hit one of our production boxes today. I took a look at
>> the dump and it seems to tally precisely with this issue.
>>
>> Here's a snippet:
>>
>> mdb -k unix.0 vmcore.0
>> Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc
>> pcplusmp scsi_vhci ufs ip hook neti sockfs arp usba uhci mm stmf_sbd
>> stmf zfs lofs idm crypto random cpc logindmux ptm kvm sd sppp nsmb
>> smbsrv nfs ipc ]
>>> ::status
>> debugging crash dump vmcore.0 (64-bit) from c7.jhb.cloudafrica.net
>> operating system: 5.11 joyent_20160330T234717Z (i86pc)
>> image uuid: (not set)
>> panic message: turnstile_block(ffffff3d8b4db508): unowned mutex
>> dump content: kernel pages only
>>> ::stack
>> vpanic()
>> turnstile_block+0x78a(0, 0, ffffff3e13c64d00, fffffffffbc07ac0, 0, 0)
>> mutex_vector_enter+0x3a3(ffffff3e13c64d00)
>>
>> If there is any further information you'd like please let me know.
>>
>> Thanks
>>
>> Cheers
>>
>> Zak
>>
> 
> 

::status:
debugging crash dump vmcore.0 (64-bit) from c1b.jhb.cloudafrica.net
operating system: 5.11 joyent_20160330T234717Z (i86pc)
image uuid: (not set)
panic message: turnstile_block(ffffff1b3466b168): unowned mutex
dump content: kernel pages only
::stack:
vpanic()
turnstile_block+0x78a(0, 0, ffffff1addd4a1b0, fffffffffbc07ac0, 0, 0)
mutex_vector_enter+0x3a3(ffffff1addd4a1b0)
vnd_mac_input+0x12a(ffffff1addd4a0f8, 0, ffffff19e8531ce0, ffffff00bb5764e0)
dls_rx_promisc+0x119(ffffff19ea49b608, 0, ffffff19e8531ce0, 0)
mac_promisc_dispatch_one+0x81(ffffff19ea02a620, ffffff1a9ac55120, 0, 1)
mac_promisc_dispatch+0xb2(ffffff19ad5f26c0, ffffff1a9ac55120, ffffff19ce6cf1d0, 
1)
mac_tx_send+0x33f(ffffff19ce6cf1d0, ffffff19ac91ad88, ffffff1a9ac55120, 
ffffff00bb576770)
mac_tx_single_ring_mode+0x6e(ffffff1a48ac6340, ffffff1a9ac55120, 0, 1, 0)
mac_tx+0xda(ffffff19ce6cf1d0, ffffff1a9ac55120, 0, 1, 0)
str_mdata_raw_fastpath_put+0x85(ffffff19ea49b018, ffffff1a9ac55120, 0, 1)
vnd_squeue_tx_one+0x6a(ffffff19f7ea2c28, ffffff1a9ac55120)
vnd_squeue_tx_drain+0x112(ffffff19f7ea2c28, 0, 0, 0)
vnd_squeue_tx_append+0x103(ffffff19f7ea2c28, ffffff1a9ac55120, ffffff19ac390740
, 0)
squeue_enter+0x41c(ffffff19ac390740, ffffff1a9ac55120, ffffff1a9ac55120, 1, 0, 4
)
gsqueue_enter_one+0x43(ffffff19ac390740, ffffff1a9ac55120, fffffffff80da2d0, 
ffffff19f7ea2c28, 4, 44)
vnd_frameio_write+0x10e(ffffff19ea029cc8, f9c8c0, 200083)
vnd_ioctl+0x270(11b00000003, 766e6431, f9c8c0, 200083, ffffff19dca5bef0, 
ffffff00bb576ea8)
cdev_ioctl+0x39(11b00000003, 766e6431, f9c8c0, 200083, ffffff19dca5bef0, 
ffffff00bb576ea8)
spec_ioctl+0x60(ffffff1a1d60ea40, 766e6431, f9c8c0, 200083, ffffff19dca5bef0, 
ffffff00bb576ea8)
fop_ioctl+0x55(ffffff1a1d60ea40, 766e6431, f9c8c0, 200083, ffffff19dca5bef0, 
ffffff00bb576ea8)
ioctl+0x9b(a, 766e6431, f9c8c0)
sys_syscall+0x1bd()
::panicinfo:
             cpu               13
          thread ffffff19ef7fc140
         message turnstile_block(ffffff1b3466b168): unowned mutex
             rdi fffffffffbb6dce8
             rsi ffffff00bb576280
             rdx                6
             rcx ffffff1a060aeae0
              r8               ad
              r9                2
             rax ffffff00bb5762a0
             rbx ffffff1b3466b168
             rbp ffffff00bb5762e0
             r10 ffffff1b3466b198
             r11         ffffffff
             r12 fffffffffbc07ac0
             r13 ffffff1addd4a1b0
             r14 ffffff19ef7fc140
             r15 ffffff19fd3edec0
          fsbase fffffd7fff102a40
          gsbase ffffff1992c28580
              ds               4b
              es               4b
              fs                0
              gs                0
          trapno                0
             err                0
             rip fffffffffb8612f0
              cs               30
          rflags              282
             rsp ffffff00bb576278
              ss               38
          gdt_hi                0
          gdt_lo         2000ffff
          idt_hi                0
          idt_lo         1000ffff
             ldt                0
            task               70
             cr0         8005003b
             cr2     7f442b395314
             cr3       16307c1000
             cr4             26f8
::cpuinfo -v:
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc327c0  1f    0    0 109   no    no t-1    ffffff00b80e6c40 sched
                       |         |
            RUNNING <--+         +--> PIL THREAD
              READY                    10 ffffff00b80e6c40
           QUIESCED                     - ffffff19e9ae9b00 qemu-system-x86_
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  1 ffffff1992a4eb00  1f    1    0  -1   no    no t-1    ffffff00b842bc40
 (idle)
                       |    |
            RUNNING <--+    +-->  PRI THREAD           PROC
              READY                60 ffffff00bc070c40 sched
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  2 ffffff1992c44500  1f    0    0  -1   no    no t-1    ffffff00b84b6c40
 (idle)
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  3 ffffff1992c43000  1f    0    0  59   no    no t-22   ffffff19eef51420 
child_setup
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  4 ffffff1992c3a580  1f    0    0   1   no    no t-0    ffffff19ed715780 
qemu-system-x86_
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  5 ffffff1992c39080  1f    0    0  -1   no    no t-1    ffffff00b86f7c40
 (idle)
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  6 ffffff1992c37a80  1f    0    0  -1   no    no t-1    ffffff00b877dc40
 (idle)
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  7 ffffff1992c32540  1f    0    0   2   no    no t-1    ffffff19e0cfb4e0 cnc.js
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  8 ffffff1992c31040  1f    0    0  -1   no    no t-1    ffffff00b8889c40
 (idle)
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  9 ffffff1992c2fb00  1f    0    0  -1   no    no t-1    ffffff00b890fc40
 (idle)
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
 10 ffffff1992c2e500  1f    0    0  -1   no    no t-1    ffffff00b8995c40
 (idle)
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
 11 ffffff1992c2d000  1f    0    0  -1   no    no t-3    ffffff00b8a1bc40
 (idle)
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
 12 ffffff1992c2bac0  1f    0    0  -1   no    no t-0    ffffff00b8aa1c40
 (idle)
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
 13 fffffffffbc3d0c0  1b    0    0   1   no    no t-5    ffffff19ef7fc140 
qemu-system-x86_
                       |    
            RUNNING <--+    
              READY         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
 14 ffffff1992c27080  1f    0    0  -1   no    no t-1    ffffff00b8badc40
 (idle)
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
 15 ffffff1992e54a80  1f    0    0   1   no    no t-5    ffffff19ceca70c0 fmd
                       |    
            RUNNING <--+    
              READY         
           QUIESCED         
             EXISTS         
             ENABLE         

::msgbuf:
MESSAGE                                                               
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ac9cc6000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x80fe6e50 data fffffd7fffdfe770
unhandled wrmsr: 0x80fe6e50 data fffffd7fffdfe770
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ac9cc6000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
WARNING: ip_interface_cleanup: cannot open /devices/pseudo/udp6@0:udp6: error 13
WARNING: ip_interface_cleanup: cannot open /devices/pseudo/udp@0:udp: error 13
NOTICE: vnic1094 unregistered
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
/pseudo/zconsnex@1/zcons@2 (zcons2) online
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
/pseudo/zconsnex@1/zcons@2 (zcons2) online
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: vnic1095 registered
kvm_lapic_reset: vcpu=ffffff1a19d6f000, id=0, base_msr= fee00100 PRIx64 base_add
ress=fee00000
vmcs revision_id = f
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00000 PRIx64 base_add
ress=fee00000
vmcs revision_id = f
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
unhandled wrmsr: 0x1010101 data fffffd7fffdfe860
unhandled wrmsr: 0x1010101 data fffffd7fffdfe860
unhandled wrmsr: 0xff30ccec data fffffd7fffdfe830
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
WARNING: ip_interface_cleanup: cannot open /devices/pseudo/udp6@0:udp6: error 13
WARNING: ip_interface_cleanup: cannot open /devices/pseudo/udp@0:udp: error 13
NOTICE: vnic1095 unregistered
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
/pseudo/zconsnex@1/zcons@2 (zcons2) online
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
/pseudo/zconsnex@1/zcons@2 (zcons2) online
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: vnic1096 registered
kvm_lapic_reset: vcpu=ffffff355a62e000, id=0, base_msr= fee00100 PRIx64 base_add
ress=fee00000
vmcs revision_id = f
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00000 PRIx64 base_add
ress=fee00000
vmcs revision_id = f
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: IRQ19 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
unhandled wrmsr: 0x1010101 data fffffd7fffdfe860
unhandled wrmsr: 0x1010101 data fffffd7fffdfe860
unhandled wrmsr: 0xff30ccec data fffffd7fffdfe830
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
unhandled rdmsr: 0x3c0003f
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x82954490 data fffffd7fffdfe770
unhandled wrmsr: 0x82954490 data fffffd7fffdfe770
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x0 data 0
unhandled wrmsr: 0x810463f0 data fffffd7fffdfe770
unhandled wrmsr: 0x810463f0 data fffffd7fffdfe770
vcpu 1 received sipi with vector # 10
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000
unhandled wrmsr: 0x0 data 0
vcpu 1 received sipi with vector # 97
kvm_lapic_reset: vcpu=ffffff1ad131a000, id=1, base_msr= fee00800 PRIx64 base_add
ress=fee00000


panic[cpu13]/thread=ffffff19ef7fc140: 
turnstile_block(ffffff1b3466b168): unowned mutex


ffffff00bb576380 genunix:turnstile_block+78a ()
ffffff00bb5763f0 unix:mutex_vector_enter+3a3 ()
ffffff00bb5764c0 vnd:vnd_mac_input+12a ()
ffffff00bb576580 dls:dls_rx_promisc+119 ()
ffffff00bb5765e0 mac:mac_promisc_dispatch_one+81 ()
ffffff00bb576660 mac:mac_promisc_dispatch+b2 ()
ffffff00bb576750 mac:mac_tx_send+33f ()
ffffff00bb5767f0 mac:mac_tx_single_ring_mode+6e ()
ffffff00bb5768a0 mac:mac_tx+da ()
ffffff00bb576950 dld:str_mdata_raw_fastpath_put+85 ()
ffffff00bb576990 vnd:vnd_squeue_tx_one+6a ()
ffffff00bb576a20 vnd:vnd_squeue_tx_drain+112 ()
ffffff00bb576ac0 vnd:vnd_squeue_tx_append+103 ()
ffffff00bb576b50 ip:squeue_enter+41c ()
ffffff00bb576ba0 gsqueue:gsqueue_enter_one+43 ()
ffffff00bb576c40 vnd:vnd_frameio_write+10e ()
ffffff00bb576cc0 vnd:vnd_ioctl+270 ()
ffffff00bb576d00 genunix:cdev_ioctl+39 ()
ffffff00bb576d50 specfs:spec_ioctl+60 ()
ffffff00bb576de0 genunix:fop_ioctl+55 ()
ffffff00bb576f00 genunix:ioctl+9b ()
ffffff00bb576f10 unix:brand_sys_syscall+238 ()

syncing file systems...
 done


-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Reply via email to