Hi

This has now happened to 3 hosts in the last 3 days. Any idea what we can
look at ?

It seems to happen under high load on those systems, all older E5 based
hosts.

We just had another reboot and this is in /var/adm/messages

2016-09-19T11:49:39.065057+00:00 c1a unix: [ID 836849 kern.notice]
#012#015panic[cpu14]/thread=ffffff19e4687420:
2016-09-19T11:49:39.065068+00:00 c1a genunix: [ID 761616 kern.notice]
turnstile_block(ffffff19ab6e9230): unowned mutex
2016-09-19T11:49:39.065074+00:00 c1a unix: [ID 100000 kern.notice] #012
2016-09-19T11:49:39.065079+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689380 genunix:turnstile_block+78a ()
2016-09-19T11:49:39.065084+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba6893f0 unix:mutex_vector_enter+3a3 ()
2016-09-19T11:49:39.065089+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba6894c0 vnd:vnd_mac_input+12a ()
2016-09-19T11:49:39.065094+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689580 dls:dls_rx_promisc+119 ()
2016-09-19T11:49:39.065099+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba6895e0 mac:mac_promisc_dispatch_one+81 ()
2016-09-19T11:49:39.065104+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689660 mac:mac_promisc_dispatch+b2 ()
2016-09-19T11:49:39.065109+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689750 mac:mac_tx_send+33f ()
2016-09-19T11:49:39.065114+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba6897f0 mac:mac_tx_single_ring_mode+6e ()
2016-09-19T11:49:39.065132+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba6898a0 mac:mac_tx+da ()
2016-09-19T11:49:39.065139+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689950 dld:str_mdata_raw_fastpath_put+85 ()
2016-09-19T11:49:39.065144+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689990 vnd:vnd_squeue_tx_one+6a ()
2016-09-19T11:49:39.065149+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689a20 vnd:vnd_squeue_tx_drain+112 ()
2016-09-19T11:49:39.065164+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689ac0 vnd:vnd_squeue_tx_append+103 ()
2016-09-19T11:49:39.065170+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689b50 ip:squeue_enter+41c ()
2016-09-19T11:49:39.065175+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689ba0 gsqueue:gsqueue_enter_one+43 ()
2016-09-19T11:49:39.065179+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689c40 vnd:vnd_frameio_write+10e ()
2016-09-19T11:49:39.065184+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689cc0 vnd:vnd_ioctl+270 ()
2016-09-19T11:49:39.065196+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689d00 genunix:cdev_ioctl+39 ()
2016-09-19T11:49:39.065202+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689d50 specfs:spec_ioctl+60 ()
2016-09-19T11:49:39.065208+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689de0 genunix:fop_ioctl+55 ()
2016-09-19T11:49:39.065213+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689f00 genunix:ioctl+9b ()
2016-09-19T11:49:39.065218+00:00 c1a genunix: [ID 655072 kern.notice]
ffffff00ba689f10 unix:brand_sys_syscall+238 ()
2016-09-19T11:49:39.065224+00:00 c1a unix: [ID 100000 kern.notice]


Thanks
Len

On Mon, 19 Sep 2016 at 09:34 Zak McGregor <z...@cloudafrica.net> wrote:

> Hi Brian
>
> Thanks, here's a full stack trace.
>
> Ciao
>
> Zak
>
> On 18 September 2016 at 21:58, Brian Bennett <brian.benn...@joyent.com>
> wrote:
> > Zak,
> >
> > Considering that illumos #3917 is three years old, you've probably hit a
> different bug involving mutexes. It would be best if you can give the full
> stack trace, not just the top two frames. Having the full stack trace, I
> may be able to identify the particular crash you encountered.
> >
> > Thanks.
> >
> > --
> > Brian Bennett
> > Systems Engineer, Cloud Operations
> > Joyent, Inc. | www.joyent.com
> >
> >> On Sep 16, 2016, at 4:04 AM, Zak McGregor <z...@cloudafrica.net> wrote:
> >>
> >> Hi
> >>
> >> This issue here:
> >> https://illumos.org/issues/3917
> >>
> >> seems to have hit one of our production boxes today. I took a look at
> >> the dump and it seems to tally precisely with this issue.
> >>
> >> Here's a snippet:
> >>
> >> mdb -k unix.0 vmcore.0
> >> Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc
> >> pcplusmp scsi_vhci ufs ip hook neti sockfs arp usba uhci mm stmf_sbd
> >> stmf zfs lofs idm crypto random cpc logindmux ptm kvm sd sppp nsmb
> >> smbsrv nfs ipc ]
> >>> ::status
> >> debugging crash dump vmcore.0 (64-bit) from c7.jhb.cloudafrica.net
> >> operating system: 5.11 joyent_20160330T234717Z (i86pc)
> >> image uuid: (not set)
> >> panic message: turnstile_block(ffffff3d8b4db508): unowned mutex
> >> dump content: kernel pages only
> >>> ::stack
> >> vpanic()
> >> turnstile_block+0x78a(0, 0, ffffff3e13c64d00, fffffffffbc07ac0, 0, 0)
> >> mutex_vector_enter+0x3a3(ffffff3e13c64d00)
> >>
> >> If there is any further information you'd like please let me know.
> >>
> >> Thanks
> >>
> >> Cheers
> >>
> >> Zak
> >>
> >
> >
>
>



-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription: 
https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com

Reply via email to