Len or Zak, would either of you be able to provide us with a crash dump? It should be in /var/crash/volatile on any CN that had this panic occur.
-- Brian Bennett Systems Engineer, Cloud Operations Joyent, Inc. | www.joyent.com <http://www.joyent.com/> > On Sep 19, 2016, at 5:09 AM, Len Weincier <l...@cloudafrica.net> wrote: > > Hi > > This has now happened to 3 hosts in the last 3 days. Any idea what we can > look at ? > > It seems to happen under high load on those systems, all older E5 based hosts. > > We just had another reboot and this is in /var/adm/messages > > 2016-09-19T11:49:39.065057+00:00 c1a unix: [ID 836849 kern.notice] > #012#015panic[cpu14]/thread=ffffff19e4687420: > 2016-09-19T11:49:39.065068+00:00 c1a genunix: [ID 761616 kern.notice] > turnstile_block(ffffff19ab6e9230): unowned mutex > 2016-09-19T11:49:39.065074+00:00 c1a unix: [ID 100000 kern.notice] #012 > 2016-09-19T11:49:39.065079+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689380 genunix:turnstile_block+78a () > 2016-09-19T11:49:39.065084+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba6893f0 unix:mutex_vector_enter+3a3 () > 2016-09-19T11:49:39.065089+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba6894c0 vnd:vnd_mac_input+12a () > 2016-09-19T11:49:39.065094+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689580 dls:dls_rx_promisc+119 () > 2016-09-19T11:49:39.065099+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba6895e0 mac:mac_promisc_dispatch_one+81 () > 2016-09-19T11:49:39.065104+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689660 mac:mac_promisc_dispatch+b2 () > 2016-09-19T11:49:39.065109+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689750 mac:mac_tx_send+33f () > 2016-09-19T11:49:39.065114+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba6897f0 mac:mac_tx_single_ring_mode+6e () > 2016-09-19T11:49:39.065132+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba6898a0 mac:mac_tx+da () > 2016-09-19T11:49:39.065139+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689950 dld:str_mdata_raw_fastpath_put+85 () > 2016-09-19T11:49:39.065144+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689990 vnd:vnd_squeue_tx_one+6a () > 2016-09-19T11:49:39.065149+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689a20 vnd:vnd_squeue_tx_drain+112 () > 2016-09-19T11:49:39.065164+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689ac0 vnd:vnd_squeue_tx_append+103 () > 2016-09-19T11:49:39.065170+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689b50 ip:squeue_enter+41c () > 2016-09-19T11:49:39.065175+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689ba0 gsqueue:gsqueue_enter_one+43 () > 2016-09-19T11:49:39.065179+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689c40 vnd:vnd_frameio_write+10e () > 2016-09-19T11:49:39.065184+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689cc0 vnd:vnd_ioctl+270 () > 2016-09-19T11:49:39.065196+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689d00 genunix:cdev_ioctl+39 () > 2016-09-19T11:49:39.065202+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689d50 specfs:spec_ioctl+60 () > 2016-09-19T11:49:39.065208+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689de0 genunix:fop_ioctl+55 () > 2016-09-19T11:49:39.065213+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689f00 genunix:ioctl+9b () > 2016-09-19T11:49:39.065218+00:00 c1a genunix: [ID 655072 kern.notice] > ffffff00ba689f10 unix:brand_sys_syscall+238 () > 2016-09-19T11:49:39.065224+00:00 c1a unix: [ID 100000 kern.notice] > > > Thanks > Len > > On Mon, 19 Sep 2016 at 09:34 Zak McGregor <z...@cloudafrica.net > <mailto:z...@cloudafrica.net>> wrote: > Hi Brian > > Thanks, here's a full stack trace. > > Ciao > > Zak > > On 18 September 2016 at 21:58, Brian Bennett <brian.benn...@joyent.com > <mailto:brian.benn...@joyent.com>> wrote: > > Zak, > > > > Considering that illumos #3917 is three years old, you've probably hit a > > different bug involving mutexes. It would be best if you can give the full > > stack trace, not just the top two frames. Having the full stack trace, I > > may be able to identify the particular crash you encountered. > > > > Thanks. > > > > -- > > Brian Bennett > > Systems Engineer, Cloud Operations > > Joyent, Inc. | www.joyent.com <http://www.joyent.com/> > > > >> On Sep 16, 2016, at 4:04 AM, Zak McGregor <z...@cloudafrica.net > >> <mailto:z...@cloudafrica.net>> wrote: > >> > >> Hi > >> > >> This issue here: > >> https://illumos.org/issues/3917 <https://illumos.org/issues/3917> > >> > >> seems to have hit one of our production boxes today. I took a look at > >> the dump and it seems to tally precisely with this issue. > >> > >> Here's a snippet: > >> > >> mdb -k unix.0 vmcore.0 > >> Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc > >> pcplusmp scsi_vhci ufs ip hook neti sockfs arp usba uhci mm stmf_sbd > >> stmf zfs lofs idm crypto random cpc logindmux ptm kvm sd sppp nsmb > >> smbsrv nfs ipc ] > >>> ::status > >> debugging crash dump vmcore.0 (64-bit) from c7.jhb.cloudafrica.net > >> <http://c7.jhb.cloudafrica.net/> > >> operating system: 5.11 joyent_20160330T234717Z (i86pc) > >> image uuid: (not set) > >> panic message: turnstile_block(ffffff3d8b4db508): unowned mutex > >> dump content: kernel pages only > >>> ::stack > >> vpanic() > >> turnstile_block+0x78a(0, 0, ffffff3e13c64d00, fffffffffbc07ac0, 0, 0) > >> mutex_vector_enter+0x3a3(ffffff3e13c64d00) > >> > >> If there is any further information you'd like please let me know. > >> > >> Thanks > >> > >> Cheers > >> > >> Zak > >> > > > > > > smartos-discuss | Archives > <https://www.listbox.com/member/archive/184463/=now> > <https://www.listbox.com/member/archive/rss/184463/26986985-d0246faa> | > Modify <https://www.listbox.com/member/?&;> Your Subscription > <http://www.listbox.com/>
smime.p7s
Description: S/MIME cryptographic signature
------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
