Hi Brian,
Any feedback on this? Not sure if you've had a chance to look? Kind regards, Angelo. From: Brian Bennett [mailto:[email protected]] Sent: Tuesday, 20 September 2016 20:21 To: [email protected] Subject: Re: [smartos-discuss] Re-occurrence of bug 3917 Len or Zak, would either of you be able to provide us with a crash dump? It should be in /var/crash/volatile on any CN that had this panic occur. -- Brian Bennett Systems Engineer, Cloud Operations Joyent, Inc. | www.joyent.com <http://www.joyent.com> On Sep 19, 2016, at 5:09 AM, Len Weincier <[email protected] <mailto:[email protected]> > wrote: Hi This has now happened to 3 hosts in the last 3 days. Any idea what we can look at ? It seems to happen under high load on those systems, all older E5 based hosts. We just had another reboot and this is in /var/adm/messages 2016-09-19T11:49:39.065057+00:00 c1a unix: [ID 836849 kern.notice] #012#015panic[cpu14]/thread=ffffff19e4687420: 2016-09-19T11:49:39.065068+00:00 c1a genunix: [ID 761616 kern.notice] turnstile_block(ffffff19ab6e9230): unowned mutex 2016-09-19T11:49:39.065074+00:00 c1a unix: [ID 100000 kern.notice] #012 2016-09-19T11:49:39.065079+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689380 genunix:turnstile_block+78a () 2016-09-19T11:49:39.065084+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba6893f0 unix:mutex_vector_enter+3a3 () 2016-09-19T11:49:39.065089+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba6894c0 vnd:vnd_mac_input+12a () 2016-09-19T11:49:39.065094+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689580 dls:dls_rx_promisc+119 () 2016-09-19T11:49:39.065099+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba6895e0 mac:mac_promisc_dispatch_one+81 () 2016-09-19T11:49:39.065104+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689660 mac:mac_promisc_dispatch+b2 () 2016-09-19T11:49:39.065109+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689750 mac:mac_tx_send+33f () 2016-09-19T11:49:39.065114+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba6897f0 mac:mac_tx_single_ring_mode+6e () 2016-09-19T11:49:39.065132+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba6898a0 mac:mac_tx+da () 2016-09-19T11:49:39.065139+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689950 dld:str_mdata_raw_fastpath_put+85 () 2016-09-19T11:49:39.065144+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689990 vnd:vnd_squeue_tx_one+6a () 2016-09-19T11:49:39.065149+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689a20 vnd:vnd_squeue_tx_drain+112 () 2016-09-19T11:49:39.065164+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689ac0 vnd:vnd_squeue_tx_append+103 () 2016-09-19T11:49:39.065170+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689b50 ip:squeue_enter+41c () 2016-09-19T11:49:39.065175+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689ba0 gsqueue:gsqueue_enter_one+43 () 2016-09-19T11:49:39.065179+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689c40 vnd:vnd_frameio_write+10e () 2016-09-19T11:49:39.065184+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689cc0 vnd:vnd_ioctl+270 () 2016-09-19T11:49:39.065196+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689d00 genunix:cdev_ioctl+39 () 2016-09-19T11:49:39.065202+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689d50 specfs:spec_ioctl+60 () 2016-09-19T11:49:39.065208+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689de0 genunix:fop_ioctl+55 () 2016-09-19T11:49:39.065213+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689f00 genunix:ioctl+9b () 2016-09-19T11:49:39.065218+00:00 c1a genunix: [ID 655072 kern.notice] ffffff00ba689f10 unix:brand_sys_syscall+238 () 2016-09-19T11:49:39.065224+00:00 c1a unix: [ID 100000 kern.notice] Thanks Len On Mon, 19 Sep 2016 at 09:34 Zak McGregor <[email protected] <mailto:[email protected]> > wrote: Hi Brian Thanks, here's a full stack trace. Ciao Zak On 18 September 2016 at 21:58, Brian Bennett <[email protected] <mailto:[email protected]> > wrote: > Zak, > > Considering that illumos #3917 is three years old, you've probably hit a different bug involving mutexes. It would be best if you can give the full stack trace, not just the top two frames. Having the full stack trace, I may be able to identify the particular crash you encountered. > > Thanks. > > -- > Brian Bennett > Systems Engineer, Cloud Operations > Joyent, Inc. | www.joyent.com <http://www.joyent.com/> > >> On Sep 16, 2016, at 4:04 AM, Zak McGregor <[email protected] <mailto:[email protected]> > wrote: >> >> Hi >> >> This issue here: >> https://illumos.org/issues/3917 >> >> seems to have hit one of our production boxes today. I took a look at >> the dump and it seems to tally precisely with this issue. >> >> Here's a snippet: >> >> mdb -k unix.0 vmcore.0 >> Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc >> pcplusmp scsi_vhci ufs ip hook neti sockfs arp usba uhci mm stmf_sbd >> stmf zfs lofs idm crypto random cpc logindmux ptm kvm sd sppp nsmb >> smbsrv nfs ipc ] >>> ::status >> debugging crash dump vmcore.0 (64-bit) from c7.jhb.cloudafrica.net <http://c7.jhb.cloudafrica.net/> >> operating system: 5.11 joyent_20160330T234717Z (i86pc) >> image uuid: (not set) >> panic message: turnstile_block(ffffff3d8b4db508): unowned mutex >> dump content: kernel pages only >>> ::stack >> vpanic() >> turnstile_block+0x78a(0, 0, ffffff3e13c64d00, fffffffffbc07ac0, 0, 0) >> mutex_vector_enter+0x3a3(ffffff3e13c64d00) >> >> If there is any further information you'd like please let me know. >> >> Thanks >> >> Cheers >> >> Zak >> > > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
