Re: [OpenIndiana-discuss] bad mutex crash?
Rich fixed the account for me (thanks Rich!) -Original Message- From: Jason Matthews [mailto:ja...@broken.net] Sent: Wednesday, October 31, 2012 5:01 AM To: 'Discussion list for OpenIndiana' Subject: Re: [OpenIndiana-discuss] bad mutex crash? I will take a shot in the dark here. I imagine the request will time out and be deleted in a few days. Then you could try again -- but I am just speculating. If Jesus was a web developer, that's how he would do it. j. -Original Message- From: Dan Swartzendruber [mailto:dswa...@druber.com] Sent: Tuesday, October 30, 2012 9:56 PM To: 'Discussion list for OpenIndiana' Subject: Re: [OpenIndiana-discuss] bad mutex crash? Wonderful fail on the illumos site. I went to the page to sign up to submit issues. When the activation mail arrived, somehow I deleted it by mistake. Guess what? Apparently there is no way to request a new one. I *can* click on the link to change my password, but that doesn't help because the account has not been activated, so I'm stuck. Fail 2 is the apparent lack of any email address (contact us) anywhere I can see on the top-level page. Seriously? I guess I will now create an email alias and a different user name and see if *that* works. Sigh... ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
Well, definitely ticket time. I seem to be able to make this happen at will. Shut down once via ESXi ACPI and once via 'init 6'. Kernel panic both times... -Original Message- From: Richard Lowe [mailto:richl...@richlowe.net] Sent: Monday, October 29, 2012 10:13 PM To: Discussion list for OpenIndiana Subject: Re: [OpenIndiana-discuss] bad mutex crash? Ruled out what I was wondering. I'd file an illumos bug describing the configuration, etc, and make the crashdump available somehow. -- Rich ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
Wonderful fail on the illumos site. I went to the page to sign up to submit issues. When the activation mail arrived, somehow I deleted it by mistake. Guess what? Apparently there is no way to request a new one. I *can* click on the link to change my password, but that doesn't help because the account has not been activated, so I'm stuck. Fail 2 is the apparent lack of any email address (contact us) anywhere I can see on the top-level page. Seriously? I guess I will now create an email alias and a different user name and see if *that* works. Sigh... ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
2012-10-28 23:55, Dan Swartzendruber wrote: fill). I guess I can create a 32GB virtual disk and move the rpool to it, but it seems like kind of a waste just for the off chance I get a kernel panic. Are there any ways to dodge this, or do I need to just bite the bullet? I think you can configure dumpadm to use another pool for dumps. You can certainly use another pool for swap area, and free up space on rpool - if rpool/dump must indeed be on rpool. Quite likely, you can also expand (live or offline) your VM's harddisk and then autoexpand your rpool, though that would likely require merging and destruction of older VM (hypervizor's) snapshots. HTH, //Jim ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
Oh, anyway, no VM snapshots for this VM anyway, since it has a passed-in HBA therefore VM snapshots are disabled... -Original Message- From: Jim Klimov [mailto:jimkli...@cos.ru] Sent: Monday, October 29, 2012 5:33 PM To: openindiana-discuss@openindiana.org Subject: Re: [OpenIndiana-discuss] bad mutex crash? 2012-10-28 23:55, Dan Swartzendruber wrote: fill). I guess I can create a 32GB virtual disk and move the rpool to it, but it seems like kind of a waste just for the off chance I get a kernel panic. Are there any ways to dodge this, or do I need to just bite the bullet? I think you can configure dumpadm to use another pool for dumps. You can certainly use another pool for swap area, and free up space on rpool - if rpool/dump must indeed be on rpool. Quite likely, you can also expand (live or offline) your VM's harddisk and then autoexpand your rpool, though that would likely require merging and destruction of older VM (hypervizor's) snapshots. HTH, //Jim ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
Ruled out what I was wondering. I'd file an illumos bug describing the configuration, etc, and make the crashdump available somehow. -- Rich ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
Okay thanks. -Original Message- From: Richard Lowe [mailto:richl...@richlowe.net] Sent: Monday, October 29, 2012 10:13 PM To: Discussion list for OpenIndiana Subject: Re: [OpenIndiana-discuss] bad mutex crash? Ruled out what I was wondering. I'd file an illumos bug describing the configuration, etc, and make the crashdump available somehow. -- Rich ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
[OpenIndiana-discuss] bad mutex crash?
I've got a virtualized OI151a7 running under ESXi 5.1. A small root pool VM disk, and two SATA disks passed as RDMs. I wanted to back it up to amazon glacier, so I removed the two RDMs before doing the backup. The problem happened when I was shutting down the OI VM to be able to edit the config for it. I clicked on the shutdown VM option in the vsphere menu for the VM and almost instantly saw the panic. I believe ESXi does this kind of stuff under the sheets using the vmware tools to simulate an ACPI shutdown. I do have a crash dump if anyone wants to look at it. The fmdump info: root@nas2:~# fmdump -Vp -u e78c0b4e-63ee-ef79-b964-974ca7eca6a0 TIME UUID SUNW-MSG-ID Oct 28 2012 14:34:13.78610 e78c0b4e-63ee-ef79-b964-974ca7eca6a0 SUNOS-8000-KL TIME CLASS ENA Oct 28 14:34:03.7006 ireport.os.sunos.panic.dump_pending_on_device 0x nvlist version: 0 version = 0x0 class = list.suspect uuid = e78c0b4e-63ee-ef79-b964-974ca7eca6a0 code = SUNOS-8000-KL diag-time = 1351449253 766125 de = fmd:///module/software-diagnosis fault-list-sz = 0x1 fault-list = (array of embedded nvlists) (start fault-list[0]) nvlist version: 0 version = 0x0 class = defect.sunos.kernel.panic certainty = 0x64 asru = sw:///:path=/var/crash/openindiana2/.e78c0b4e-63ee-ef79-b964-974ca7eca6a0 resource = sw:///:path=/var/crash/openindiana2/.e78c0b4e-63ee-ef79-b964-974ca7eca6a0 savecore-succcess = 0 os-instance-uuid = e78c0b4e-63ee-ef79-b964-974ca7eca6a0 panicstr = mutex_enter: bad mutex, lp=c0391e78 owner=ff019cb008a0 thread=ff00072cac40 panicstack = unix:mutex_panic+73 () | unix:mutex_vector_enter+446 () | genunix:cv_timedwait_hires+fd () | genunix:cv_timedwait_sig_hires+336 () | genunix:cv_timedwait_sig+4c () | f7f9a9c5 () | unix:thread_start+8 () | crashtime = 1351448035 panic-time = October 28, 2012 02:13:55 PM EDT EDT (end fault-list[0]) fault-status = 0x1 severity = Major __ttl = 0x1 __tod = 0x508d7aa5 0x2edaef20 ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
If you save the dump (savecore), extract it (savecore -vf vmdump.0 -d .), and run mdb (mdb unix.0 vmcore.0) What does: f7f9a9c5::whatis say? The stack looks reasonable enough that I'm wondering if you have a module loaded someone's stripped that's actually at fault, rather than it being our fault and a damage stack. If it says it's a 3rd party module, I'd suggest reporting a bug to wherever it came from. Otherwise, please file an illumos bug and make the crash dump available (over http ideally)? -- Rich ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
Hmmm. Doing that yields: root@nas2:~# savecore -vf vmdump.0 -d . savecore: stat(vmdump.0): No such file or directory savecore: open(vmdump.0): No such file or directory -Original Message- From: Richard Lowe [mailto:richl...@richlowe.net] Sent: Sunday, October 28, 2012 2:51 PM To: Discussion list for OpenIndiana Subject: Re: [OpenIndiana-discuss] bad mutex crash? If you save the dump (savecore), extract it (savecore -vf vmdump.0 -d .), and run mdb (mdb unix.0 vmcore.0) What does: f7f9a9c5::whatis say? The stack looks reasonable enough that I'm wondering if you have a module loaded someone's stripped that's actually at fault, rather than it being our fault and a damage stack. If it says it's a 3rd party module, I'd suggest reporting a bug to wherever it came from. Otherwise, please file an illumos bug and make the crash dump available (over http ideally)? -- Rich ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
I note from the fmdump output references to /var/crash, but there is no such directory on my system? ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
Sorry, you have to run savecore twice. Once: # savecore should write out the vmdump.0 The second # savecore -vf vmdump.0 Will extract it. -- Rich ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
Ah, thanks. I did in fact need to create /var/crash/openindiana2 (the hostname is openindiana2). root@nas2:/var/crash/openindiana2# mdb unix.0 vmcore.0 Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci zfs sd mpt ip hook neti sockfs arp usba stmf stmf_sbd fctl md lofs random idm nfs crypto cpc fcp fcip ufs logindmux ptm sppp nsmb smbsrv ] f7f9a9c5::whatis f7f9a9c5 is f7f9a000+9c5, freed from the heaptext vmem arena: ADDR TYPESTART END SIZE THREADTIMESTAMP ff0194e9e930 FREE f7f9a000 f7f9d00012288 (doesn't look real useful?) -Original Message- From: Richard Lowe [mailto:richl...@richlowe.net] Sent: Sunday, October 28, 2012 3:20 PM To: Discussion list for OpenIndiana Subject: Re: [OpenIndiana-discuss] bad mutex crash? Sorry, you have to run savecore twice. Once: # savecore should write out the vmdump.0 The second # savecore -vf vmdump.0 Will extract it. -- Rich ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] bad mutex crash?
On a related topic: this is my backup OI VM. It mainly just receives 5-minute snaps to a replicate of the main ESXi datastore share. Both VMs have fairly small root disks, but I noticed I didn't have a rpool/dump on the main OI VM, so I went to create it and dumpadm bitched me out for having insufficient space. The reason is that the backup OI has 8GB ram (which is fine, since it is 'write only'), but the main OI has 20GB, since I want a decent hit in ARC (as well as enough RAM to allow the two 128GB SSDs to fill). I guess I can create a 32GB virtual disk and move the rpool to it, but it seems like kind of a waste just for the off chance I get a kernel panic. Are there any ways to dodge this, or do I need to just bite the bullet? ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
