Hello, I have opened a GitHub issue for this; https://github.com/joyent/illumos-joyent/issues/135 <https://github.com/joyent/illumos-joyent/issues/135>
Adam > On 5 Jan 2017, at 17:16, Adam Richmond-Gordon <[email protected]> > wrote: > > Thanks Robert - I’ll get it uploaded and create an issue. > > The dump device size is set to 256GB (equal to RAM) - I’ve been gradually > increasing it after every dumpless crash. I’d been considering connecting a > USB disk and pointing dumpadm at that. > >> On 5 Jan 2017, at 16:52, Robert Mustacchi <[email protected] >> <mailto:[email protected]>> wrote: >> >> On 1/5/17 6:55 , Adam Richmond-Gordon wrote: >>> Afternoon! >>> >>> I’ve been trying to diagnose a box that’s been crashing after anywhere >>> between 3 and 30 days uptime. For a little while, I’d suspected the issue >>> might be storage related, as the crashes never created a dump, but sending >>> an NMI always did. >>> >>> Today, the box actually created a dump, and on first investigation, it >>> appears that the mpt_sas driver may be involved. Maybe. The last message in >>> the buffer points to the PCI address that the SAS controller occupies, but >>> a lot of the messages before that appear to be related to KVM. >>> >>> If anyone has the time, I’d really appreciate some pointers on narrowing >>> this down. I’ve not raised an issue on GitHub yet, because this could >>> easily be a hardware-related issue. >> >> The stack that you have is when exiting a mutex something incorrect >> happen. We tried to unlock the lock to something we don't have. If you >> can upload the dump somewhere that folks can dig into it, that'd be >> helpful. If you need help doing so or need a location to put it, please >> let me know. I'd also create a github ticket for that. >> >> For the crashes not creating a dump, I'd double check your dump device size. >> >> Robert >> >>> Through some poking around at the end of last year, I have also noticed >>> that the onboard SAS controller (LSI/Avago 3008) isn’t running the firmware >>> we specified to the VAR - not sure if this is likely to make a difference, >>> but it’s easily flashed. It is currently running the IR firmware on and >>> older phase, when it should really be running the recent IT firmware. >>> >>> Here are the potentially interesting bits from the dump; >>> >>>> ::status >>> debugging crash dump vmcore.0 (64-bit) from bri-triw-001 >>> operating system: 5.11 joyent_20161208T003358Z (i86pc) >>> image uuid: (not set) >>> panic message: >>> mutex_exit: not owner, lp=fffffea460ba2020 owner=fffffea3b90aa460 >>> thread=fffffea3f40b1be0 >>> dump content: kernel pages only >>> >>>> $C >>> fffffe426a808fe8 vpanic() >>> fffffe426a809008 mutex_panic+0x58(fffffffffb94dc45, fffffea460ba2020) >>> fffffe426a809038 mutex_vector_exit+0x40(fffffea460ba2020) >>> fffffe426a809070 gfn_to_memslot_unaliased+0x6f() >>> fffffe426a809090 gfn_to_hva+0x27() >>> fffffe426a8090c0 kvm_read_guest_page+0x29() >>> fffffe426a809110 kvm_read_guest+0x43() >>> fffffe426a809190 paging64_walk_addr+0xef() >>> fffffe426a809230 paging64_gva_to_gpa+0x43() >>> fffffe426a809260 kvm_mmu_gva_to_gpa_read+0x45() >>> fffffe426a8092b0 emulator_read_emulated+0x7c() >>> fffffe426a809350 x86_emulate_insn+0x1af() >>> fffffe426a809390 emulate_instruction+0x1e9() >>> fffffe426a8093c0 kvm_mmu_page_fault+0x60() >>> fffffe426a8093f0 handle_ept_violation+0x111() >>> fffffe426a809430 vmx_handle_exit+0x16a() >>> fffffe426a809460 vcpu_enter_guest+0x3ea() >>> fffffe426a8094a0 __vcpu_run+0x8b() >>> fffffe426a8094e0 kvm_arch_vcpu_ioctl_run+0x112() >>> fffffe426a809cc0 kvm_ioctl+0x466() >>> fffffe426a809d00 cdev_ioctl+0x39(3400000068, 2000ae80, 0, 202003, >>> fffffea3b7d370d0, >>> fffffe426a809ea8) >>> fffffe426a809d50 spec_ioctl+0x60(fffffea4440b5040, 2000ae80, 0, 202003, >>> fffffea3b7d370d0, >>> fffffe426a809ea8, 0) >>> fffffe426a809de0 fop_ioctl+0x55(fffffea4440b5040, 2000ae80, 0, 202003, >>> fffffea3b7d370d0, >>> fffffe426a809ea8, 0) >>> fffffe426a809f00 ioctl+0x9b(16, 2000ae80, 0) >>> fffffe426a809f10 sys_syscall+0x1a2() >>> >>>> ::msgbuf >>> MESSAGE >>> vcpu 3 received sipi with vector # 10 >>> vcpu 5 received sipi with vector # 10 >>> kvm_lapic_reset: vcpu=fffffea4434cf000, id=3, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea4434ff000, id=5, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 9 received sipi with vector # 10 >>> vcpu 10 received sipi with vector # 10 >>> kvm_lapic_reset: vcpu=fffffea44354f000, id=9, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 4 received sipi with vector # 10 >>> kvm_lapic_reset: vcpu=fffffea493522000, id=10, base_msr= fee00800 PRIx64 >>> base_address=fee0 >>> 0000 >>> vcpu 6 received sipi with vector # 10 >>> vcpu 7 received sipi with vector # 10 >>> kvm_lapic_reset: vcpu=fffffea44348f000, id=1, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 8 received sipi with vector # 10 >>> vcpu 2 received sipi with vector # 10 >>> vcpu 11 received sipi with vector # 10 >>> kvm_lapic_reset: vcpu=fffffea4434bf000, id=4, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea44352f000, id=6, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea443517000, id=7, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea443557000, id=8, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea44347f000, id=2, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea49351a000, id=11, base_msr= fee00800 PRIx64 >>> base_address=fee0 >>> 0000 >>> unhandled wrmsr: 0x0 data 0 >>> vcpu 1 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea44348f000, id=1, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 2 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea44347f000, id=2, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 3 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea4434cf000, id=3, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 4 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea4434bf000, id=4, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 5 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea4434ff000, id=5, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 6 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea44352f000, id=6, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 7 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea443517000, id=7, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 8 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea443557000, id=8, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 9 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea44354f000, id=9, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 10 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea493522000, id=10, base_msr= fee00800 PRIx64 >>> base_address=fee0 >>> 0000 >>> vcpu 11 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea49351a000, id=11, base_msr= fee00800 PRIx64 >>> base_address=fee0 >>> 0000 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720 >>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720 >>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720 >>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720 >>> vcpu 1 received sipi with vector # 10 >>> vcpu 3 received sipi with vector # 10 >>> vcpu 2 received sipi with vector # 10 >>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> unhandled wrmsr: 0x0 data 0 >>> vcpu 1 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 2 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 3 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720 >>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720 >>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720 >>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720 >>> vcpu 2 received sipi with vector # 10 >>> vcpu 1 received sipi with vector # 10 >>> vcpu 3 received sipi with vector # 10 >>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> unhandled wrmsr: 0x0 data 0 >>> vcpu 1 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 2 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 3 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x0 data 0 >>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720 >>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720 >>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720 >>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720 >>> vcpu 3 received sipi with vector # 10 >>> vcpu 1 received sipi with vector # 10 >>> vcpu 2 received sipi with vector # 10 >>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> unhandled wrmsr: 0x0 data 0 >>> vcpu 1 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 2 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> vcpu 3 received sipi with vector # 1 >>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>> base_address=fee00 >>> 000 >>> >>> panic[cpu20]/thread=fffffea3f40b1be0: >>> mutex_exit: not owner, lp=fffffea460ba2020 owner=fffffea3b90aa460 >>> thread=fffffea3f40b1be0 >>> >>> >>>>> warning! 8-byte aligned %fp = fffffe426a809008 >>> fffffe426a809008 unix:mutex_panic+58 () >>>>> warning! 8-byte aligned %fp = fffffe426a809038 >>> fffffe426a809038 unix:mutex_vector_exit+40 () >>> fffffe426a809070 kvm:gfn_to_memslot_unaliased+6f () >>> fffffe426a809090 kvm:gfn_to_hva+27 () >>> fffffe426a8090c0 kvm:kvm_read_guest_page+29 () >>> fffffe426a809110 kvm:kvm_read_guest+43 () >>> fffffe426a809190 kvm:paging64_walk_addr+ef () >>> fffffe426a809230 kvm:paging64_gva_to_gpa+43 () >>> fffffe426a809260 kvm:kvm_mmu_gva_to_gpa_read+45 () >>> fffffe426a8092b0 kvm:emulator_read_emulated+7c () >>> fffffe426a809350 kvm:x86_emulate_insn+1af () >>> fffffe426a809390 kvm:emulate_instruction+1e9 () >>> fffffe426a8093c0 kvm:kvm_mmu_page_fault+60 () >>> fffffe426a8093f0 kvm:handle_ept_violation+111 () >>> fffffe426a809430 kvm:vmx_handle_exit+16a () >>> fffffe426a809460 kvm:vcpu_enter_guest+3ea () >>> fffffe426a8094a0 kvm:__vcpu_run+8b () >>> fffffe426a8094e0 kvm:kvm_arch_vcpu_ioctl_run+112 () >>> fffffe426a809cc0 kvm:kvm_ioctl+466 () >>> fffffe426a809d00 genunix:cdev_ioctl+39 () >>> fffffe426a809d50 specfs:spec_ioctl+60 () >>> fffffe426a809de0 genunix:fop_ioctl+55 () >>> fffffe426a809f00 genunix:ioctl+9b () >>> fffffe426a809f10 unix:brand_sys_syscall+21d () >>> >>> WARNING: /pci@0,0/pci8086,6f02@1/pci15d9,808@0 (mpt_sas0): >>> mptsas_handle_event_sync: event 0xf, IOCStatus=0x8000, >>> IOCLogInfo=0x31170000 >>> /pci@0,0/pci8086,6f02@1/pci15d9,808@0 (mpt_sas0): >>> Log info 0x31170000 received for target 16 w5000cca07d0109a9. >>> scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc >>> >> >> > smartos-discuss | Archives > <https://www.listbox.com/member/archive/184463/=now> > <https://www.listbox.com/member/archive/rss/184463/28443474-1732e24d> | > Modify <https://www.listbox.com/member/?&;> Your Subscription ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
