On 1/6/17 4:28 , Adam Richmond-Gordon wrote: > Hello, > > I have opened a GitHub issue for this; > https://github.com/joyent/illumos-joyent/issues/135 > <https://github.com/joyent/illumos-joyent/issues/135>
Hi Adam, Thanks for following up there. If you can contact me privately, I'll follow up and grab the dump so we can get some eyes on it at Joyent. Thanks, Robert >> On 5 Jan 2017, at 17:16, Adam Richmond-Gordon <[email protected]> >> wrote: >> >> Thanks Robert - I’ll get it uploaded and create an issue. >> >> The dump device size is set to 256GB (equal to RAM) - I’ve been gradually >> increasing it after every dumpless crash. I’d been considering connecting a >> USB disk and pointing dumpadm at that. >> >>> On 5 Jan 2017, at 16:52, Robert Mustacchi <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> On 1/5/17 6:55 , Adam Richmond-Gordon wrote: >>>> Afternoon! >>>> >>>> I’ve been trying to diagnose a box that’s been crashing after anywhere >>>> between 3 and 30 days uptime. For a little while, I’d suspected the issue >>>> might be storage related, as the crashes never created a dump, but sending >>>> an NMI always did. >>>> >>>> Today, the box actually created a dump, and on first investigation, it >>>> appears that the mpt_sas driver may be involved. Maybe. The last message >>>> in the buffer points to the PCI address that the SAS controller occupies, >>>> but a lot of the messages before that appear to be related to KVM. >>>> >>>> If anyone has the time, I’d really appreciate some pointers on narrowing >>>> this down. I’ve not raised an issue on GitHub yet, because this could >>>> easily be a hardware-related issue. >>> >>> The stack that you have is when exiting a mutex something incorrect >>> happen. We tried to unlock the lock to something we don't have. If you >>> can upload the dump somewhere that folks can dig into it, that'd be >>> helpful. If you need help doing so or need a location to put it, please >>> let me know. I'd also create a github ticket for that. >>> >>> For the crashes not creating a dump, I'd double check your dump device size. >>> >>> Robert >>> >>>> Through some poking around at the end of last year, I have also noticed >>>> that the onboard SAS controller (LSI/Avago 3008) isn’t running the >>>> firmware we specified to the VAR - not sure if this is likely to make a >>>> difference, but it’s easily flashed. It is currently running the IR >>>> firmware on and older phase, when it should really be running the recent >>>> IT firmware. >>>> >>>> Here are the potentially interesting bits from the dump; >>>> >>>>> ::status >>>> debugging crash dump vmcore.0 (64-bit) from bri-triw-001 >>>> operating system: 5.11 joyent_20161208T003358Z (i86pc) >>>> image uuid: (not set) >>>> panic message: >>>> mutex_exit: not owner, lp=fffffea460ba2020 owner=fffffea3b90aa460 >>>> thread=fffffea3f40b1be0 >>>> dump content: kernel pages only >>>> >>>>> $C >>>> fffffe426a808fe8 vpanic() >>>> fffffe426a809008 mutex_panic+0x58(fffffffffb94dc45, fffffea460ba2020) >>>> fffffe426a809038 mutex_vector_exit+0x40(fffffea460ba2020) >>>> fffffe426a809070 gfn_to_memslot_unaliased+0x6f() >>>> fffffe426a809090 gfn_to_hva+0x27() >>>> fffffe426a8090c0 kvm_read_guest_page+0x29() >>>> fffffe426a809110 kvm_read_guest+0x43() >>>> fffffe426a809190 paging64_walk_addr+0xef() >>>> fffffe426a809230 paging64_gva_to_gpa+0x43() >>>> fffffe426a809260 kvm_mmu_gva_to_gpa_read+0x45() >>>> fffffe426a8092b0 emulator_read_emulated+0x7c() >>>> fffffe426a809350 x86_emulate_insn+0x1af() >>>> fffffe426a809390 emulate_instruction+0x1e9() >>>> fffffe426a8093c0 kvm_mmu_page_fault+0x60() >>>> fffffe426a8093f0 handle_ept_violation+0x111() >>>> fffffe426a809430 vmx_handle_exit+0x16a() >>>> fffffe426a809460 vcpu_enter_guest+0x3ea() >>>> fffffe426a8094a0 __vcpu_run+0x8b() >>>> fffffe426a8094e0 kvm_arch_vcpu_ioctl_run+0x112() >>>> fffffe426a809cc0 kvm_ioctl+0x466() >>>> fffffe426a809d00 cdev_ioctl+0x39(3400000068, 2000ae80, 0, 202003, >>>> fffffea3b7d370d0, >>>> fffffe426a809ea8) >>>> fffffe426a809d50 spec_ioctl+0x60(fffffea4440b5040, 2000ae80, 0, 202003, >>>> fffffea3b7d370d0, >>>> fffffe426a809ea8, 0) >>>> fffffe426a809de0 fop_ioctl+0x55(fffffea4440b5040, 2000ae80, 0, 202003, >>>> fffffea3b7d370d0, >>>> fffffe426a809ea8, 0) >>>> fffffe426a809f00 ioctl+0x9b(16, 2000ae80, 0) >>>> fffffe426a809f10 sys_syscall+0x1a2() >>>> >>>>> ::msgbuf >>>> MESSAGE >>>> vcpu 3 received sipi with vector # 10 >>>> vcpu 5 received sipi with vector # 10 >>>> kvm_lapic_reset: vcpu=fffffea4434cf000, id=3, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea4434ff000, id=5, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 9 received sipi with vector # 10 >>>> vcpu 10 received sipi with vector # 10 >>>> kvm_lapic_reset: vcpu=fffffea44354f000, id=9, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 4 received sipi with vector # 10 >>>> kvm_lapic_reset: vcpu=fffffea493522000, id=10, base_msr= fee00800 PRIx64 >>>> base_address=fee0 >>>> 0000 >>>> vcpu 6 received sipi with vector # 10 >>>> vcpu 7 received sipi with vector # 10 >>>> kvm_lapic_reset: vcpu=fffffea44348f000, id=1, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 8 received sipi with vector # 10 >>>> vcpu 2 received sipi with vector # 10 >>>> vcpu 11 received sipi with vector # 10 >>>> kvm_lapic_reset: vcpu=fffffea4434bf000, id=4, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea44352f000, id=6, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea443517000, id=7, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea443557000, id=8, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea44347f000, id=2, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea49351a000, id=11, base_msr= fee00800 PRIx64 >>>> base_address=fee0 >>>> 0000 >>>> unhandled wrmsr: 0x0 data 0 >>>> vcpu 1 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea44348f000, id=1, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 2 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea44347f000, id=2, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 3 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea4434cf000, id=3, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 4 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea4434bf000, id=4, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 5 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea4434ff000, id=5, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 6 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea44352f000, id=6, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 7 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea443517000, id=7, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 8 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea443557000, id=8, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 9 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea44354f000, id=9, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 10 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea493522000, id=10, base_msr= fee00800 PRIx64 >>>> base_address=fee0 >>>> 0000 >>>> vcpu 11 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea49351a000, id=11, base_msr= fee00800 PRIx64 >>>> base_address=fee0 >>>> 0000 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720 >>>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720 >>>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720 >>>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720 >>>> vcpu 1 received sipi with vector # 10 >>>> vcpu 3 received sipi with vector # 10 >>>> vcpu 2 received sipi with vector # 10 >>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> unhandled wrmsr: 0x0 data 0 >>>> vcpu 1 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 2 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 3 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720 >>>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720 >>>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720 >>>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720 >>>> vcpu 2 received sipi with vector # 10 >>>> vcpu 1 received sipi with vector # 10 >>>> vcpu 3 received sipi with vector # 10 >>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> unhandled wrmsr: 0x0 data 0 >>>> vcpu 1 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 2 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 3 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x0 data 0 >>>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720 >>>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720 >>>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720 >>>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720 >>>> vcpu 3 received sipi with vector # 10 >>>> vcpu 1 received sipi with vector # 10 >>>> vcpu 2 received sipi with vector # 10 >>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> unhandled wrmsr: 0x0 data 0 >>>> vcpu 1 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 2 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> vcpu 3 received sipi with vector # 1 >>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64 >>>> base_address=fee00 >>>> 000 >>>> >>>> panic[cpu20]/thread=fffffea3f40b1be0: >>>> mutex_exit: not owner, lp=fffffea460ba2020 owner=fffffea3b90aa460 >>>> thread=fffffea3f40b1be0 >>>> >>>> >>>>>> warning! 8-byte aligned %fp = fffffe426a809008 >>>> fffffe426a809008 unix:mutex_panic+58 () >>>>>> warning! 8-byte aligned %fp = fffffe426a809038 >>>> fffffe426a809038 unix:mutex_vector_exit+40 () >>>> fffffe426a809070 kvm:gfn_to_memslot_unaliased+6f () >>>> fffffe426a809090 kvm:gfn_to_hva+27 () >>>> fffffe426a8090c0 kvm:kvm_read_guest_page+29 () >>>> fffffe426a809110 kvm:kvm_read_guest+43 () >>>> fffffe426a809190 kvm:paging64_walk_addr+ef () >>>> fffffe426a809230 kvm:paging64_gva_to_gpa+43 () >>>> fffffe426a809260 kvm:kvm_mmu_gva_to_gpa_read+45 () >>>> fffffe426a8092b0 kvm:emulator_read_emulated+7c () >>>> fffffe426a809350 kvm:x86_emulate_insn+1af () >>>> fffffe426a809390 kvm:emulate_instruction+1e9 () >>>> fffffe426a8093c0 kvm:kvm_mmu_page_fault+60 () >>>> fffffe426a8093f0 kvm:handle_ept_violation+111 () >>>> fffffe426a809430 kvm:vmx_handle_exit+16a () >>>> fffffe426a809460 kvm:vcpu_enter_guest+3ea () >>>> fffffe426a8094a0 kvm:__vcpu_run+8b () >>>> fffffe426a8094e0 kvm:kvm_arch_vcpu_ioctl_run+112 () >>>> fffffe426a809cc0 kvm:kvm_ioctl+466 () >>>> fffffe426a809d00 genunix:cdev_ioctl+39 () >>>> fffffe426a809d50 specfs:spec_ioctl+60 () >>>> fffffe426a809de0 genunix:fop_ioctl+55 () >>>> fffffe426a809f00 genunix:ioctl+9b () >>>> fffffe426a809f10 unix:brand_sys_syscall+21d () >>>> >>>> WARNING: /pci@0,0/pci8086,6f02@1/pci15d9,808@0 (mpt_sas0): >>>> mptsas_handle_event_sync: event 0xf, IOCStatus=0x8000, >>>> IOCLogInfo=0x31170000 >>>> /pci@0,0/pci8086,6f02@1/pci15d9,808@0 (mpt_sas0): >>>> Log info 0x31170000 received for target 16 w5000cca07d0109a9. >>>> scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc >>>> >>> >>> >> smartos-discuss | Archives >> <https://www.listbox.com/member/archive/184463/=now> >> <https://www.listbox.com/member/archive/rss/184463/28443474-1732e24d> | >> Modify <https://www.listbox.com/member/?&;> Your Subscription > > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
