Hi Robert,
I really appreciate you picking this up!
SFTP server:
213.249.128.37
Port:
61205
User:
joyent
Password:
;URe#bVg+|Yn_6(3!5eO~y>Y'E\cU}w@&DN<YxWv6d|eG_do{V<tTL1`q<KoR<C
Pretty sure there’s everything that'sl needed to get started there - I’ve
compressed the dump, it unpacks to about 20GB.
Let me know if you or anyone else needs anything else from me!
Adam
> On 6 Jan 2017, at 22:56, Robert Mustacchi <[email protected]> wrote:
>
> On 1/6/17 4:28 , Adam Richmond-Gordon wrote:
>> Hello,
>>
>> I have opened a GitHub issue for this;
>> https://github.com/joyent/illumos-joyent/issues/135
>> <https://github.com/joyent/illumos-joyent/issues/135>
>
> Hi Adam,
>
> Thanks for following up there. If you can contact me privately, I'll
> follow up and grab the dump so we can get some eyes on it at Joyent.
>
> Thanks,
> Robert
>
>>> On 5 Jan 2017, at 17:16, Adam Richmond-Gordon <[email protected]>
>>> wrote:
>>>
>>> Thanks Robert - I’ll get it uploaded and create an issue.
>>>
>>> The dump device size is set to 256GB (equal to RAM) - I’ve been gradually
>>> increasing it after every dumpless crash. I’d been considering connecting a
>>> USB disk and pointing dumpadm at that.
>>>
>>>> On 5 Jan 2017, at 16:52, Robert Mustacchi <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> On 1/5/17 6:55 , Adam Richmond-Gordon wrote:
>>>>> Afternoon!
>>>>>
>>>>> I’ve been trying to diagnose a box that’s been crashing after anywhere
>>>>> between 3 and 30 days uptime. For a little while, I’d suspected the issue
>>>>> might be storage related, as the crashes never created a dump, but
>>>>> sending an NMI always did.
>>>>>
>>>>> Today, the box actually created a dump, and on first investigation, it
>>>>> appears that the mpt_sas driver may be involved. Maybe. The last message
>>>>> in the buffer points to the PCI address that the SAS controller occupies,
>>>>> but a lot of the messages before that appear to be related to KVM.
>>>>>
>>>>> If anyone has the time, I’d really appreciate some pointers on narrowing
>>>>> this down. I’ve not raised an issue on GitHub yet, because this could
>>>>> easily be a hardware-related issue.
>>>>
>>>> The stack that you have is when exiting a mutex something incorrect
>>>> happen. We tried to unlock the lock to something we don't have. If you
>>>> can upload the dump somewhere that folks can dig into it, that'd be
>>>> helpful. If you need help doing so or need a location to put it, please
>>>> let me know. I'd also create a github ticket for that.
>>>>
>>>> For the crashes not creating a dump, I'd double check your dump device
>>>> size.
>>>>
>>>> Robert
>>>>
>>>>> Through some poking around at the end of last year, I have also noticed
>>>>> that the onboard SAS controller (LSI/Avago 3008) isn’t running the
>>>>> firmware we specified to the VAR - not sure if this is likely to make a
>>>>> difference, but it’s easily flashed. It is currently running the IR
>>>>> firmware on and older phase, when it should really be running the recent
>>>>> IT firmware.
>>>>>
>>>>> Here are the potentially interesting bits from the dump;
>>>>>
>>>>>> ::status
>>>>> debugging crash dump vmcore.0 (64-bit) from bri-triw-001
>>>>> operating system: 5.11 joyent_20161208T003358Z (i86pc)
>>>>> image uuid: (not set)
>>>>> panic message:
>>>>> mutex_exit: not owner, lp=fffffea460ba2020 owner=fffffea3b90aa460
>>>>> thread=fffffea3f40b1be0
>>>>> dump content: kernel pages only
>>>>>
>>>>>> $C
>>>>> fffffe426a808fe8 vpanic()
>>>>> fffffe426a809008 mutex_panic+0x58(fffffffffb94dc45, fffffea460ba2020)
>>>>> fffffe426a809038 mutex_vector_exit+0x40(fffffea460ba2020)
>>>>> fffffe426a809070 gfn_to_memslot_unaliased+0x6f()
>>>>> fffffe426a809090 gfn_to_hva+0x27()
>>>>> fffffe426a8090c0 kvm_read_guest_page+0x29()
>>>>> fffffe426a809110 kvm_read_guest+0x43()
>>>>> fffffe426a809190 paging64_walk_addr+0xef()
>>>>> fffffe426a809230 paging64_gva_to_gpa+0x43()
>>>>> fffffe426a809260 kvm_mmu_gva_to_gpa_read+0x45()
>>>>> fffffe426a8092b0 emulator_read_emulated+0x7c()
>>>>> fffffe426a809350 x86_emulate_insn+0x1af()
>>>>> fffffe426a809390 emulate_instruction+0x1e9()
>>>>> fffffe426a8093c0 kvm_mmu_page_fault+0x60()
>>>>> fffffe426a8093f0 handle_ept_violation+0x111()
>>>>> fffffe426a809430 vmx_handle_exit+0x16a()
>>>>> fffffe426a809460 vcpu_enter_guest+0x3ea()
>>>>> fffffe426a8094a0 __vcpu_run+0x8b()
>>>>> fffffe426a8094e0 kvm_arch_vcpu_ioctl_run+0x112()
>>>>> fffffe426a809cc0 kvm_ioctl+0x466()
>>>>> fffffe426a809d00 cdev_ioctl+0x39(3400000068, 2000ae80, 0, 202003,
>>>>> fffffea3b7d370d0,
>>>>> fffffe426a809ea8)
>>>>> fffffe426a809d50 spec_ioctl+0x60(fffffea4440b5040, 2000ae80, 0, 202003,
>>>>> fffffea3b7d370d0,
>>>>> fffffe426a809ea8, 0)
>>>>> fffffe426a809de0 fop_ioctl+0x55(fffffea4440b5040, 2000ae80, 0, 202003,
>>>>> fffffea3b7d370d0,
>>>>> fffffe426a809ea8, 0)
>>>>> fffffe426a809f00 ioctl+0x9b(16, 2000ae80, 0)
>>>>> fffffe426a809f10 sys_syscall+0x1a2()
>>>>>
>>>>>> ::msgbuf
>>>>> MESSAGE
>>>>> vcpu 3 received sipi with vector # 10
>>>>> vcpu 5 received sipi with vector # 10
>>>>> kvm_lapic_reset: vcpu=fffffea4434cf000, id=3, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea4434ff000, id=5, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 9 received sipi with vector # 10
>>>>> vcpu 10 received sipi with vector # 10
>>>>> kvm_lapic_reset: vcpu=fffffea44354f000, id=9, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 4 received sipi with vector # 10
>>>>> kvm_lapic_reset: vcpu=fffffea493522000, id=10, base_msr= fee00800 PRIx64
>>>>> base_address=fee0
>>>>> 0000
>>>>> vcpu 6 received sipi with vector # 10
>>>>> vcpu 7 received sipi with vector # 10
>>>>> kvm_lapic_reset: vcpu=fffffea44348f000, id=1, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 8 received sipi with vector # 10
>>>>> vcpu 2 received sipi with vector # 10
>>>>> vcpu 11 received sipi with vector # 10
>>>>> kvm_lapic_reset: vcpu=fffffea4434bf000, id=4, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea44352f000, id=6, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea443517000, id=7, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea443557000, id=8, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea44347f000, id=2, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea49351a000, id=11, base_msr= fee00800 PRIx64
>>>>> base_address=fee0
>>>>> 0000
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> vcpu 1 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea44348f000, id=1, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 2 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea44347f000, id=2, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 3 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea4434cf000, id=3, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 4 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea4434bf000, id=4, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 5 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea4434ff000, id=5, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 6 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea44352f000, id=6, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 7 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea443517000, id=7, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 8 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea443557000, id=8, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 9 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea44354f000, id=9, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 10 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea493522000, id=10, base_msr= fee00800 PRIx64
>>>>> base_address=fee0
>>>>> 0000
>>>>> vcpu 11 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea49351a000, id=11, base_msr= fee00800 PRIx64
>>>>> base_address=fee0
>>>>> 0000
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720
>>>>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720
>>>>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720
>>>>> unhandled wrmsr: 0xffbfe0 data fffffd7fffdfe720
>>>>> vcpu 1 received sipi with vector # 10
>>>>> vcpu 3 received sipi with vector # 10
>>>>> vcpu 2 received sipi with vector # 10
>>>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> vcpu 1 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 2 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 3 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720
>>>>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720
>>>>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720
>>>>> unhandled wrmsr: 0x11e4720 data fffffd7fffdfe720
>>>>> vcpu 2 received sipi with vector # 10
>>>>> vcpu 1 received sipi with vector # 10
>>>>> vcpu 3 received sipi with vector # 10
>>>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> vcpu 1 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 2 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 3 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720
>>>>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720
>>>>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720
>>>>> unhandled wrmsr: 0x11e4a20 data fffffd7fffdfe720
>>>>> vcpu 3 received sipi with vector # 10
>>>>> vcpu 1 received sipi with vector # 10
>>>>> vcpu 2 received sipi with vector # 10
>>>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> unhandled wrmsr: 0x0 data 0
>>>>> vcpu 1 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea5ac6e8000, id=1, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 2 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea443527000, id=2, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>> vcpu 3 received sipi with vector # 1
>>>>> kvm_lapic_reset: vcpu=fffffea625738000, id=3, base_msr= fee00800 PRIx64
>>>>> base_address=fee00
>>>>> 000
>>>>>
>>>>> panic[cpu20]/thread=fffffea3f40b1be0:
>>>>> mutex_exit: not owner, lp=fffffea460ba2020 owner=fffffea3b90aa460
>>>>> thread=fffffea3f40b1be0
>>>>>
>>>>>
>>>>>>> warning! 8-byte aligned %fp = fffffe426a809008
>>>>> fffffe426a809008 unix:mutex_panic+58 ()
>>>>>>> warning! 8-byte aligned %fp = fffffe426a809038
>>>>> fffffe426a809038 unix:mutex_vector_exit+40 ()
>>>>> fffffe426a809070 kvm:gfn_to_memslot_unaliased+6f ()
>>>>> fffffe426a809090 kvm:gfn_to_hva+27 ()
>>>>> fffffe426a8090c0 kvm:kvm_read_guest_page+29 ()
>>>>> fffffe426a809110 kvm:kvm_read_guest+43 ()
>>>>> fffffe426a809190 kvm:paging64_walk_addr+ef ()
>>>>> fffffe426a809230 kvm:paging64_gva_to_gpa+43 ()
>>>>> fffffe426a809260 kvm:kvm_mmu_gva_to_gpa_read+45 ()
>>>>> fffffe426a8092b0 kvm:emulator_read_emulated+7c ()
>>>>> fffffe426a809350 kvm:x86_emulate_insn+1af ()
>>>>> fffffe426a809390 kvm:emulate_instruction+1e9 ()
>>>>> fffffe426a8093c0 kvm:kvm_mmu_page_fault+60 ()
>>>>> fffffe426a8093f0 kvm:handle_ept_violation+111 ()
>>>>> fffffe426a809430 kvm:vmx_handle_exit+16a ()
>>>>> fffffe426a809460 kvm:vcpu_enter_guest+3ea ()
>>>>> fffffe426a8094a0 kvm:__vcpu_run+8b ()
>>>>> fffffe426a8094e0 kvm:kvm_arch_vcpu_ioctl_run+112 ()
>>>>> fffffe426a809cc0 kvm:kvm_ioctl+466 ()
>>>>> fffffe426a809d00 genunix:cdev_ioctl+39 ()
>>>>> fffffe426a809d50 specfs:spec_ioctl+60 ()
>>>>> fffffe426a809de0 genunix:fop_ioctl+55 ()
>>>>> fffffe426a809f00 genunix:ioctl+9b ()
>>>>> fffffe426a809f10 unix:brand_sys_syscall+21d ()
>>>>>
>>>>> WARNING: /pci@0,0/pci8086,6f02@1/pci15d9,808@0 (mpt_sas0):
>>>>> mptsas_handle_event_sync: event 0xf, IOCStatus=0x8000,
>>>>> IOCLogInfo=0x31170000
>>>>> /pci@0,0/pci8086,6f02@1/pci15d9,808@0 (mpt_sas0):
>>>>> Log info 0x31170000 received for target 16 w5000cca07d0109a9.
>>>>> scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc
>>>>>
>>>>
>>>>
>>> smartos-discuss | Archives
>>> <https://www.listbox.com/member/archive/184463/=now>
>>> <https://www.listbox.com/member/archive/rss/184463/28443474-1732e24d> |
>>> Modify <https://www.listbox.com/member/?&;> Your Subscription
>>
>>
>
>
-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription:
https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com
