Re: bhyve: centos 7.1 with multiple virtual processors
On 15/07/2015 10:02, Neel Natu wrote: > > Could you update the host with the following patch? > https://people.freebsd.org/~neel/patches/ktr_stray_nmi.patch > > It enables KTR_GEN logging by default and turns it off when a VM-exit > due to NMI is detected. If there is a pattern to the NMIs then it can > be correlated with the guest activity around that time. Neel, sorry that it took so long to gather the additional data. Yesterday I got another host NMI while running a VM with KTR turned on. I panic-ed the system from the debugger prompt after the NMI. Below is several dozen of last messages taken from the crash dump. It does not look like there was a guest NMI or, at least, it didn't get a chance to be logged. Please let me know if the full log dump would be more useful. Initially, the last reported address 0x7f8c1185129b looked suspicious to me, but grepping the dump for 'extintr' I see many addresses like that. $ ktrdump -N /boot/kernel.dn/kernel -M /var/crash/vmcore.21 | head -40 index trace -- - 253313 vm centos7[0]: Resume execution at 0x7f8c1185129b 253312 vm centos7[0]: vmcb clean 0x3bf 253311 vm centos7[0]: handled extintr vmexit at 0x7f8c1185129b/0 253310 vm centos7[0]: Resume execution at 0x8105059c 253309 vm centos7[0]: vmcb clean 0x3bf 253308 vm centos7[0]: Guest interrupt blocking cleared due to rip change: 0x81050596/0x8105059c 253307 vm centos7[0]: vcpu state changed from frozen to running 253306 vm centos7[0]: nextrip updated to 0x8105059c after instruction decoding 253305 vm centos7[0]: inst_emul fault accessing gpa 0xfed000f0 253304 vm centos7[0]: vcpu state changed from running to frozen 253303 vm centos7[0]: unhandled nptfault vmexit at 0x81050596/0 253302 vm centos7[0]: inst_emul fault for gpa 0xfed000f0/0x10004 at rip 0x81050596 253301 vm centos7[0]: Resume execution at 0x8105058f 253300 vm centos7[0]: vmcb clean 0x3bf 253299 vm centos7[0]: Guest interrupt blocking cleared due to rip change: 0x8105058d/0x8105058f 253298 vm centos7[0]: vcpu state changed from frozen to running 253297 vm centos7[0]: nextrip updated to 0x8105058f after instruction decoding 253296 vm centos7[0]: inst_emul fault accessing gpa 0xfed00108 253295 vm centos7[0]: vcpu state changed from running to frozen 253294 vm centos7[0]: unhandled nptfault vmexit at 0x8105058d/0 253293 vm centos7[0]: inst_emul fault for gpa 0xfed00108/0x10006 at rip 0x8105058d 253292 vm centos7[0]: Resume execution at 0x8105057c 253291 vm centos7[0]: vmcb clean 0x3bf 253290 vm centos7[0]: Guest interrupt blocking cleared due to rip change: 0x81050576/0x8105057c 253289 vm centos7[0]: vcpu state changed from frozen to running 253288 vm centos7[0]: nextrip updated to 0x8105057c after instruction decoding 253287 vm centos7[0]: inst_emul fault accessing gpa 0xfed000f0 253286 vm centos7[0]: vcpu state changed from running to frozen 253285 vm centos7[0]: unhandled nptfault vmexit at 0x81050576/0 253284 vm centos7[0]: inst_emul fault for gpa 0xfed000f0/0x10004 at rip 0x81050576 253283 vm centos7[0]: Resume execution at 0x8104accc 253282 vm centos7[0]: vmcb clean 0x3b7 253281 vm centos7[0]: Clearing V_IRQ interrupt injection 253280 vm centos7[0]: Guest interrupt blocking cleared due to rip change: 0x8104acc6/0x8104accc 253279 vm centos7[0]: vcpu state changed from frozen to running 253278 vm centos7[0]: vlapic_update_ppr 0x00 253277 vm centos7[0]: vlapic_process_eoi isr7 0x 253276 vm centos7[0]: vlapic_process_eoi isr6 0x And a snippet from the stack-trace: ... #9 0x8081ed84 in trap (frame=0xfe01dc531f30) at /usr/src/sys/amd64/amd64/trap.c:372 #10 0x808074e3 in nmi_calltrap () at /usr/src/sys/libkern/explicit_bzero.c:28 #11 0x81c92327 in enable_gintr () at /usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895 #12 0x81c91c57 in svm_vmrun (arg=0xfe006d29c000, vcpu=0, rip=-2130377316, pmap=0xf801e4771138, evinfo=0xfe02b8c45820) at /usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:2022 #13 0x81c7661e in vm_run (vm=0xfe0076277000, vmrun=0xf8013fe43000) at /usr/src/sys/modules/vmm/../../amd64/vmm/vmm.c:1643 #14 0x81c79b78 in vmmdev_ioctl (cdev=, cmd=, data=0xf8013fe43000 "", fflag=-1, td=0x0) at /usr/src/sys/modules/vmm/../../amd64/vmm/vmm_dev.c:392 ... -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
Hi Andriy, On Tue, Jul 14, 2015 at 6:55 AM, Andriy Gapon wrote: > On 23/06/2015 11:05, Andriy Gapon wrote: >> On 23/06/2015 10:26, Neel Natu wrote: > [snip] >>> Does this ever happen with a single vcpu guest? >> >> Never seen the problem with a single CPU so far. >> Also, never had that problem with FreeBSD guests. >> >>> The other mystery is the NMIs the host is receiving. I (re)verified to >>> make sure that bhyve/vmm.ko do not assert NMIs so it has to be >>> something else on the host that's doing it ... >> >> But the correlation with the multi-CPU non-FreeBSD guests seems to be >> significant. > > Today I've got another NMI with a Linux guest, but a few things were > different: > - the host was fresh after a reboot > - the guest was single CPU > - the NMI happened during boot up of the very first VM > > kernel: NMI ISA 2c, EISA ff > kernel: NMI ... going to debugger > > Not what causes those NMIs but when they happen they always happen during a > bhyve guest boot... > Could you update the host with the following patch? https://people.freebsd.org/~neel/patches/ktr_stray_nmi.patch It enables KTR_GEN logging by default and turns it off when a VM-exit due to NMI is detected. If there is a pattern to the NMIs then it can be correlated with the guest activity around that time. best Neel > -- > Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
On 23/06/2015 11:05, Andriy Gapon wrote: > On 23/06/2015 10:26, Neel Natu wrote: [snip] >> Does this ever happen with a single vcpu guest? > > Never seen the problem with a single CPU so far. > Also, never had that problem with FreeBSD guests. > >> The other mystery is the NMIs the host is receiving. I (re)verified to >> make sure that bhyve/vmm.ko do not assert NMIs so it has to be >> something else on the host that's doing it ... > > But the correlation with the multi-CPU non-FreeBSD guests seems to be > significant. Today I've got another NMI with a Linux guest, but a few things were different: - the host was fresh after a reboot - the guest was single CPU - the NMI happened during boot up of the very first VM kernel: NMI ISA 2c, EISA ff kernel: NMI ... going to debugger Not what causes those NMIs but when they happen they always happen during a bhyve guest boot... -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
On 23/06/2015 11:05, Andriy Gapon wrote: > P.S. meanwhile I found this old-ish thread that seems to describe exactly the > problem I am seeing but on real hardware: > http://thread.gmane.org/gmane.linux.kernel/1483297 An update: the problem is fixed by http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/commit/?id=c4288334818c81c946acb23d2319881f58c3d497 P.S. Perhaps, it's worth checking if those spurious interrupts (from PIT) are just a result of timing or if bhyve might deviate from the specifications there. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
On 23/06/2015 10:26, Neel Natu wrote: > Hi Andriy, > > On Mon, Jun 22, 2015 at 11:45 PM, Andriy Gapon wrote: >> On 23/06/2015 05:37, Neel Natu wrote: >>> Hi Andriy, >>> >>> FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus >>> fine on my host with 8 physical cores. >>> >>> I have some questions about your setup inline. >>> >>> On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote: If I run a CentOS 7.1 VM with more than one CPU more often than not it would hang on startup and bhyve would start spinning. The following are the last messages seen in the VM: Switching to clocksource hpet [ cut here ] WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 Hardware name: BHYVE, BIOS 1.00 03/14/2014 cab5bdb6 88003fc03e08 81604eaa 88003fc03e40 8106e34b 800f423f 800f423f 81915440 88003fc03e50 Call Trace: [] dump_stack+0x19/0x1b [] warn_slowpath_common+0x6b/0xb0 [] warn_slowpath_null+0x1a/0x20 [] clockevents_program_event+0xdb/0xf0 [] tick_handle_periodic_broadcast+0x41/0x50 [] timer_interrupt+0x15/0x20 [] handle_irq_event_percpu+0x3e/0x1e0 [] handle_irq_event+0x3d/0x60 [] handle_edge_irq+0x77/0x130 [] handle_irq+0xbf/0x150 [] ? irq_enter+0x17/0xa0 [] do_IRQ+0x4f/0xf0 [] common_interrupt+0x6d/0x6d [] ? selinux_inode_alloc_security+0x59/0xa0 [] ? __d_instantiate+0xbf/0x100 [] ? __d_instantiate+0x9f/0x100 [] d_instantiate+0x3d/0x70 [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 [] __create_file+0x1c2/0x2c0 [] ? set_graph_function+0x1f/0x1f [] debugfs_create_dir+0x1b/0x20 [] tracing_init_dentry_tr+0x7e/0x90 [] tracing_init_dentry+0x10/0x20 [] ftrace_init_debugfs+0x13/0x1fd [] ? set_graph_function+0x1f/0x1f [] do_one_initcall+0xb8/0x230 [] kernel_init_freeable+0x18b/0x22a [] ? initcall_blacklist+0xb0/0xb0 [] ? rest_init+0x80/0x80 [] kernel_init+0xe/0xf0 [] ret_from_fork+0x7c/0xb0 [] ? rest_init+0x80/0x80 ---[ end trace d5caa1cab8e7e98d ]--- >>> >>> A few questions to narrow this down: >>> - Is the host very busy when the VM is started (or what is the host >>> doing when this happened)? >> >> The host typically is not heavily loaded. There is X server running and some >> applications. I'd imagine that those could cause some additional latency but >> not CPU starvation. >> > > Yup, I agree. > > Does this ever happen with a single vcpu guest? Never seen the problem with a single CPU so far. Also, never had that problem with FreeBSD guests. > The other mystery is the NMIs the host is receiving. I (re)verified to > make sure that bhyve/vmm.ko do not assert NMIs so it has to be > something else on the host that's doing it ... But the correlation with the multi-CPU non-FreeBSD guests seems to be significant. P.S. meanwhile I found this old-ish thread that seems to describe exactly the problem I am seeing but on real hardware: http://thread.gmane.org/gmane.linux.kernel/1483297 -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
Hi Andriy,
On Mon, Jun 22, 2015 at 11:45 PM, Andriy Gapon wrote:
> On 23/06/2015 05:37, Neel Natu wrote:
>> Hi Andriy,
>>
>> FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus
>> fine on my host with 8 physical cores.
>>
>> I have some questions about your setup inline.
>>
>> On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote:
>>>
>>> If I run a CentOS 7.1 VM with more than one CPU more often than not it would
>>> hang on startup and bhyve would start spinning.
>>>
>>> The following are the last messages seen in the VM:
>>>
>>> Switching to clocksource hpet
>>> [ cut here ]
>>> WARNING: at kernel/time/clockevents.c:239
>>> clockevents_program_event+0xdb/0xf0()
>>> Modules linked in:
>>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1
>>> Hardware name: BHYVE, BIOS 1.00 03/14/2014
>>> cab5bdb6 88003fc03e08 81604eaa
>>> 88003fc03e40 8106e34b 800f423f 800f423f
>>> 81915440 88003fc03e50
>>> Call Trace:
>>>[] dump_stack+0x19/0x1b
>>> [] warn_slowpath_common+0x6b/0xb0
>>> [] warn_slowpath_null+0x1a/0x20
>>> [] clockevents_program_event+0xdb/0xf0
>>> [] tick_handle_periodic_broadcast+0x41/0x50
>>> [] timer_interrupt+0x15/0x20
>>> [] handle_irq_event_percpu+0x3e/0x1e0
>>> [] handle_irq_event+0x3d/0x60
>>> [] handle_edge_irq+0x77/0x130
>>> [] handle_irq+0xbf/0x150
>>> [] ? irq_enter+0x17/0xa0
>>> [] do_IRQ+0x4f/0xf0
>>> [] common_interrupt+0x6d/0x6d
>>>[] ? selinux_inode_alloc_security+0x59/0xa0
>>> [] ? __d_instantiate+0xbf/0x100
>>> [] ? __d_instantiate+0x9f/0x100
>>> [] d_instantiate+0x3d/0x70
>>> [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130
>>> [] __create_file+0x1c2/0x2c0
>>> [] ? set_graph_function+0x1f/0x1f
>>> [] debugfs_create_dir+0x1b/0x20
>>> [] tracing_init_dentry_tr+0x7e/0x90
>>> [] tracing_init_dentry+0x10/0x20
>>> [] ftrace_init_debugfs+0x13/0x1fd
>>> [] ? set_graph_function+0x1f/0x1f
>>> [] do_one_initcall+0xb8/0x230
>>> [] kernel_init_freeable+0x18b/0x22a
>>> [] ? initcall_blacklist+0xb0/0xb0
>>> [] ? rest_init+0x80/0x80
>>> [] kernel_init+0xe/0xf0
>>> [] ret_from_fork+0x7c/0xb0
>>> [] ? rest_init+0x80/0x80
>>> ---[ end trace d5caa1cab8e7e98d ]---
>>>
>>
>> A few questions to narrow this down:
>> - Is the host very busy when the VM is started (or what is the host
>> doing when this happened)?
>
> The host typically is not heavily loaded. There is X server running and some
> applications. I'd imagine that those could cause some additional latency but
> not CPU starvation.
>
Yup, I agree.
Does this ever happen with a single vcpu guest?
The other mystery is the NMIs the host is receiving. I (re)verified to
make sure that bhyve/vmm.ko do not assert NMIs so it has to be
something else on the host that's doing it ...
best
Neel
>> - How many vcpus are you giving to the VM?
>> - How many cores on the host?
>
> I tried only 2 / 2.
>
>>>
>>> At the same time sometimes there is one or more of spurious NMIs on the
>>> _host_
>>> system:
>>> NMI ISA c, EISA ff
>>> NMI ... going to debugger
>>>
>>
>> Hmm, that's interesting. Are you using hwpmc to do instruction sampling?
>
> hwpmc driver is in the kernel, but it was not used.
>
>>> bhyve seems to spin here:
>>> vmm.ko`svm_vmrun+0x894
>>> vmm.ko`vm_run+0xbb7
>>> vmm.ko`vmmdev_ioctl+0x5a4
>>> kernel`devfs_ioctl_f+0x13b
>>> kernel`kern_ioctl+0x1e1
>>> kernel`sys_ioctl+0x16a
>>> kernel`amd64_syscall+0x3ca
>>> kernel`0x8088997b
>>>
>>> (kgdb) list *svm_vmrun+0x894
>>> 0x813c9194 is in svm_vmrun
>>> (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895).
>>> 1890
>>> 1891static __inline void
>>> 1892enable_gintr(void)
>>> 1893{
>>> 1894
>>> 1895__asm __volatile("stgi");
>>> 1896}
>>> 1897
>>> 1898/*
>>> 1899 * Start vcpu with specified RIP.
>>>
>>
>> Yeah, that's not surprising because host interrupts are blocked when
>> the cpu is executing in guest context. The 'enable_gintr()' enables
>> interrupts so it gets blamed by the interrupt-based sampling.
>>
>> In this case it just means that the cpu was in guest context when a
>> host-interrupt fired.
>
> I see. FWIW, that was captured with DTrace.
>
> --
> Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
On 23/06/2015 05:37, Neel Natu wrote:
> Hi Andriy,
>
> FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus
> fine on my host with 8 physical cores.
>
> I have some questions about your setup inline.
>
> On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote:
>>
>> If I run a CentOS 7.1 VM with more than one CPU more often than not it would
>> hang on startup and bhyve would start spinning.
>>
>> The following are the last messages seen in the VM:
>>
>> Switching to clocksource hpet
>> [ cut here ]
>> WARNING: at kernel/time/clockevents.c:239
>> clockevents_program_event+0xdb/0xf0()
>> Modules linked in:
>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1
>> Hardware name: BHYVE, BIOS 1.00 03/14/2014
>> cab5bdb6 88003fc03e08 81604eaa
>> 88003fc03e40 8106e34b 800f423f 800f423f
>> 81915440 88003fc03e50
>> Call Trace:
>>[] dump_stack+0x19/0x1b
>> [] warn_slowpath_common+0x6b/0xb0
>> [] warn_slowpath_null+0x1a/0x20
>> [] clockevents_program_event+0xdb/0xf0
>> [] tick_handle_periodic_broadcast+0x41/0x50
>> [] timer_interrupt+0x15/0x20
>> [] handle_irq_event_percpu+0x3e/0x1e0
>> [] handle_irq_event+0x3d/0x60
>> [] handle_edge_irq+0x77/0x130
>> [] handle_irq+0xbf/0x150
>> [] ? irq_enter+0x17/0xa0
>> [] do_IRQ+0x4f/0xf0
>> [] common_interrupt+0x6d/0x6d
>>[] ? selinux_inode_alloc_security+0x59/0xa0
>> [] ? __d_instantiate+0xbf/0x100
>> [] ? __d_instantiate+0x9f/0x100
>> [] d_instantiate+0x3d/0x70
>> [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130
>> [] __create_file+0x1c2/0x2c0
>> [] ? set_graph_function+0x1f/0x1f
>> [] debugfs_create_dir+0x1b/0x20
>> [] tracing_init_dentry_tr+0x7e/0x90
>> [] tracing_init_dentry+0x10/0x20
>> [] ftrace_init_debugfs+0x13/0x1fd
>> [] ? set_graph_function+0x1f/0x1f
>> [] do_one_initcall+0xb8/0x230
>> [] kernel_init_freeable+0x18b/0x22a
>> [] ? initcall_blacklist+0xb0/0xb0
>> [] ? rest_init+0x80/0x80
>> [] kernel_init+0xe/0xf0
>> [] ret_from_fork+0x7c/0xb0
>> [] ? rest_init+0x80/0x80
>> ---[ end trace d5caa1cab8e7e98d ]---
>>
>
> A few questions to narrow this down:
> - Is the host very busy when the VM is started (or what is the host
> doing when this happened)?
The host typically is not heavily loaded. There is X server running and some
applications. I'd imagine that those could cause some additional latency but
not CPU starvation.
> - How many vcpus are you giving to the VM?
> - How many cores on the host?
I tried only 2 / 2.
>>
>> At the same time sometimes there is one or more of spurious NMIs on the
>> _host_
>> system:
>> NMI ISA c, EISA ff
>> NMI ... going to debugger
>>
>
> Hmm, that's interesting. Are you using hwpmc to do instruction sampling?
hwpmc driver is in the kernel, but it was not used.
>> bhyve seems to spin here:
>> vmm.ko`svm_vmrun+0x894
>> vmm.ko`vm_run+0xbb7
>> vmm.ko`vmmdev_ioctl+0x5a4
>> kernel`devfs_ioctl_f+0x13b
>> kernel`kern_ioctl+0x1e1
>> kernel`sys_ioctl+0x16a
>> kernel`amd64_syscall+0x3ca
>> kernel`0x8088997b
>>
>> (kgdb) list *svm_vmrun+0x894
>> 0x813c9194 is in svm_vmrun
>> (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895).
>> 1890
>> 1891static __inline void
>> 1892enable_gintr(void)
>> 1893{
>> 1894
>> 1895__asm __volatile("stgi");
>> 1896}
>> 1897
>> 1898/*
>> 1899 * Start vcpu with specified RIP.
>>
>
> Yeah, that's not surprising because host interrupts are blocked when
> the cpu is executing in guest context. The 'enable_gintr()' enables
> interrupts so it gets blamed by the interrupt-based sampling.
>
> In this case it just means that the cpu was in guest context when a
> host-interrupt fired.
I see. FWIW, that was captured with DTrace.
--
Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
Hi Andriy,
FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus
fine on my host with 8 physical cores.
I have some questions about your setup inline.
On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote:
>
> If I run a CentOS 7.1 VM with more than one CPU more often than not it would
> hang on startup and bhyve would start spinning.
>
> The following are the last messages seen in the VM:
>
> Switching to clocksource hpet
> [ cut here ]
> WARNING: at kernel/time/clockevents.c:239
> clockevents_program_event+0xdb/0xf0()
> Modules linked in:
> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1
> Hardware name: BHYVE, BIOS 1.00 03/14/2014
> cab5bdb6 88003fc03e08 81604eaa
> 88003fc03e40 8106e34b 800f423f 800f423f
> 81915440 88003fc03e50
> Call Trace:
>[] dump_stack+0x19/0x1b
> [] warn_slowpath_common+0x6b/0xb0
> [] warn_slowpath_null+0x1a/0x20
> [] clockevents_program_event+0xdb/0xf0
> [] tick_handle_periodic_broadcast+0x41/0x50
> [] timer_interrupt+0x15/0x20
> [] handle_irq_event_percpu+0x3e/0x1e0
> [] handle_irq_event+0x3d/0x60
> [] handle_edge_irq+0x77/0x130
> [] handle_irq+0xbf/0x150
> [] ? irq_enter+0x17/0xa0
> [] do_IRQ+0x4f/0xf0
> [] common_interrupt+0x6d/0x6d
>[] ? selinux_inode_alloc_security+0x59/0xa0
> [] ? __d_instantiate+0xbf/0x100
> [] ? __d_instantiate+0x9f/0x100
> [] d_instantiate+0x3d/0x70
> [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130
> [] __create_file+0x1c2/0x2c0
> [] ? set_graph_function+0x1f/0x1f
> [] debugfs_create_dir+0x1b/0x20
> [] tracing_init_dentry_tr+0x7e/0x90
> [] tracing_init_dentry+0x10/0x20
> [] ftrace_init_debugfs+0x13/0x1fd
> [] ? set_graph_function+0x1f/0x1f
> [] do_one_initcall+0xb8/0x230
> [] kernel_init_freeable+0x18b/0x22a
> [] ? initcall_blacklist+0xb0/0xb0
> [] ? rest_init+0x80/0x80
> [] kernel_init+0xe/0xf0
> [] ret_from_fork+0x7c/0xb0
> [] ? rest_init+0x80/0x80
> ---[ end trace d5caa1cab8e7e98d ]---
>
A few questions to narrow this down:
- Is the host very busy when the VM is started (or what is the host
doing when this happened)?
- How many vcpus are you giving to the VM?
- How many cores on the host?
>
> At the same time sometimes there is one or more of spurious NMIs on the _host_
> system:
> NMI ISA c, EISA ff
> NMI ... going to debugger
>
Hmm, that's interesting. Are you using hwpmc to do instruction sampling?
> bhyve seems to spin here:
> vmm.ko`svm_vmrun+0x894
> vmm.ko`vm_run+0xbb7
> vmm.ko`vmmdev_ioctl+0x5a4
> kernel`devfs_ioctl_f+0x13b
> kernel`kern_ioctl+0x1e1
> kernel`sys_ioctl+0x16a
> kernel`amd64_syscall+0x3ca
> kernel`0x8088997b
>
> (kgdb) list *svm_vmrun+0x894
> 0x813c9194 is in svm_vmrun
> (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895).
> 1890
> 1891static __inline void
> 1892enable_gintr(void)
> 1893{
> 1894
> 1895__asm __volatile("stgi");
> 1896}
> 1897
> 1898/*
> 1899 * Start vcpu with specified RIP.
>
Yeah, that's not surprising because host interrupts are blocked when
the cpu is executing in guest context. The 'enable_gintr()' enables
interrupts so it gets blamed by the interrupt-based sampling.
In this case it just means that the cpu was in guest context when a
host-interrupt fired.
best
Neel
> --
> Andriy Gapon
> ___
> freebsd-virtualization@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
On 22/06/2015 17:41, Peter Grehan wrote: > Hi Andriy, > >> If I run a CentOS 7.1 VM with more than one CPU more often than not it would >> hang on startup and bhyve would start spinning. > > Looks like an AMD host - what's the version of FreeBSD you are running there > ? Yes, this is an AMD host, the version is head@283188. -- Andriy Gapon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve: centos 7.1 with multiple virtual processors
Hi Andriy, If I run a CentOS 7.1 VM with more than one CPU more often than not it would hang on startup and bhyve would start spinning. Looks like an AMD host - what's the version of FreeBSD you are running there ? later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
