Run bhyve in KVM on Intel E5-2660 v4 issues

[Elena Mihailescu]

Hello,

We have some issues on using bhyve from inside a FreeBSD KVM virtual
machine on Intel E5-2660 v4 servers.  The host operating systems we
tried and the errors are described below.

If the host is one of the following operating systems: Ubuntu 18.04,
Ubuntu 18.04 for servers or CentOS8 we get the following behavior:
Tested on FreeBSD-Current
(FreeBSD-13.0-CURRENT-amd64-20200213-r357847) and FreeBSD12.1:
- starting bhyve using bhyveload as a loader ends with a kernel panic
in the bhyve VM due to APIC error:
# sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1G -d guest.img -i -I
FreeBSD-12.1-RELEASE-amd64-disc1.iso machine

Event timer "LAPIC" quality 600
ACPI APIC Table: 
panic: APIC: CPU with APIC ID 255 is not enabled
cpuid = 0
time = 1
KDB: stack backtrace:
#0 0x80c1d297 at kdb_backtrace+0x67
#1 0x80bd05cd at vpanic+0x19d
#2 0x80bd0423 at panic+0x43
#3 0x811f7569 at madt_set_ids+0x99
#4 0x80b66918 at mi_startup+0x118
#5 0x8036b02c at btext+0x2c
Uptime: 1s


- starting bhyve using uefi as a loader seems to stuck or run in an
infinite loop  - nothing is displayed on screen after starting the VM,
we cannot interact with it.

bhyve -c 1 -m 1G -H -P -S -s 0:0,hostbridge -s 1:0,lpc -s
4:0,ahci-hd,guest.img -s 3:0,ahci-cd, -l com1,stdio -l
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd 
*stuck - does not receive input, does not display anything.

We also tried disabling x2apic from the host kernel options and also
from the KVM guest options. As for libvirt options, we tried both cpu
options: host-model and host-passthrough.

If the host is a CentOS7, we didn't manage to get the FreeBSD KVM
virtual machine to see the hardware virtualization features and we
cannot run `kldload vmm` successfully (error: vmx_init: processor does
not support desired basic capabilities). We have tested with a CentOS7
guest and nested virtualization with kvm does not have any issues.

In the document at [1], we centralized the tests we run and their
output, including virsh xml files, bhyve commands, and the errors we
have encounters. As a parallel, we installed FreeBSD directly on the
server and ran the same bhyve commands that worked fine.

Does anyone know any way to solve this issue?

[1]  
https://docs.google.com/document/d/1DIzdJKHNZNLZXk9dJquUDwwYJpTH2zDkT9eF2sVIb-A/edit?usp=sharing

Thank you,
Elena
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve in KVM

[gabor]

2016-08-10 21:30 időpontban Peter Grehan ezt írta:

Hi Gábor,


First of all, what is this Disabled in BIOS?
Second, what needs to use bhyve, az kldloading vmm I got:
vmx_init: processor does not support desired basic capabilities
module_register_init: MOD_LOAD (vmm, 0x8221c4e0, 0) error 22

What's missing?


 There are some checks that have to be relaxed (and some slight
rework). See the thread starting at
http://lists.freebsd.org/pipermail/freebsd-virtualization/2015-October/003870.html,
and also PR 203994.

later,

Peter.


Thanks for the info.

Zahemszky, Gábor
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve in KVM

[Peter Grehan]

Hi Gábor,


First of all, what is this Disabled in BIOS?
Second, what needs to use bhyve, az kldloading vmm I got:
vmx_init: processor does not support desired basic capabilities
module_register_init: MOD_LOAD (vmm, 0x8221c4e0, 0) error 22

What's missing?


 There are some checks that have to be relaxed (and some slight 
rework). See the thread starting at 
http://lists.freebsd.org/pipermail/freebsd-virtualization/2015-October/003870.html, 
and also PR 203994.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve in KVM

[gabor]

Hi!

I tried the 11-beta4 in a KVM (qemu) virtual machine. In Linux, the 
nested KVM functionality is enabled. I started the VM with:


qemu-system-x86_64 -enable-kvm -cpu host,+vmx -m 512 FBSD.qcow2

And here is processor info in the FreeBSD VM:

CPU: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz (2294.49-MHz K8-class CPU)
 Origin="GenuineIntel"  Id=0x306d4  Family=0x6  Model=0x3d  Stepping=4
 
Features=0xf83fbff
 
Features2=0xfffa3223

 AMD Features=0x2c100800
 AMD Features2=0x121
 Structured Extended 
Features=0x1c0f3a

 XSAVE Features=0x1
 VT-x: (disabled in BIOS) PAT,HLT,PAUSE,EPT,UG
Hypervisor: Origin = "KVMKVMKVM"

First of all, what is this Disabled in BIOS?
Second, what needs to use bhyve, az kldloading vmm I got:
vmx_init: processor does not support desired basic capabilities
module_register_init: MOD_LOAD (vmm, 0x8221c4e0, 0) error 22

What's missing?

Thanks,

Gábor Zahemszky
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issues with bhyve under KVM

[Peter Grehan]

Hi Cory,


vmx_init: ept initialization failed (22)
module_register_init: MOD_LOAD (vmm, 0x81e16fd0, 0) error 22

A more useful error, but still incorrect I am pretty sure.


 This one is because KVM doesn't support the invvpid instruction when 
running nested. The bhyve ept init code requires for this, though it may 
not be strictly necessary.


 I've created PR 203994 to track the issues for running nested under KVM.

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issues with bhyve under KVM

[Peter Grehan]

Hi Cory,


`dmesg` prints a message saying that VT-x is "disabled by BIOS" (I
passed -L and -bios pointed at OVMF just in case it was a strange
SEABIOS compile option) to no change.  -cpu is set to host, I have also
tried passing -cpu qemu64,+vmx manually to no change.


 That's not strictly true, and in this case harmless. I'll make sure 
that is fixed.



Oct 14 06:13:19  kernel: vmx_init: processor does not support desired
basic capabilities


 bhyve expects that the CPU supports INS/OUTS exits. This is signalled 
in bit 54 of the VMX_BASIC MSR. I'll have a talk with Neel about the 
implications of ignoring this bit.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issues with bhyve under KVM

[Peter Grehan]

Hi Cory,


 bhyve expects that the CPU supports INS/OUTS exits. This is signalled
in bit 54 of the VMX_BASIC MSR. I'll have a talk with Neel about the
implications of ignoring this bit.


Hmm.  Does Linux KVM ignore it?


 It must do (as did bhyve for a long time).


(does -host not emulate all MSRs?)


 Probably just not that bit. You might want to try the this patch which 
comments out that test, and see if it gets you any further.

 https://people.freebsd.org/~grehan/patches/vmx_ins.diff

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issues with bhyve under KVM

[Cory Smelosky]

On 2015-10-21 23:15, Peter Grehan wrote:

Hi Cory,


`dmesg` prints a message saying that VT-x is "disabled by BIOS" (I
passed -L and -bios pointed at OVMF just in case it was a strange
SEABIOS compile option) to no change.  -cpu is set to host, I have 
also

tried passing -cpu qemu64,+vmx manually to no change.


 That's not strictly true, and in this case harmless. I'll make sure
that is fixed.


Ahhh.  Yeah - misleading errors are annoying!




Oct 14 06:13:19  kernel: vmx_init: processor does not support desired
basic capabilities


 bhyve expects that the CPU supports INS/OUTS exits. This is signalled
in bit 54 of the VMX_BASIC MSR. I'll have a talk with Neel about the
implications of ignoring this bit.


Hmm.  Does Linux KVM ignore it?

(does -host not emulate all MSRs?)



later,

Peter.


--
--
Cory Smelosky
iXsystems
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issues with bhyve under KVM

[Peter Grehan]

Hi Cory,


 Probably just not that bit. You might want to try the this patch
which comments out that test, and see if it gets you any further.
 https://people.freebsd.org/~grehan/patches/vmx_ins.diff


--- all_subdir_vmm ---
/usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:535:11: error:
unused variable 'basic' [-Werror,-Wunused-variable]
 uint64_t basic, fixed0, fixed1, feature_control;
  ^
1 error generated.

Aww. I need to actually run a later SVN release instead of 28. Tried
to save some time. ;)


 Apologies - I didn't compile-test the patch, but the basic idea is to 
comment out that block. You can delete the unused variable 'basic' which 
should get you going.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"