bhyve and virtualbox on the same host using EPT
All, I'm trying to run a bhyve VM on a host that's already running Virtualbox, the problem I encounter is bhyve doesn't detect EPT capabilities but the Virtualbox logs indicate EPT is in use and functional. My question is; is it possible for two pieces of virtualization software to use EPT on the same host? See below for the various logs, happy to provide more info if needed. Bhyve generates the following error after the bootloader screen: - Booting... 4 vCPUs requested but only 1 available - A quick look in the source code indicates that this error means bhyve can't use EPT and falls back to using 1 CPU, now the machine in question does have EPT capabilites (it lists POPCNT in dmesg): - uname -a FreeBSD becks.debank.tv 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r253788M: Tue Jul 30 12:02:57 NZST 2013 r...@becks.debank.tv:/usr/obj/usr/src/sys/GENERIC amd64 - - from dmesg CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz (2664.82-MHz K8-class CPU) Origin = GenuineIntel Id = 0x106a5 Family = 0x6 Model = 0x1a Stepping = 5 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0x98e3bdSSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT AMD Features=0x28100800SYSCALL,NX,RDTSCP,LM AMD Features2=0x1LAHF TSC: P-state invariant, performance statistics real memory = 25769803776 (24576 MB) avail memory = 24060841984 (22946 MB) - - root@becks:~/bhyve # kldstat Id Refs AddressSize Name 1 27 0x8020 150e0d0 kernel 21 0x8170f000 224a98 zfs.ko 32 0x81934000 5c30 opensolaris.ko 41 0x8193a000 2d40 coretemp.ko 53 0x8193d000 51a68vboxdrv.ko 62 0x81a12000 28c0 vboxnetflt.ko 72 0x81a15000 881d netgraph.ko 81 0x81a1e000 15ff ng_ether.ko 91 0x81a2 3ec0 vboxnetadp.ko 101 0x81aba000 1037 cpuctl.ko 111 0x81abc000 1686ac vmm.ko - The Virtualbox logs below show EPT in use: - 00:00:00.419806 HWACCM: MSR_IA32_VMX_PROCBASED_CTLS2 = 7f 00:00:00.419807 HWACCM:VMX_VMCS_CTRL_PROC_EXEC2_VIRT_APIC 00:00:00.419807 HWACCM:VMX_VMCS_CTRL_PROC_EXEC2_EPT 00:00:00.419808 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_DESCRIPTOR_INSTR_EXIT 00:00:00.419808 HWACCM:VMX_VMCS_CTRL_PROC_EXEC2_RDTSCP 00:00:00.419809 HWACCM:VMX_VMCS_CTRL_PROC_EXEC2_X2APIC 00:00:00.419809 HWACCM:VMX_VMCS_CTRL_PROC_EXEC2_VPID 00:00:00.419810 HWACCM:VMX_VMCS_CTRL_PROC_EXEC2_WBINVD_EXIT snip -- 00:00:00.419916 HWACCM: 32-bit and 64-bit guests supported. 00:00:00.419917 HWACCM: VMX enabled! 00:00:00.419918 HWACCM: Enabled nested paging 00:00:00.419919 HWACCM: EPT root page = b6167000 00:00:00.419922 HWACCM: Large page support enabled! - Thanks, Rob Evers ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve and virtualbox on the same host using EPT
Hi Rob, Bhyve generates the following error after the bootloader screen: - Booting... 4 vCPUs requested but only 1 available - What this means is that the model of CPU that you have doesn't support the unrestricted guest feature, which allows a guest to run in 16-bit mode. bhyve boots FreeBSD directly into 64-bit mode on the main CPU, but secondary CPUs are started through a boostrap from 16-bit mode to 64-bit mode. There was a hack early on in bhyve to avoid this by starting the secondaries directly in 64-bit mode, but it was bhyve- and FreeBSD-specific and difficult to maintain. Given that there was only a small timeframe of CPUs that supported EPT but not unrestricted guest (early Nehalems), we decided to the 64-bit AP spinup. You can still run single vCPU guests though. The Virtualbox logs below show EPT in use: On the issue of running VBox and bhyve at the same time: that's not recommended and will probably result in a crash. To do this requires cooperation from hypervisors in that they will completely flush state any time they could be context-switched away to avoid the other hypervisor's state from colliding. This results in poor performance which is why it usually isn't done by default. I think we know how to fix this in bhyve, but aren't there yet. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: bhyve and virtualbox on the same host using EPT
On Tue, Aug 6, 2013 at 3:02 PM, mailingli...@debank.tv wrote: All, I'm trying to run a bhyve VM on a host that's already running Virtualbox, the problem I encounter is bhyve doesn't detect EPT capabilities but the Virtualbox logs indicate EPT is in use and functional. My question is; is it possible for two pieces of virtualization software to use EPT on the same host? See below for the various logs, happy to provide more info if needed. Bhyve generates the following error after the bootloader screen: - Booting... 4 vCPUs requested but only 1 available - A quick look in the source code indicates that this error means bhyve can't use EPT and falls back to using 1 CPU, now the machine in question does have EPT capabilites (it lists POPCNT in dmesg): - uname -a FreeBSD becks.debank.tv [1] 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r253788M: Tue Jul 30 12:02:57 NZST 2013 r...@becks.debank.tv:/usr/obj/usr/src/sys/GENERIC amd64 - - from dmesg CPU: Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz (2664.82-MHz K8-class CPU) Origin = GenuineIntel Id = 0x106a5 Family = 0x6 Model = 0x1a Stepping = 5 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0x98e3bdSSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT AMD Features=0x28100800SYSCALL,NX,RDTSCP,LM AMD Features2=0x1LAHF TSC: P-state invariant, performance statistics real memory = 25769803776 (24576 MB) avail memory = 24060841984 (22946 MB) - - root@becks:~/bhyve # kldstat Id Refs Address Size Name 1 27 0x8020 150e0d0 kernel 2 1 0x8170f000 224a98 zfs.ko 3 2 0x81934000 5c30 opensolaris.ko 4 1 0x8193a000 2d40 coretemp.ko 5 3 0x8193d000 51a68 vboxdrv.ko 6 2 0x81a12000 28c0 vboxnetflt.ko 7 2 0x81a15000 881d netgraph.ko 8 1 0x81a1e000 15ff ng_ether.ko 9 1 0x81a2 3ec0 vboxnetadp.ko 10 1 0x81aba000 1037 cpuctl.ko 11 1 0x81abc000 1686ac vmm.ko - The Virtualbox logs below show EPT in use: - 00:00:00.419806 HWACCM: MSR_IA32_VMX_PROCBASED_CTLS2 = 7f 00:00:00.419807 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_VIRT_APIC 00:00:00.419807 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_EPT 00:00:00.419808 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_DESCRIPTOR_INSTR_EXIT 00:00:00.419808 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_RDTSCP 00:00:00.419809 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_X2APIC 00:00:00.419809 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_VPID 00:00:00.419810 HWACCM: VMX_VMCS_CTRL_PROC_EXEC2_WBINVD_EXIT snip -- 00:00:00.419916 HWACCM: 32-bit and 64-bit guests supported. 00:00:00.419917 HWACCM: VMX enabled! 00:00:00.419918 HWACCM: Enabled nested paging 00:00:00.419919 HWACCM: EPT root page = b6167000 00:00:00.419922 HWACCM: Large page support enabled! - Thanks, Rob Evers ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization [2] To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org On 2013-08-07 10:33, Anish wrote: Each processor has its own EPT root pointer in VMCS. How many processors are there in system and how many are given to Virtualbox. error = vmcs_set_defaults(vmx-vmcs[i], (u_long)vmx_longjmp, (u_long)vmx-ctx[i], vtophys(vmx-pml4ept), pinbased_ctls, procbased_ctls, procbased_ctls2, exit_ctls, entry_ctls, vtophys(vmx-msr_bitmap), vpid); Anish, The machine has one physical CPU with 4 cores using HT giving 8 threads. I can't quickly find an option to Virtualbox to limit the amount of cpu's used but did try to disable Virtualbox (no rc.d scripts ran and disabled loading the kernel module) after a reboot of the machine, the problem persists. - root@becks:~/bhyve # kldstat Id Refs Address Size Name 1 17 0x8020 150e0d0 kernel 2 1 0x8170f000 224a98 zfs.ko 3 2 0x81934000 5c30 opensolaris.ko 4 1 0x8193a000 2d40 coretemp.ko 5 1 0x81a12000 1037 cpuctl.ko 6 1 0x81a14000 1686ac vmm.ko - - root@becks:~/bhyve # sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 2048 fbsd_10 Launching virtual machine fbsd_10 ... Consoles: userboot
Re: bhyve and virtualbox on the same host using EPT
On 2013-08-07 11:06, Peter Grehan wrote: Hi Rob, Bhyve generates the following error after the bootloader screen: - Booting... 4 vCPUs requested but only 1 available - What this means is that the model of CPU that you have doesn't support the unrestricted guest feature, which allows a guest to run in 16-bit mode. bhyve boots FreeBSD directly into 64-bit mode on the main CPU, but secondary CPUs are started through a boostrap from 16-bit mode to 64-bit mode. There was a hack early on in bhyve to avoid this by starting the secondaries directly in 64-bit mode, but it was bhyve- and FreeBSD-specific and difficult to maintain. Given that there was only a small timeframe of CPUs that supported EPT but not unrestricted guest (early Nehalems), we decided to the 64-bit AP spinup. You can still run single vCPU guests though. The Virtualbox logs below show EPT in use: On the issue of running VBox and bhyve at the same time: that's not recommended and will probably result in a crash. To do this requires cooperation from hypervisors in that they will completely flush state any time they could be context-switched away to avoid the other hypervisor's state from colliding. This results in poor performance which is why it usually isn't done by default. I think we know how to fix this in bhyve, but aren't there yet. later, Peter. Peter, Thanks for your answer, this clears things up, I was under the impression that EPT support automatically included the unrestricted guest feature. Unfortunately I'll have to stick with Virtualbox as I need multiple CPU support for my vm's. Thanks for the great software though, I'll be back running bHyve once I get my hands on a more capable machine! Thanks, Rob Evers ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
