Re: High vCPU Counts in bhyve
Peter, Thanks for getting back! Patching /usr/src/usr.sbin/bhyve/acpi.c does indeed allow me to run a bhyve VM with 32 vCPUs! root@overkill:~ # sysctl hw.ncpu hw.model hw.ncpu: 32 hw.model: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz root@overkill:~ # dmesg | grep Hypervisor Hypervisor: Origin = "bhyve bhyve " root@overkill:~ # Thanks for your help and for all of your work on bhyve in general! -Trent @pr1ntf ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: High vCPU Counts in bhyve
On 2016-09-17 19:55, Peter Grehan wrote: > Hi Trent, > >> If anyone has any hints on how to get more information from this or ideas >> on the apparent APIC error would be greatly appreciated. > > Thanks for the boot logs. The bhyve bug can be seen from: > > ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C, > should be 0x0A (20160527/tbprint-229) > [7/1843] > MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ 9 > > .. which is pointing to the MADT table being overwritten. There is only > 256 bytes allocated for this table when it is being created: > > acpi.c > * MADT -> 0xf2500 (depends on #CPUs) > * FADT -> 0xf2600 (268 bytes) > ... > #define MADT_OFFSET 0x100 > #define FADT_OFFSET 0x200 > > The MADT* for bhyve has a 44-byte fixed header, followed by a array of > 8-byte 'Processor Local APIC' entries, one for each vCPU. The end of the > table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source > Override' entries, and a 6-byte 'Local APIC NMI' entry. > > Looking at the max #CPUs that can fit into 256 bytes: > > 256 = 44 + N*8 + 12 + 2*10 + 6, which gives N = 21. The fact that it > worked for slightly larger values is probably due to the table entries > at the end being ignored, until eventually the I/O APIC table entry was > corrupted by the FADT overwrite. > > A quick fix to get more vCPUs is to bump the addresses of the tables in > acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs. > > +#define FADT_OFFSET 0x700 > +#define HPET_OFFSET 0x840 > +#define MCFG_OFFSET 0x880 > +#define FACS_OFFSET 0x8C0 > +#define DSDT_OFFSET 0x900 > -#define FADT_OFFSET 0x200 > -#define HPET_OFFSET 0x340 > -#define MCFG_OFFSET 0x380 > -#define FACS_OFFSET 0x3C0 > -#define DSDT_OFFSET 0x400 > > I'll create a bug for this so that the overwrite will be detected at > run-time, and also bump up the space to allow for some growth. > > later, > > Peter. > > * see the ACPI spec at http://www.acpi.info/spec.htm for table details. > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" Ha, so the only reason I didn't run into this, is the biggest VM i've created just happened to be 20 vCPUs on my 24 CPU system. Dodged a bullet, or, delay the discovery of this bug by 2 years. One of the two. -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: High vCPU Counts in bhyve
Hi Trent, If anyone has any hints on how to get more information from this or ideas on the apparent APIC error would be greatly appreciated. Thanks for the boot logs. The bhyve bug can be seen from: ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C, should be 0x0A (20160527/tbprint-229) [7/1843] MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ 9 .. which is pointing to the MADT table being overwritten. There is only 256 bytes allocated for this table when it is being created: acpi.c * MADT -> 0xf2500 (depends on #CPUs) * FADT -> 0xf2600 (268 bytes) ... #define MADT_OFFSET 0x100 #define FADT_OFFSET 0x200 The MADT* for bhyve has a 44-byte fixed header, followed by a array of 8-byte 'Processor Local APIC' entries, one for each vCPU. The end of the table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source Override' entries, and a 6-byte 'Local APIC NMI' entry. Looking at the max #CPUs that can fit into 256 bytes: 256 = 44 + N*8 + 12 + 2*10 + 6, which gives N = 21. The fact that it worked for slightly larger values is probably due to the table entries at the end being ignored, until eventually the I/O APIC table entry was corrupted by the FADT overwrite. A quick fix to get more vCPUs is to bump the addresses of the tables in acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs. +#define FADT_OFFSET 0x700 +#define HPET_OFFSET 0x840 +#define MCFG_OFFSET 0x880 +#define FACS_OFFSET 0x8C0 +#define DSDT_OFFSET 0x900 -#define FADT_OFFSET 0x200 -#define HPET_OFFSET 0x340 -#define MCFG_OFFSET 0x380 -#define FACS_OFFSET 0x3C0 -#define DSDT_OFFSET 0x400 I'll create a bug for this so that the overwrite will be detected at run-time, and also bump up the space to allow for some growth. later, Peter. * see the ACPI spec at http://www.acpi.info/spec.htm for table details. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
High vCPU Counts in bhyve
Good day to all! I was recently given the hint that I could edit vmm.h to allow for more than 16 vCPUs in bhyve, so I did just that to see what would happen. I understand it's not supported, just wanted to try it out. When trying to boot FreeBSD 12-CURRENT with anything up to 25 vCPUs the guest will boot and run just fine. At 26+ vCPUs the guest OS will panic at boot time. I have posted all of these outputs and more to GitHub [1]. I started by installing FreeBSD with 16 vCPUs and then incrementing until it wouldn't go any further. You can find the dmesg.boot output of the guest at 16 vCPUs on GitHub [2]. Here is the relevant line in /usr/src/sys/amd64/include/vmm.h: #define VM_MAXCPU32/* maximum virtual cpus */ Here is what appears in the serial console when attempting to boot with 26 vCPUs (The full output can also be found on GitHub [3]): Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x60 fault code = supervisor write data, page not present instruction pointer = 0x20:0x80ff3859 stack pointer = 0x28:0x821c19d0 frame pointer = 0x28:0x821c1a50 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 0 (swapper) [ thread pid 0 tid 10 ] Stopped at acpi_install_wakeup_handler+0x1d9: movq%rdi,0x60(%r14) db> If anyone has any hints on how to get more information from this or ideas on the apparent APIC error would be greatly appreciated. Thanks and happy hacking! -Trent @pr1ntf [1] https://gist.github.com/pr1ntf/270efd6ec3fe376e68d1f9dad2deef31 [2] https://gist.githubusercontent.com/pr1ntf/270efd6ec3fe376e68d1f9dad2deef31/raw/09dde8362a5e488794e6d7c8b70bb0368b964b49/3-guest-dmesg.boot.txt [3] https://gist.githubusercontent.com/pr1ntf/270efd6ec3fe376e68d1f9dad2deef31/raw/09dde8362a5e488794e6d7c8b70bb0368b964b49/4-guest-26vcpucrash.txt ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
