Hi Trent,

If anyone has any hints on how to get more information from this or ideas
on the apparent APIC error would be greatly appreciated.

 Thanks for the boot logs. The bhyve bug can be seen from:

ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C, should be 0x0A (20160527/tbprint-229)
MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ 9

.. which is pointing to the MADT table being overwritten. There is only 256 bytes allocated for this table when it is being created:

 *       MADT  ->   0xf2500  (depends on #CPUs)
 *       FADT  ->   0xf2600  (268 bytes)
#define MADT_OFFSET             0x100
#define FADT_OFFSET             0x200

The MADT* for bhyve has a 44-byte fixed header, followed by a array of 8-byte 'Processor Local APIC' entries, one for each vCPU. The end of the table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source Override' entries, and a 6-byte 'Local APIC NMI' entry.

Looking at the max #CPUs that can fit into 256 bytes:

256 = 44 + N*8 + 12 + 2*10 + 6, which gives N = 21. The fact that it worked for slightly larger values is probably due to the table entries at the end being ignored, until eventually the I/O APIC table entry was corrupted by the FADT overwrite.

A quick fix to get more vCPUs is to bump the addresses of the tables in acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs.

+#define FADT_OFFSET             0x700
+#define HPET_OFFSET             0x840
+#define MCFG_OFFSET             0x880
+#define FACS_OFFSET             0x8C0
+#define DSDT_OFFSET             0x900
-#define FADT_OFFSET             0x200
-#define HPET_OFFSET             0x340
-#define MCFG_OFFSET             0x380
-#define FACS_OFFSET             0x3C0
-#define DSDT_OFFSET             0x400

I'll create a bug for this so that the overwrite will be detected at run-time, and also bump up the space to allow for some growth.



* see the ACPI spec at http://www.acpi.info/spec.htm for table details.
freebsd-virtualization@freebsd.org mailing list
To unsubscribe, send any mail to 

Reply via email to