On Tue, May 18, 2021 at 02:51:22PM -0600, John Doherty via freebsd-virtualization wrote: > I'm experimenting with bhyve VMs with more than 16 vCPUs on FreeBSD > 12.2-RELEASE. > > As I understand it, this limit is defined as VM_MAXCPU in > /usr/src/sys/amd64/include/vmm.h. I also understand that as of about two > years ago, it should be possible to increase it beyond 21[1]. > > I began working on a machine with only eight CPU cores/threads, so rather > than trying to increase VM_MAXCPU, I tried to decrease it, which seemed to > work fine. My procedure was: > > edit /usr/src/sys/amd64/include/vmm.h > cd /usr/src/lib/libvmmapi ; make clean ; make ; make install > cd /usr/src/usr.sbin/bhyve ; make clean ; make ; make install > cd /usr/src/usr.sbin/bhyvectl ; make clean ; make ; make install > cd /usr/src/usr.sbin/bhyveload ; make clean ; make ; make install > cd /usr/src ; make buildkernel KERNCONF=JLD ; make installkernel > KERNCONF=JLD > reboot > > where JLD is a kernel config file that differs from GENERIC only in the > "ident" line. > > I tried several values (1, 2, and 4) for VM_MAXCPU and verified that VMs > with a number of vCPUs less than or equal to the max booted fine and that > VMs with more didn't. > > So then I tried increasing VM_MAXCPU on a machine with two CPUs, 14 cores/28 > threads each, using the same procedure. I set VM_MAXCPU to 28 and configured > a VM with 24 vCPUs but that didn't boot. Also tried setting the max to 20 > and booting a VM with 18 vCPUs but that didn't work either. In all cases, > VMs with 16 or fewer vCPUs continued to boot fine. > > All the tests were done with VMs created before doing any of this stuff. I > am using vm-bhyve to manage VMs. The VMs in these tests were running one of > FreeBSD 12.2-RELEASE, alpine linux 3.13, and debian-testing. > > So it seems like I'm overlooking something but I don't know what, or maybe > this just isn't going to work. Anybody have experience actually doing this? > > Thanks very much for any advice or further information.
HardenedBSD bumped max vCPUs up to 48 in a recent commit[0]. Our 14-CURRENT/amd64 package building VM is set up with 30 (out of 36 available) vCPUs (`vmrun.sh -c 30`). We've completed one full package build (so, 31,000+ ports) without issue. The one package building VM we have on the system (so no other VMs) is running HardenedBSD 14-CURRENT. Meaning, we don't run anything other than HardenedBSD VMs. [0]: https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/issues/30 Thanks, -- Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
signature.asc
Description: PGP signature
