Re: How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance?
On 2014-11-12 17:31, Neel Natu wrote: > Hi Tinker, > > On Wed, Nov 12, 2014 at 2:08 PM, Tinker wrote: >> On 2014-11-12 23:55, Allan Jude wrote: >>> >>> On 2014-11-12 16:39, ti...@openmailbox.org wrote: Hi! In order justify giving energy to BHyVe, I need to know if it's "future-proof" in that the 16 vCPU limit can be increased? Please let the world know if BHyVe's 16 vCPU limit can be lifted in some way, by configuration, patch, etc. (and if you want to, why this limit is in place by default today). Thanks!! Tinker ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org" >>> >>> >>> You can increase the limit by editing sys/amd64/include/vmm.h >>> >>> #define VM_MAXCPU 16 >>> >>> From what I've been told, things scale badly above 24 CPUs. They plan to >>> solve this issue, but have not yet. If you system has enough cores to >>> support using more than 16 per VM, you can modify the file and recompile >>> the kernel and use as many CPUs as you want, but not much testing has >>> happened with bigger numbers. >> >> >> Dear Allan, >> >> Thank you very much for responding. >> >> Aha, great! >> >> >> Only for completeness, do you have any particular idea about >> * When the scaling above 24 vCPU:s will be optimized, like approx how many >> years away is it (like 1 or more than 1)?, and > > bhyve allocates memory for the maximum number of vcpus when the > virtual machine is created. This is simple to implement and also fits > in nicely with bhyve's loader model where the guest loader > (bhyveload/grub-bhyve) is separate from bhyve. > > The downside is that if VM_MAXCPU is a large number then there is a > large memory penalty for virtual machines that only use 1 or 2 vcpus. > Hence the desire to cap it at a smallish number. > > There are a few ways to deal with this: > - patch the code to change VM_MAXCPU (this is what you need to do today) > - allow the maximum vcpus to be changed via a tunable (this can be > done in short order) > - the limit can go away when bhyve moves to a single process model > because we'll know the number of vcpus at VM creation time. > >> * What the technological reason for the scaling is, is it that somehow the >> BHyVe instances on the different cores need to inter-communicate, for >> instance that all disk and network IO is done via one single core currently? >> > > Actually, the performance depends more on the workload and the guest > OS so you should definitely try how it goes for your application. > We'll be happy to help fix any issues that come up. > > best > Neel > >> >> In all cases, your response is great news, as your baseline answer that it's >> doable and only a question of optimization and tweaking of present >> functionality - >> >> Thanks again! >> >> Tinker >> Right, I forgot that at MeetBSD it was decided that making the max cpus a sysctl was a good stop-gap measure to hold us over until the redesign of bhyveload. -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance?
Hi Tinker, On Wed, Nov 12, 2014 at 2:08 PM, Tinker wrote: > On 2014-11-12 23:55, Allan Jude wrote: >> >> On 2014-11-12 16:39, ti...@openmailbox.org wrote: >>> >>> Hi! >>> >>> In order justify giving energy to BHyVe, I need to know if it's >>> "future-proof" in that the 16 vCPU limit can be increased? >>> >>> Please let the world know if BHyVe's 16 vCPU limit can be lifted in some >>> way, by configuration, patch, etc. (and if you want to, why this limit >>> is in place by default today). >>> >>> Thanks!! >>> Tinker >>> >>> ___ >>> freebsd-virtualization@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>> To unsubscribe, send any mail to >>> "freebsd-virtualization-unsubscr...@freebsd.org" >> >> >> You can increase the limit by editing sys/amd64/include/vmm.h >> >> #define VM_MAXCPU 16 >> >> From what I've been told, things scale badly above 24 CPUs. They plan to >> solve this issue, but have not yet. If you system has enough cores to >> support using more than 16 per VM, you can modify the file and recompile >> the kernel and use as many CPUs as you want, but not much testing has >> happened with bigger numbers. > > > Dear Allan, > > Thank you very much for responding. > > Aha, great! > > > Only for completeness, do you have any particular idea about > * When the scaling above 24 vCPU:s will be optimized, like approx how many > years away is it (like 1 or more than 1)?, and bhyve allocates memory for the maximum number of vcpus when the virtual machine is created. This is simple to implement and also fits in nicely with bhyve's loader model where the guest loader (bhyveload/grub-bhyve) is separate from bhyve. The downside is that if VM_MAXCPU is a large number then there is a large memory penalty for virtual machines that only use 1 or 2 vcpus. Hence the desire to cap it at a smallish number. There are a few ways to deal with this: - patch the code to change VM_MAXCPU (this is what you need to do today) - allow the maximum vcpus to be changed via a tunable (this can be done in short order) - the limit can go away when bhyve moves to a single process model because we'll know the number of vcpus at VM creation time. > * What the technological reason for the scaling is, is it that somehow the > BHyVe instances on the different cores need to inter-communicate, for > instance that all disk and network IO is done via one single core currently? > Actually, the performance depends more on the workload and the guest OS so you should definitely try how it goes for your application. We'll be happy to help fix any issues that come up. best Neel > > In all cases, your response is great news, as your baseline answer that it's > doable and only a question of optimization and tweaking of present > functionality - > > Thanks again! > > Tinker > > ___ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance?
On 2014-11-12 23:55, Allan Jude wrote: On 2014-11-12 16:39, ti...@openmailbox.org wrote: Hi! In order justify giving energy to BHyVe, I need to know if it's "future-proof" in that the 16 vCPU limit can be increased? Please let the world know if BHyVe's 16 vCPU limit can be lifted in some way, by configuration, patch, etc. (and if you want to, why this limit is in place by default today). Thanks!! Tinker ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org" You can increase the limit by editing sys/amd64/include/vmm.h #define VM_MAXCPU 16 From what I've been told, things scale badly above 24 CPUs. They plan to solve this issue, but have not yet. If you system has enough cores to support using more than 16 per VM, you can modify the file and recompile the kernel and use as many CPUs as you want, but not much testing has happened with bigger numbers. Dear Allan, Thank you very much for responding. Aha, great! Only for completeness, do you have any particular idea about * When the scaling above 24 vCPU:s will be optimized, like approx how many years away is it (like 1 or more than 1)?, and * What the technological reason for the scaling is, is it that somehow the BHyVe instances on the different cores need to inter-communicate, for instance that all disk and network IO is done via one single core currently? In all cases, your response is great news, as your baseline answer that it's doable and only a question of optimization and tweaking of present functionality - Thanks again! Tinker ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance?
On 2014-11-12 16:39, ti...@openmailbox.org wrote: > Hi! > > In order justify giving energy to BHyVe, I need to know if it's > "future-proof" in that the 16 vCPU limit can be increased? > > Please let the world know if BHyVe's 16 vCPU limit can be lifted in some > way, by configuration, patch, etc. (and if you want to, why this limit > is in place by default today). > > Thanks!! > Tinker > > ___ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" You can increase the limit by editing sys/amd64/include/vmm.h #define VM_MAXCPU 16 From what I've been told, things scale badly above 24 CPUs. They plan to solve this issue, but have not yet. If you system has enough cores to support using more than 16 per VM, you can modify the file and recompile the kernel and use as many CPUs as you want, but not much testing has happened with bigger numbers. -- Allan Jude signature.asc Description: OpenPGP digital signature
