Re: bhyve and contention
On 15/02/2018 05:40, Rodney W. Grimes wrote: > [ Charset UTF-8 unsupported, converting... ] >> On 14/02/2018 10:18, Peter Grehan wrote: >> Be sure not to pass the -S option to bhyve as it wires the guest memory. >> This was seen recently as sysutils/chyves uses it by default. > > This should be evaluated, if you do not pass -S you run into the > possible performance impact of swapping guest memory, which does > infact allow you more over commit, but at a pretty large > performance penality. The original question specified having 5 guests with 8G each on a host with 32G, so overcommitting ram was expected, wiring each guest in this setup will cause panics not swapping. > You need swap space if you are going to not pass -S and over commit, > if you do not have enough swap space for the over commit your system > runs out of swap and usually ends up killing random processes, and > random usually means the most important things you are running! The host has 64G swap setup. -- FreeBSD - the place to B...Software Developing Shane Ambler ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve and contention
[ Charset UTF-8 unsupported, converting... ] > On 14/02/2018 10:18, Peter Grehan wrote: > >>> 2. > >>> In the following context, the server is the same but this time all five > >>> guests have -c 4 per guest, so bhyve is asking 12 more cores than that > >>> existing in hardware. Does the guest fail to load, do either guest or > >>> server crash? > >> > >> The is core over commit, very common in the virtualization world, > >> bhyve does its best effort to give the guests cores as needed. > > > > ?To add to what Rod said - bhyve uses a thread for each vCPU. It's up to > > the FreeBSD scheduler to determine where/when these threads run. > > > > ?It is possible for a guest to fail, for example if a spinlock times out > > due to vCPUs not being able to run to release a lock. This is a general > > problem with virtualization and can occur even on VMWare ESXi with heavy > > oversubscription. > > There are some bhyve options you need to check - > > You want to use the -H option which prevents the guest keeping 100% cpu. Very true, this yields the vCPU when a halt is executed. > Be sure not to pass the -S option to bhyve as it wires the guest memory. > This was seen recently as sysutils/chyves uses it by default. This should be evaluated, if you do not pass -S you run into the possible performance impact of swapping guest memory, which does infact allow you more over commit, but at a pretty large performance penality. You need swap space if you are going to not pass -S and over commit, if you do not have enough swap space for the over commit your system runs out of swap and usually ends up killing random processes, and random usually means the most important things you are running! > https://www.mail-archive.com/freebsd-virtualization@freebsd.org/msg05665.html > > If you use zfs, you may want to set the arc_max. This is not a may, if you use ZFS you must restrict its buffer cache, or expect unstable systems. ZFS's default situation is to use up to 95% of memory, that leaves almost nothing for a running VM. > One thing that I have noticed, as a guest allocates ram, the host > allocates ram to its bhyve instance, but when the guest releases it, the > host is unaware of the ram now being free. Apparently there is work on > getting the guest to notify the host of released ram. Until that is > available you may want to restart guests that temporarily allocate large > amounts of ram. Correct, we need an implementation of what is called a memory ballon driver so that the guest can pass back to the host un used memory. > -- > FreeBSD - the place to B...Sharing Devices > Shane Ambler -- Rod Grimes rgri...@freebsd.org ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve and contention
Thanks everyone for answering my query -- J. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve and contention
On 14/02/2018 10:18, Peter Grehan wrote: >>> 2. >>> In the following context, the server is the same but this time all five >>> guests have -c 4 per guest, so bhyve is asking 12 more cores than that >>> existing in hardware. Does the guest fail to load, do either guest or >>> server crash? >> >> The is core over commit, very common in the virtualization world, >> bhyve does its best effort to give the guests cores as needed. > > To add to what Rod said - bhyve uses a thread for each vCPU. It's up to > the FreeBSD scheduler to determine where/when these threads run. > > It is possible for a guest to fail, for example if a spinlock times out > due to vCPUs not being able to run to release a lock. This is a general > problem with virtualization and can occur even on VMWare ESXi with heavy > oversubscription. There are some bhyve options you need to check - You want to use the -H option which prevents the guest keeping 100% cpu. Be sure not to pass the -S option to bhyve as it wires the guest memory. This was seen recently as sysutils/chyves uses it by default. https://www.mail-archive.com/freebsd-virtualization@freebsd.org/msg05665.html If you use zfs, you may want to set the arc_max. One thing that I have noticed, as a guest allocates ram, the host allocates ram to its bhyve instance, but when the guest releases it, the host is unaware of the ram now being free. Apparently there is work on getting the guest to notify the host of released ram. Until that is available you may want to restart guests that temporarily allocate large amounts of ram. -- FreeBSD - the place to B...Sharing Devices Shane Ambler ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve and contention
2. In the following context, the server is the same but this time all five guests have -c 4 per guest, so bhyve is asking 12 more cores than that existing in hardware. Does the guest fail to load, do either guest or server crash? The is core over commit, very common in the virtualization world, bhyve does its best effort to give the guests cores as needed. To add to what Rod said - bhyve uses a thread for each vCPU. It's up to the FreeBSD scheduler to determine where/when these threads run. It is possible for a guest to fail, for example if a spinlock times out due to vCPUs not being able to run to release a lock. This is a general problem with virtualization and can occur even on VMWare ESXi with heavy oversubscription. That being said, there is certainly scope to provide more information to the FreeBSD scheduler so it can make better decisions when running VM guests. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: bhyve and contention
> Hello virtualizations, > > Please can anyone tell me what happens in the following scenarios: I can speculate on some of what may happen, this may not be exact, but should give you an idea. > 1. > freebsd-11 server, 5x freebsd-11 guests > The server has 32GB ram installed. it's an i7 so 8 cores. There's 64GB > swap. Each guest that bhyve loads is with m 8192M, so ram is > overcommitted by 6GB. 5 x 8G == 40G, 32G ram, so 8G overcommit, not 6. Further more you have base sytem memory which includes such things as kernel, page tables, loaded modules, buffer cache, possibly ZFS arc cache since you dont mention that. ZFS arc cache will attempt to grow to something like 95% of memory, so you can actually be in full 40G over commit situation on memory. > What happens to the guests and the server? Do either (or both) crash? Or > is there something in bhyve that will make memory allocated on a > best-effort basis? With 64G of swap space avaliable, and assuming your either not running ZFS or have limited zfs arc cache to say 2G or 4G the system should not crash. You however are in a memory overcommit and things are likely to be sluggish and may even appear crashed. Bhyve uses swap backed annonymous vm so the vm system shall apply best effort to meet the demands. Swapping vm guest memory pages though not idea does work, but as I said, things are going to get very sluggish both on the host and the guest. > 2. > In the following context, the server is the same but this time all five > guests have -c 4 per guest, so bhyve is asking 12 more cores than that > existing in hardware. Does the guest fail to load, do either guest or > server crash? The is core over commit, very common in the virtualization world, bhyve does its best effort to give the guests cores as needed. I am typing this on a system with a 4 core CPU (2 real x 2 thread) that is running 10 bhyve VM's, each just single core. My host load average is: 6:01PM up 3 days, 16:01, 7 users, load averages: 3.77, 3.34, 3.22 One of those guests is running a stable/10 make world in a memory contrained situation (512MB). The host system is responsive and working fine, as are the guests. I am fine with CPU over commit, but I watch the memory situation like a hawk. I only have 12GB of memory, I have restricted ZFS arc cache to 6G The sums of the memory of all the running VM's is about 2G This leaves me about 4G of fiddle around room on the host for occasional VM spin up, or other memory heavy things. I run all my VM's with the wired memory option. > > thanks, > -- > J. > ___ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org" > -- Rod Grimes rgri...@freebsd.org ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
bhyve and contention
Hello virtualizations, Please can anyone tell me what happens in the following scenarios: 1. freebsd-11 server, 5x freebsd-11 guests The server has 32GB ram installed. it's an i7 so 8 cores. There's 64GB swap. Each guest that bhyve loads is with m 8192M, so ram is overcommitted by 6GB. What happens to the guests and the server? Do either (or both) crash? Or is there something in bhyve that will make memory allocated on a best-effort basis? 2. In the following context, the server is the same but this time all five guests have -c 4 per guest, so bhyve is asking 12 more cores than that existing in hardware. Does the guest fail to load, do either guest or server crash? thanks, -- J. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
