Re: New bhyve user

2018-09-29 Thread D'Arcy Cain 
On 9/28/18 11:07 AM, D'Arcy Cain wrote:
>>> I tried to boot into a Linux install but even though I set grahics to yes, 
>>> it doesn't seem to be serving VNC.  On the console I can only get into the 
>>> live CD.  How do I get it installed?
>>
>> Did you have uefi="yes" in the configuration? Graphics are only available 
>> when using UEFI boot.
> 
> Yes.  I copied it right off the web site.

For future searches I found the problem.  I can connect to VNC OK if I
just run from a regular terminal but it fails in screen.  I am not sure
why as it seemed OK when I connected to a FreeBSD host running
vncconnect manually.

However, I have a new issue.  The install goes fine right up to near the
end when it fails with a "Executing 'grub-install /dev/vda' failed"
error and the install fails.  A second attempt says that there is a
system already on the disk but it won't boot up and leaves the console
at a grub prompt.  I have grub2-bhyve installed.  Here is my config:

guest="linux"
uefi="yes"
loader="grub"
grub_run_partition="msdos1"
cpu=2
memory=1024
network0_type="virtio-net"
network0_switch="public"
network0_mac="22:22:22:22:22:03"
disk0_type="virtio-blk"
disk0_name="disk0.img"
uuid="532fb166-c1bb-11e8-9e80-b4b52fcc4894"
graphics="yes"
xhci_mouse="yes"


-- 
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vex.net, VoIP: sip:da...@druid.net
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: New bhyve user

2018-09-28 Thread Jakub Chromy 




You seemed to have heard incorrectly.  There is little to no issues
overcommiting CPU's in bhyve, I have a 2 core, 4 thread system with
6 VM's, each vm using 1 vCPU, this is a 50% overcommit and it my
base line load.


No I have not. As far as you stick with 1 vCPU per virtual host, you 
should be fine. The problem is with multi-core VMs and spinlocks:


https://lists.freebsd.org/pipermail/freebsd-virtualization/2018-July/006613.html

quote from Alan Somers below:

An anonymous BHyve expert has explained things to me off-list.  Details
below.

On Tue, Jul 24, 2018 at 3:30 PM, Alan Somers > wrote:


/What are people's experiences with overcommitting CPUs in BHyve? I have />/an 8-core machine that often runs VMs totalling up to 5 allocated CPUs />/without problems. But today I got greedy. I assigned 8 cores to one VM />/for a big build job. Obviously, some of those were shared with the host. />/I also assigned it 8GB of RAM (out of 16 total). Build performance fell />/through the floor, even though the host was idle. Eventually I killed the />/build and restarted it with a more modest 2 make jobs (but the VM still 

had />/8 cores). Performance improved. But eventually the system seemed to be />/mostly hung, while 
I had a build job running on the host as well as in the />/VM. I killed both build jobs, which resolved 
the hung processes. Then I />/restarted the host's build alone, and my system completely hung, with 
/>/top(1) indicating that many processes were in the pfault state. />//>/So my questions are: 
/>/1) Is it a known problem to overcommit CPUs with BHyve? />//
Yes it's a problem, and it's not just BHyve.  The problem comes from stuff
like spinlocks.  Unlike normal userland locks, when two CPUs contend on a
spinlock both are running at the same time.  When two vCPUs are contending
on a spinlock, the host has no idea how to prioritize them.  Normally
that's not a problem, because physical CPUs are always supposed to be able
to run.  But when you overcommit vCPUs, some of them must get swapped out
at all times.  If a spinlock is being contended by both a running vCPU and
a swapped out vCPU, then it might be contended for a long time.  The host's
scheduler simply isn't able to fix that problem.  The problem is even worse
when you're using hyperthreading (which I am) because those eight logical
cores are really only four physical cores, and spinning on a spinlock
doesn't generate enough pipeline stalls to cause a hyperthread switch.  So
it's probably best to stick with the n - 1 rule.  Overcommitting is ok if
all guests are single-cored because then they won't use spinlocks.  But my
guests aren't all single-cored.

2) Could this be related to the pfault hang, even though the guest was idle

/at the time? />//

The expert suspects the ZFS ARC was competing with the guest for RAM.
IIUC, ZFS will sometimes greedily grow its ARC by swapping out idle parts
of the guest's RAM.  But the guest isn't aware of this behavior, and will
happily allocate memory from the swapped-out portion.  The result is a
battle between the ARC and the guest for physical RAM.  The best solution
is to limit the maximum amount of RAM used by the ARC with the
vfs.zfs.arc_max sysctl.

More info:https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222916

Thanks to everyone who commented, especially the Anonymous Coward.

-Alan




Jakub
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: New bhyve user

2018-09-28 Thread Adam 
On Fri, Sep 28, 2018 at 10:08 AM D'Arcy Cain  wrote:

> > This is the number of virtual cpus that the guest will see. Remember
> that as far as the host is concerned, the guests are processes that are
> using resources, just like any other program. A guest that is not doing
> much will not being using much cpu time on the host, and the host will
> happily run other guests (or system processes) on the same physical cpus.
>
> So if I have 16 CPUs and 8 clients, there is no problem giving them each
> 4 CPUs?  Is that the maximum that they can use?
>

The comments of this bug report contain more detail about what you can
expect.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222916

That stuff would be good in the wiki.

-- 
Adam
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: New bhyve user

2018-09-28 Thread ghislain 

It is all a mater of Load, as long as your host load stays
below the number of CPU threads you actually have things
in this aspect tend to work just fine.  And if you do exceed
this everyone slows down in a fairly fair fashion.



well virtualisation is used for a lot of things but if you want to 
isolate things you do not want a rogue VM taking all cpu so sharing 
cpu in the current state is a risk as the policeman that is the kernel 
do not seems to know bhyve guests :) (i speak conditionnal as i dont 
use it myself i am just curious).



Ghislain.



___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: New bhyve user

2018-09-28 Thread Rodney W. Grimes 
> >
> > ?So if one guest launch a lot of cpu hungry thread it could "starve" 
> > the others guests with less thread usgin cpu time. 
> 
> As far as what I heard here on this forum, you should NOT overcommit the 
> vCPUs.

You seemed to have heard incorrectly.  There is little to no issues
overcommiting CPU's in bhyve, I have a 2 core, 4 thread system with
6 VM's, each vm using 1 vCPU, this is a 50% overcommit and it my
base line load.

I frequently fire up 2 and 4 core VM's to do real work without
any issues.

Now, you do NOT want to end up in situation where these VM's
are over loading the actually avaliable CPU cycles, what I
am doing works becasue my base line 6 VM's only actually
consume about 1.5 cpus, so I actually do have 2.5 sitting
idle.

What does NOT work well is overcommiting Memory, that kinda
puts you in a real bad state.  I also stronly recommend 
using -S (wire memory) option, this insures you actually
can hard allocate any memory that a VM needs.

> 
> When you keep the number of vCPUs assigned to your vms lower than 
> physical CPUs (= hyperthreaded cores), you should be fine.

It is all a mater of Load, as long as your host load stays
below the number of CPU threads you actually have things
in this aspect tend to work just fine.  And if you do exceed
this everyone slows down in a fairly fair fashion.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: New bhyve user

2018-09-28 Thread D'Arcy Cain 
On 9/28/18 10:28 AM, Matt Churchyard wrote:
>> I created a switch and clients using the examples on the vm-bhyve web site.  
>> However, I could not get IP working until I put an IP address on the 
>> vm-public interface.  I duplicated the address of the interface that it is 
>> >connected to (re0 in my case) and used DHCP to assign addresses to the 
>> clients.  If this is the correct way, shouldn't it have happened 
>> automatically?
> 
> No, you should not be duplicating IP addresses.

Yes, it seemed wrong but...

> A virtual "switch" is really just an ethernet bridge. If you want guests to 
> be on the same lan as the host, then you can just create a virtual switch and 
> add your physical interface to it
> 
> # vm switch create public
> # vm switch add public re0
> 
> Any guest that is connected to the vm-public switch will be bridged to re0, 
> and as such to the network re0 is connected to. In that instance you would 
> give guests IP addresses on the same range as re0 (or they could get 
> addresses from your local DHCP server).

That's what I did at first but I could not make the clients talk to the
host.  I will try again.

> If you want to guests to have a separate network, then you can assign an 
> address to the virtual switch (using a different address range to the host)
> 
> # vm switch create guests
> # vm switch address guests 192.168.100.1/24

I have another installation where I will probably do this.

> In this case you would assign guests address in that network, with 
> 192.168.100.1 as the gateway. (Alternatively you could install something to 
> provide dhcp. There are guides on the vm-bhyve GitHub for using dnsmasq).
> For this to work you would either need to configure the host to perform NAT, 
> or configure the rest of your network to know that any traffic to 
> 192.168.100.0/24 should be routed to the bhyve host. (NAT is probably the 
> easier option)

I assume that this won't be necessary if I have a bank of live IPs.
even if I don't, wouldn't my gateway NAT be enough?

>> In any case, I saw I see that it can be added at creation time but how do I 
>> modify it later?  I saw "switch address a.b.c.d/xx|none" in the man page but 
>> no way to specify which switch the address should be applied to.  I tried 
>> >adding the switch name before and after the address but that gave me an 
>> error.
> 
> I've just tested the above address command and it seems to be working for 
> me...

What exactly did you type?  The man page suggests something like this:

 $ vm switch address 192.168.100.0/24

How does it know which switch to add the address to?

>> I tried to boot into a Linux install but even though I set grahics to yes, 
>> it doesn't seem to be serving VNC.  On the console I can only get into the 
>> live CD.  How do I get it installed?
> 
> Did you have uefi="yes" in the configuration? Graphics are only available 
> when using UEFI boot.

Yes.  I copied it right off the web site.

>> I am thinking of creating a base install with various install options and 
>> then copy that over to new installs as a starting point.  I was going to use 
>> rsync with the -S option to copy over the file as sparse.
>> Is there another way that is preferred?
> 
> For this sort of setup, ZFS is the obvious answer as you can use send/recv to 
> duplicate guests (or even clone to create an instant copy without using 
> additional disk space). If not using ZFS then rsync would be a reasonable 
> option to create copies of guests.

I am using ZFS but I am also new to that and I am still exploring.  That
sounds like a good idea.

>> In Xen there is a maxvcpus which limit the number of CPUs but they could 
>> baloon down if not busy so that other clients who are busy can use the CPUs. 
>>  In bhyve (at least in vm-bhyve) there is only a cpus line in the config.  
>> >Is this a minimum, maximum or is it a hard limit?
> 
> This is the number of virtual cpus that the guest will see. Remember that as 
> far as the host is concerned, the guests are processes that are using 
> resources, just like any other program. A guest that is not doing much will 
> not being using much cpu time on the host, and the host will happily run 
> other guests (or system processes) on the same physical cpus.

So if I have 16 CPUs and 8 clients, there is no problem giving them each
4 CPUs?  Is that the maximum that they can use?

Thanks.

-- 
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vex.net, VoIP: sip:da...@druid.net
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: New bhyve user

2018-09-28 Thread Jakub Chromy 


 So if one guest launch a lot of cpu hungry thread it could "starve" 
the others guests with less thread usgin cpu time. 


As far as what I heard here on this forum, you should NOT overcommit the 
vCPUs.


When you keep the number of vCPUs assigned to your vms lower than 
physical CPUs (= hyperthreaded cores), you should be fine.


--


   regards


Jakub Chromy


CGI Systems div.

CGI CZ s.r.o.
sa...@cgi.cz
775 144 257
234 697 102
www.cgi.cz




 So if one guest launch a lot of cpu hungry thread it could "starve" 
the others guests with less thread usgin cpu time.


Correct me if i read that wrong :)

Ghislain.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: New bhyve user

2018-09-28 Thread ghislain 



In Xen there is a maxvcpus which limit the number of CPUs but they could baloon 
down if not busy so that other clients who are busy can use the CPUs.  In bhyve 
(at least in vm-bhyve) there is only a cpus line in the config.  >Is this a 
minimum, maximum or is it a hard limit?

This is the number of virtual cpus that the guest will see. Remember that as 
far as the host is concerned, the guests are processes that are using 
resources, just like any other program. A guest that is not doing much will not 
being using much cpu time on the host, and the host will happily run other 
guests (or system processes) on the same physical cpus.


if i understood the other comments on the list on previous thread there is not any mecanism to prioritize guest from 
other guest and share fairly between them when they share a cpu core.


 The freebsd kernel will treat all process equaly. If i am correct the only way to have a fair distribution is to bind 
guest to a cpu but then you loose the sharing possibility if one is idle. If you let just everyone have the whole cpu 
access then there is no rule about which guest will have the cpu time as the kernel do not know how to partition the cpu 
time per guest, only per process.


 So if one guest launch a lot of cpu hungry thread it could "starve" the others 
guests with less thread usgin cpu time.

Correct me if i read that wrong :)

Ghislain.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: New bhyve user

2018-09-28 Thread Matt Churchyard 
-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of D'Arcy Cain
Sent: 28 September 2018 14:24
To: freebsd-virtualization@freebsd.org
Subject: New bhyve user

>Greetings.  I have just recently started using bhyve (previously a Xen user).  
>I am using vm-bhyve to manage it.  I have a few questions.

>First question, am I making the right choice by switching and, if so, is bhyve 
>the right choice to switch to?  I realize that that is an impossible question 
>but perhaps some pros and cons as well as war stories will help me.

It really depends on how well bhyve supports what you need to do. The main 
downside of bhyve at the moment is that it's very new and lacks many features 
and performance optimisations you may find in more established hypervisors. I 
can't really say yes you should use bhyve or no you should stick with Xen.

>I created a switch and clients using the examples on the vm-bhyve web site.  
>However, I could not get IP working until I put an IP address on the vm-public 
>interface.  I duplicated the address of the interface that it is >connected to 
>(re0 in my case) and used DHCP to assign addresses to the clients.  If this is 
>the correct way, shouldn't it have happened automatically?

No, you should not be duplicating IP addresses.

A virtual "switch" is really just an ethernet bridge. If you want guests to be 
on the same lan as the host, then you can just create a virtual switch and add 
your physical interface to it

# vm switch create public
# vm switch add public re0

Any guest that is connected to the vm-public switch will be bridged to re0, and 
as such to the network re0 is connected to. In that instance you would give 
guests IP addresses on the same range as re0 (or they could get addresses from 
your local DHCP server).

If you want to guests to have a separate network, then you can assign an 
address to the virtual switch (using a different address range to the host)

# vm switch create guests
# vm switch address guests 192.168.100.1/24

In this case you would assign guests address in that network, with 
192.168.100.1 as the gateway. (Alternatively you could install something to 
provide dhcp. There are guides on the vm-bhyve GitHub for using dnsmasq).
For this to work you would either need to configure the host to perform NAT, or 
configure the rest of your network to know that any traffic to 192.168.100.0/24 
should be routed to the bhyve host. (NAT is probably the easier option)

>In any case, I saw I see that it can be added at creation time but how do I 
>modify it later?  I saw "switch address a.b.c.d/xx|none" in the man page but 
>no way to specify which switch the address should be applied to.  I tried 
>>adding the switch name before and after the address but that gave me an error.

I've just tested the above address command and it seems to be working for me...

>I tried to boot into a Linux install but even though I set grahics to yes, it 
>doesn't seem to be serving VNC.  On the console I can only get into the live 
>CD.  How do I get it installed?

Did you have uefi="yes" in the configuration? Graphics are only available when 
using UEFI boot.

>I am thinking of creating a base install with various install options and then 
>copy that over to new installs as a starting point.  I was going to use rsync 
>with the -S option to copy over the file as sparse.
>Is there another way that is preferred?

For this sort of setup, ZFS is the obvious answer as you can use send/recv to 
duplicate guests (or even clone to create an instant copy without using 
additional disk space). If not using ZFS then rsync would be a reasonable 
option to create copies of guests.

>In Xen there is a maxvcpus which limit the number of CPUs but they could 
>baloon down if not busy so that other clients who are busy can use the CPUs.  
>In bhyve (at least in vm-bhyve) there is only a cpus line in the config.  >Is 
>this a minimum, maximum or is it a hard limit?

This is the number of virtual cpus that the guest will see. Remember that as 
far as the host is concerned, the guests are processes that are using 
resources, just like any other program. A guest that is not doing much will not 
being using much cpu time on the host, and the host will happily run other 
guests (or system processes) on the same physical cpus.

>That's it for now.  Thanks for any help.

>-- 
>D'Arcy J.M. Cain  |  Democracy is three wolves
>http://www.druid.net/darcy/|  and a sheep voting on
>+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
>IM: da...@vex.net, VoIP: sip:da...@druid.net 
>___
>freebsd-virtualization@freebsd.org mailing list 
>https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>To unsubscribe, send any mail to 
>"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list