RE: FreeBSD bhyve guest does not start anymore

[Matt Churchyard]

-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of Andrea Venturoli
Sent: 26 April 2021 12:36
To: freebsd-virtualization@freebsd.org
Subject: FreeBSD bhyve guest does not start anymore

>Hello.

>I've asked about this in the past (around January) and somehow I got some 
>suggestions.
>Either that helped me solve the problem, but now it's back, or I never 
>rebooted the host and simply forgot about it.
>In any case...

>Due to a 12.x specific bug, I run a 11.3 guest using vm-bhyve.
This is its config:
> loader="bhyveload"
> cpu=1
> memory=512M
> network0_type="virtio-net"
> network0_switch="public"
> disk0_type="virtio-blk"
> disk0_name="disk0"
> disk0_dev="sparse-zvol"
> uuid="..."
> network0_mac="..."

>When I start it, the guest boots, but hangs after the kernel is loaded with:
> Manual root filesystem specification:
>   : [options]
>   Mount  using filesystem 
>   and with the specified (optional) option list.
> 
> eg. ufs:/dev/da0s1a
> zfs:tank
> cd9660:/dev/cd0 ro
>   (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)
> 
>   ?   List valid disk boot devices
>   .   Yield 1 second (for background tasks)
>   Abort manual input
> 
> mountroot> 

>Then, if I just type "ufs:/dev/vtbd0a" it will start correctly.

>As can be seen from the config: the host is running with ZFS, the guest only 
>has a zvol disk and from its POV there's only one partition (UFS).


>Volmode is set to dev (as was suggested in January):
>  zfs get volmode zroot/vm/f11b/disk0
> NAME PROPERTY  VALUESOURCE
> zroot/vm/f11b/disk0  volmode   dev  local

>Any other hint?

What does /etc/fstab look like inside the guest?

>  bye & Thanks
>   av.
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve current windows status

[Matt Churchyard]

Hello,

I'm after some general information on the current status/best practises 
for Windows on bhyve. Not entirely the correct place for this but then 
at the moment no-one else seems to really know the answers. Maybe I can 
help some of the other people who are just as unclear as me on what is 
actually the best information at this point.


What are the current recommended devices/options for Windows (2019 
server in my case) - especially with ZFS. Should I be specifying a 
512/4096 sector/block size via bhyve and/or zfs? I assume nvme & 
virtio-net are the current best options but is there a preferred virtio 
driver version. Are any of the other virtio drivers of any use to be 
installed or just the network drivers?


Are there any known problems with applications like AD/Exchange? I know 
that SQL 2012 had massive storage overhead issues on ZFS due to 512 byte 
writes, but I'm not sure if that still affects newer versions or other 
applications?


The system I am currently using is a Xeon E5-2670, which I know was 
terrible before the TPR commit. My test system seems to run reasonably 
on 12.2 (although I'd be intruiged to compare against ESXi if I had the 
time), but do you think I would expect to see any significant gains by 
using a CPU with APICv? (not that I expect anyone has done any 
benchmarking of this)


Are there any other changes in being worked on that are likely to have 
an impact on support or performance? I believe quite a bit of work is 
being done on the UEFI firmware but I expect that doesn't really affect 
much other than the boot process. I'm sure I saw reference to the devs 
having regular bhyve calls, but I have little idea what is currently 
being worked on.


Thanks for any replies,
Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Warm Migration feature for bhyve - review on Phabricator

[Matt Churchyard]

-Original Message-
From: Elena Mihailescu  
Sent: 25 January 2021 14:25
To: Matt Churchyard 
Cc: John-Mark Gurney ; freebsd-virtualization@freebsd.org
Subject: Re: Warm Migration feature for bhyve - review on Phabricator

On Mon, 25 Jan 2021 at 13:26, Matt Churchyard  
wrote:
>
> -Original Message-
> From: John-Mark Gurney 
> Sent: 25 January 2021 06:21
> To: Matt Churchyard 
> Cc: Elena Mihailescu ; 
> freebsd-virtualization@freebsd.org
> Subject: Re: Warm Migration feature for bhyve - review on Phabricator
>
> Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +:
> > > Hello, all,
> >
> > > We have recently opened a review on Phabricator for the warm migration 
> > > code for > bhyve [1]. Please take a look and let us know if it is 
> > > anything we can improve.
> >
> > > [1] https://reviews.freebsd.org/D28270
> >
> > > Thank you,
> > > Elena
> >
> > I appreciate that this isn't really related to the current review, 
> > and commend the work being put into bhyve - it's an invaluable 
> > addition to FreeBSD. I'm just wondering if any thought has been put 
> > into the future possibility for transfer of disk data during 
> > migration (i.e. the equivalent of "storage vmotion")
> >
> > The current process (from a mile high overview) effectively seems to 
> > be the following -
> >
> > * Start guest on host2 pointing at a shared disk, and halt any 
> > execution
> > * use bhyvectl to pause and begin migration on host1
> > * Start the guest on host2
> >
> > What would be the feasibility of being able to run a process such as 
> > the following? Obviously it would likely need to be orchestrated by 
> > an external tool, but to me it seems the main requirement is really 
> > just to be able to provide separate control over the pause and 
> > migrate steps on host1 -
> >
> > * send a ZFS snapshot of the running machine to host2
> > * start the guest in migrate recv mode on host2
> > * pause the guest on host1
> > * send a new snapshot
> > * initiate the migration of memory/device data
> > * start guest on host2
> >
> > Are there any major complications here I'm not aware of other than the 
> > requirement to pause the guest and kick off the state migration as two 
> > separate calls?
>
> > There's also hastd which can aid with this...
>
> Thanks for the reply. I've always been wary of the additional complexity of 
> HAST and ZFS, as it doesn't seem to have widespread usage or support, and 
> things get ugly fast when storage is involved.
>
> However, the idea of using HAST on top of zvols to provide network mirrored 
> storage for a guest is interesting. It adds a lot of extra complexity, and 
> probably performance impact though if it's just for the ability to move a 
> guest between systems that may only happen every now and then. I'm also not 
> sure it would help (or would at least be even more complex) if I have 4 hosts 
> and want to be able to move guests anywhere.
>
> The main reason for the email was to explore my theory that, by 
> leveraging ZFS, any bhyve user could roll their own storage migration 
> ability with a few commands as long as the following two (seemingly 
> minor) abilities were present
>
> 1) the ability to suspend a guest without it being done as part of the 
> migrate call. (I assume suspend/resume support via bhyvectl is planned 
> anyway, if not already in place at this point)
> 2) modification of the migrate call to skip suspend if the guest is already 
> suspended.
>
> The main thing I'm not sure of is whether the migrate call has any specific 
> reliance on doing the suspend itself (e.g. if it needs to do anything before 
> the suspend, which will obviously be problematic if suspend & migrate are 
> called separately). Or if there's something else I missed that means this is 
> not feasible. I'm not really after massive changes to the current review to 
> implement disk migration in bhyve itself.

> Thank you for your input related to the disk migration. We were (still are, 
> actually) focusing on the live migration feature for bhyve and did not take 
> into consideration the disk migration. As > Mihai said, the patches for the 
> two (guest's state and guest's disk migration) are or will be quite big by 
> themselves and we want the review process to go as smoothly as possible.

> After we have a pretty clear view of the live migration implementation (a 
> patch on this feature will be uploaded on Phabricator in a couple of weeks) 
> and how we should improve it, we will  look into the disk migration feature 

RE: Warm Migration feature for bhyve - review on Phabricator

[Matt Churchyard]

-Original Message-
From: Miroslav Lachman <000.f...@quip.cz> 
Sent: 25 January 2021 10:37
To: Matt Churchyard 
Subject: Re: Warm Migration feature for bhyve - review on Phabricator

On 22/01/2021 11:09, Matt Churchyard wrote:

[...]

> Shared storage is great but becomes complex and expensive if you want high 
> performance and reliability (seeing as the storage quickly becomes a major 
> single point of failure without enterprise active/active kit). I suspect the 
> most common deployment of bhyve is independent hosts with local ZFS pools as 
> this is easy/cheap and gives great performance. Most hypervisors only had 
> "shared storage" migration for a long time but the big ones now also support 
> transferring disk data live. It would be great to be able to do this out of 
> the gate.
> 
> I did have a poor mans version of this in vm-bhyve, but obviously relied on 
> stopping and restarting the guest. I always had problems with the nc commands 
> not exiting cleanly though and hanging the process so never made it official.

>I don't know much details about your setup and your problems with nc
> (netcat) but I guess it is used for zfs send & receive. You can try to use 
> mbuffer from ports instead of nc:
> https://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/

> It will add some dependency on other port but if somebody wants storage 
> migration with your greate tool vm-bhyve I think it is small price to pay. 
> (another way is to properly setup SSH and > do not use nc nor mbuffer)

Thanks for the reply. Yes, I was using it to send zfs snapshots between hosts. 
I think I actually have the same issue when doing it by hand. The nc call seems 
like it hangs once complete but will instantly return to the prompt when 
pressing enter.

As you say, a better solution would be to use ssh. It may even be possible to 
trigger the receive command directly from the sending host if I get a bit hacky 
and split off a second background task. I may have another look at it. It 
worked pretty well all in all even if it did have to stop/start the guest 
briefly.

Matt

> Kind regards
> Miroslav Lachman
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Warm Migration feature for bhyve - review on Phabricator

[Matt Churchyard]

-Original Message-
From: John-Mark Gurney  
Sent: 25 January 2021 06:21
To: Matt Churchyard 
Cc: Elena Mihailescu ; 
freebsd-virtualization@freebsd.org
Subject: Re: Warm Migration feature for bhyve - review on Phabricator

Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +:
> > Hello, all,
> 
> > We have recently opened a review on Phabricator for the warm migration code 
> > for > bhyve [1]. Please take a look and let us know if it is anything we 
> > can improve.
> 
> > [1] https://reviews.freebsd.org/D28270
> 
> > Thank you,
> > Elena
> 
> I appreciate that this isn't really related to the current review, and 
> commend the work being put into bhyve - it's an invaluable addition to 
> FreeBSD. I'm just wondering if any thought has been put into the future 
> possibility for transfer of disk data during migration (i.e. the equivalent 
> of "storage vmotion")
> 
> The current process (from a mile high overview) effectively seems to be the 
> following -
> 
> * Start guest on host2 pointing at a shared disk, and halt any execution
> * use bhyvectl to pause and begin migration on host1
> * Start the guest on host2
> 
> What would be the feasibility of being able to run a process such as the 
> following? Obviously it would likely need to be orchestrated by an external 
> tool, but to me it seems the main requirement is really just to be able to 
> provide separate control over the pause and migrate steps on host1 -
> 
> * send a ZFS snapshot of the running machine to host2
> * start the guest in migrate recv mode on host2
> * pause the guest on host1
> * send a new snapshot
> * initiate the migration of memory/device data
> * start guest on host2
> 
> Are there any major complications here I'm not aware of other than the 
> requirement to pause the guest and kick off the state migration as two 
> separate calls?

> There's also hastd which can aid with this...

Thanks for the reply. I've always been wary of the additional complexity of 
HAST and ZFS, as it doesn't seem to have widespread usage or support, and 
things get ugly fast when storage is involved.

However, the idea of using HAST on top of zvols to provide network mirrored 
storage for a guest is interesting. It adds a lot of extra complexity, and 
probably performance impact though if it's just for the ability to move a guest 
between systems that may only happen every now and then. I'm also not sure it 
would help (or would at least be even more complex) if I have 4 hosts and want 
to be able to move guests anywhere.

The main reason for the email was to explore my theory that, by leveraging ZFS, 
any bhyve user could roll their own storage migration ability with a few 
commands as long as the following two (seemingly minor) abilities were present

1) the ability to suspend a guest without it being done as part of the migrate 
call. (I assume suspend/resume support via bhyvectl is planned anyway, if not 
already in place at this point)
2) modification of the migrate call to skip suspend if the guest is already 
suspended.

The main thing I'm not sure of is whether the migrate call has any specific 
reliance on doing the suspend itself (e.g. if it needs to do anything before 
the suspend, which will obviously be problematic if suspend & migrate are 
called separately). Or if there's something else I missed that means this is 
not feasible. I'm not really after massive changes to the current review to 
implement disk migration in bhyve itself.

Matt

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Warm Migration feature for bhyve - review on Phabricator

[Matt Churchyard]

> Hello, all,

> We have recently opened a review on Phabricator for the warm migration code 
> for > bhyve [1]. Please take a look and let us know if it is anything we can 
> improve.

> [1] https://reviews.freebsd.org/D28270

> Thank you,
> Elena

I appreciate that this isn't really related to the current review, and commend 
the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm 
just wondering if any thought has been put into the future possibility for 
transfer of disk data during migration (i.e. the equivalent of "storage 
vmotion")

The current process (from a mile high overview) effectively seems to be the 
following -

* Start guest on host2 pointing at a shared disk, and halt any execution
* use bhyvectl to pause and begin migration on host1
* Start the guest on host2

What would be the feasibility of being able to run a process such as the 
following? Obviously it would likely need to be orchestrated by an external 
tool, but to me it seems the main requirement is really just to be able to 
provide separate control over the pause and migrate steps on host1 -

* send a ZFS snapshot of the running machine to host2
* start the guest in migrate recv mode on host2
* pause the guest on host1
* send a new snapshot
* initiate the migration of memory/device data
* start guest on host2

Are there any major complications here I'm not aware of other than the 
requirement to pause the guest and kick off the state migration as two separate 
calls?

Shared storage is great but becomes complex and expensive if you want high 
performance and reliability (seeing as the storage quickly becomes a major 
single point of failure without enterprise active/active kit). I suspect the 
most common deployment of bhyve is independent hosts with local ZFS pools as 
this is easy/cheap and gives great performance. Most hypervisors only had 
"shared storage" migration for a long time but the big ones now also support 
transferring disk data live. It would be great to be able to do this out of the 
gate.

I did have a poor mans version of this in vm-bhyve, but obviously relied on 
stopping and restarting the guest. I always had problems with the nc commands 
not exiting cleanly though and hanging the process so never made it official.

Regards,
Matt

___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Cannot run FreeBSD 11 as a vm-bhyve guest under FreeBSD 12

[Matt Churchyard via freebsd-virtualization]

Hello.

I'd like to run a FreeBSD 11.3 VM under FreeBSD 12.1.
As I'm already using vm-bhyve to run a Windows guest, I think the best (or 
possibly only) choice is to use bhyve for the FreeBSD guest too.

I've got the following conf file:

> guest="freebsd"
> loader="bhyveload"
> cpu=1
> memory=512M
> network0_type="virtio-net"
> network0_switch="public"
> disk0_type="virtio-blk"
> disk0_name="disk0"
> disk0_dev="sparse-zvol"

Running "vm install -f f11b FreeBSD-11.3-RELEASE-amd64-disc1.iso", will boot 
the guest, but it will hang after a while with:

> /boot/kernel/kernel text=0x1564b08 data=0x145330+0x4cdf30 
> syms=[0x8+0x16daf0+0x8+0x186a43] Booting...
 > |

"vm stop" will stop it (through an ACIP shutdown) after a while.



So I tried with UEFI; here's the config file:
> guest="freebsd"
> loader="uefi" 
>   
> cpu=1
> memory=512M
> network0_type="virtio-net"
> network0_switch="public"
> disk0_type="virtio-blk"
> disk0_name="disk0"
> disk0_dev="sparse-zvol"
> disk1_type="ahci-cd"
> disk1_dev="custom"
> disk1_name="/zroot/vm/.iso/FreeBSD-11.3-RELEASE-amd64-disc1.iso"
> graphics="yes"
> xhci_mouse="yes"
> graphics_listen="192.168.xxx.1"

Starting it with "vm start" and connecting through VNC, it goes through the 
same stages as above, but ends up with a blank screen with a white block cursor 
in the first column of the first row.



>Any hint on how to get past this?

This is likely caused by an issue in the way vm-bhyve handled stdio, which only 
worked correctly with older builds of bhyve.

I would recommend installing vm-bhyve from ports (you should get version 
1.4.2), which hopefully will boot using bhyveload with no issues.

Matt


 > bye & Thanks
>   av.
___
>>freebsd-virtualization@freebsd.org mailing list 
>>https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>To unsubscribe, send any mail to 
>"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

System (ps/top) & bhyvectl memory usage differences

[Matt Churchyard via freebsd-virtualization]

Hello,

Is there a reason why the resident memory used by a bhyve guest is quite 
different when comparing ps/top & bhyvectl?
Does bhyvectl take in account something in kernel space that top/ps doesn't see?

USER   PID %CPU %MEM VSZRSS TT  STAT STARTED  TIME COMMAND
root 12670  0.1  1.4 2120984 951764  1  SC+  22Jan20 157:29.22 bhyve: smtp-a 
(bhyve)

# bhyvectl --get-stats --vm=smtp-a | grep Res
Resident memory 1439875072

1.4G vs 925M

-

Another thing I've just remembered, which is probably unrelated to the above 
but still concerning memory.
I have a guest with 2G memory allocated, and dmesg lists 2048MB real memory. 
The real & avail figures are also quite close which resembles the output I 
generally expect on real hardware.

Hypervisor: Origin = "bhyve bhyve "
real memory  = 2147483648 (2048 MB)
avail memory = 2043318272 (1948 MB)

However, I have a guest with 5G allocated, and get the following in dmesg -

Hypervisor: Origin = "bhyve bhyve "
real memory  = 6442450944 (6144 MB)
avail memory = 5141663744 (4903 MB)

bhyveload -m 5G ...
bhyve -c 2 -m 5G -AHP ...

I haven't tested where it seems to change. My only theory is that it could 
possibly be something to do with crossing the old 32bit limit?

Regards,
Matt Churchyard

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: bhyve issues on Dell C6220 node

[Matt Churchyard via freebsd-virtualization]

> Hi,

> >2016 is slow (even slower doing windows updates).  2019 is much better.  A 
> >tip I have found is a minimum of 1 cpu, 2 cores and 4 threads to get decent 
> >speed from that OS (more CPUs in bhyve >tends to make performance worse - in 
> >my observations - in 2016).  Also, use the Virtio collection from RedHat for 
> >vionet and viostor.  We are currently using 0.1.171 without issue.  The ahci 
> >>emulation in itself is extremely slow.  NVMe and virtio is really the only 
> >way to go.
> 
> Is there anything else I can check here? I haven’t got round to testing 
> networking yet but I’m using nvme for the disk.
> It’s basically unusable and there is no way I could put anything production 
> on it. Just highlighting an icon on the desktop takes several seconds.

>Windows is slow when running on Intel CPUs that don't support APICv.
>That are (nearly?) all desktop CPUs, all Xeons before Sandy Bridge and some 
>Xeons after it. The problem is that Bhyve doesn't implement TPR shadowing. I'm 
>currently working on it. The review can >be found
>here: https://reviews.freebsd.org/D22942 The speedup is about factor 6!

>I've received some feedback in a private mail, a second version that adds TPR 
>thresholds can be found in my Github branch here:
>  https://github.com/Yamagi/freebsd/commits/wip/tpr_shadowing 

>A backport to 12.1 (the branch also includes the Intel SpeedShift patches from 
>https://reviews.freebsd.org/D18028) is here:
>https://github.com/Yamagi/freebsd/commits/production/12.1

>I've applied it to 2 of my production servers about 4 hours ago. Looks good so 
>far. I'll update the review when I'm sure that it doesn't break anything, 
>maybe early next week.

I've been running the patch for a few days now and not seen any problems. It's 
a complete difference compared to running stock 12.1.
Thanks for your response and awesome work.

Don't know if any more work on it is required but it would be nice if this made 
it into 12.2 as it's much easier for me to track the binary releases than have 
to rebuild the entire system. This fairly small patch makes a massive 
difference to Windows performance on something that doesn't have APICv

I also found the fix for my original console issue (actually someone else fixed 
it, I just hadn't seen their pull request). I'm using vm-bhyve which originally 
redirected bhyve output to a log file to catch any error output. Changes to the 
way bhyve opens stdin & stderr caused this to stop working correctly.

Regards,
Matt

>Regards,
>Yamagi

--
Homepage: https://www.yamagi.org
Github:   https://github.com/yamagi
GPG:  0x1D502515
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve issues on Dell C6220 node

[Matt Churchyard via freebsd-virtualization]

> On 9 Jan 2020, at 18:58, Yamagi Burmeister  wrote:
> 
> Hi,
> 
>>> 2016 is slow (even slower doing windows updates).  2019 is much better.  A 
>>> tip I have found is a minimum of 1 cpu, 2 cores and 4 threads to get decent 
>>> speed from that OS (more CPUs in bhyve >tends to make performance worse - 
>>> in my observations - in 2016).  Also, use the Virtio collection from RedHat 
>>> for vionet and viostor.  We are currently using 0.1.171 without issue.  The 
>>> ahci >emulation in itself is extremely slow.  NVMe and virtio is really the 
>>> only way to go.
>> Is there anything else I can check here? I haven’t got round to testing 
>> networking yet but I’m using nvme for the disk.
>> It’s basically unusable and there is no way I could put anything production 
>> on it. Just highlighting an icon on the desktop takes several seconds.
> 
> Windows is slow when running on Intel CPUs that don't support APICv.
> That are (nearly?) all desktop CPUs, all Xeons before Sandy Bridge
> and some Xeons after it. The problem is that Bhyve doesn't implement
> TPR shadowing. I'm currently working on it. The review can be found
> here: https://reviews.freebsd.org/D22942 The speedup is about factor
> 6!
> 
> I've received some feedback in a private mail, a second version that
> adds TPR thresholds can be found in my Github branch here:
> https://github.com/Yamagi/freebsd/commits/wip/tpr_shadowing 
> 
> A backport to 12.1 (the branch also includes the Intel SpeedShift
> patches from https://reviews.freebsd.org/D18028) is here:
> https://github.com/Yamagi/freebsd/commits/production/12.1
> 
> I've applied it to 2 of my production servers about 4 hours ago. Looks
> good so far. I'll update the review when I'm sure that it doesn't break
> anything, maybe early next week.

I’ve had a system with the TPR changes applied for a few hours now and it seems 
to be ok. The install took less than 10 minutes compared to 40+ minutes before 
and is much more useable.

I’ll leave it running over the weekend and see how it goes.

Regards
Matt


> 
> Regards,
> Yamagi
> 
> -- 
> Homepage: https://www.yamagi.org
> Github:   https://github.com/yamagi
> GPG:  0x1D502515
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: bhyve issues on Dell C6220 node

[Matt Churchyard via freebsd-virtualization]

Hi,

> >2016 is slow (even slower doing windows updates).  2019 is much better.  A 
> >tip I have found is a minimum of 1 cpu, 2 cores and 4 threads to get decent 
> >speed from that OS (more CPUs in bhyve >tends to make performance worse - in 
> >my observations - in 2016).  Also, use the Virtio collection from RedHat for 
> >vionet and viostor.  We are currently using 0.1.171 without issue.  The ahci 
> >>emulation in itself is extremely slow.  NVMe and virtio is really the only 
> >way to go.
> 
> Is there anything else I can check here? I haven’t got round to testing 
> networking yet but I’m using nvme for the disk.
> It’s basically unusable and there is no way I could put anything production 
> on it. Just highlighting an icon on the desktop takes several seconds.

>Windows is slow when running on Intel CPUs that don't support APICv.
>That are (nearly?) all desktop CPUs, all Xeons before Sandy Bridge and some 
>Xeons after it. The problem is that Bhyve doesn't implement TPR shadowing. I'm 
>>currently working on it. The review can be found
>here: https://reviews.freebsd.org/D22942 The speedup is about factor 6!

>I've received some feedback in a private mail, a second version that adds TPR 
>thresholds can be found in my Github branch here:
>  https://github.com/Yamagi/freebsd/commits/wip/tpr_shadowing 

>A backport to 12.1 (the branch also includes the Intel SpeedShift patches from 
>https://reviews.freebsd.org/D18028) is here:
>>https://github.com/Yamagi/freebsd/commits/production/12.1

>I've applied it to 2 of my production servers about 4 hours ago. Looks good so 
>far. I'll update the review when I'm sure that it doesn't break anything, 
>maybe >early next week.

Thanks Yamagi,

That's brilliant and provides me some hope that I may not have to completely 
abandon windows guests after all...

These servers are just sat next to my desk in testing at the moment so I'll see 
if I can have a go at applying the patches and testing it in the morning.

Regards,
Matt 
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: bhyve issues on Dell C6220 node

[Matt Churchyard via freebsd-virtualization]

On Thu, 9 Jan 2020 at 08:37, Rodney W. Grimes 
mailto:freebsd-...@gndrsh.dnsmgr.net>> wrote:
>
> Loading kernel...
> /boot/kernel/kernel text=0x168fdf1 data=0x1d0a68+0x768d80 
> syms=[0x8+0x178bc0+0x8+0x1969d5]
> Loading configured modules...
> can't find '/boot/entropy'
> /
> "
>
> I get either a "|" or "/" character, then nothing else.

My experience has been when I see this it is a "wrong console" issue, ie the 
kernel
has decided to use something else for a console and your output is going there.

It might be your setup using a UEFI with a fb console up to the end of the 
loader,
then the kernel decides it is using a serial console.  Or vice versa.

>Rod is correct.  I came across this issue in 12.0 and probably is also in 11.3 
>by now.  See here:  https://wiki.freebsd.org/bhyve/UEFI for some tips that may 
>help you work out where your console is >redirecting.

>I use UEFI for all guest operating systems with a custom UEFI loader that has 
>been fixed to handle OpenBSD (as you know :-) ).

Thanks guys.
I have managed to get FreeBSD to boot using the UEFI loader with 
boot_serial=no. Shame that I can’t get it to boot with bhyveload as it seems a 
more streamlined boot process and allowed me to easily access the main console 
directly from ssh.

One thing I have now noticed is that it will boot when accessing the console 
via an nmdm device on com1. However, if I try and boot bhyve in the foreground 
using stdio (I usually just run it in a tmux window as I find it far more 
flexible than nmdm/cu), it stops, so clearly does seem to be a console issue. I 
can’t however find any loader setting that will fix it.

>
> I've also tested a windows server 2016 guest, and while that will actually 
> install via UEFI, it is noticeably slow - over a minute to boot to the login 
> screen and everything crawls along
>

>2016 is slow (even slower doing windows updates).  2019 is much better.  A tip 
>I have found is a minimum of 1 cpu, 2 cores and 4 threads to get decent speed 
>from that OS (more CPUs in bhyve >tends to make performance worse - in my 
>observations - in 2016).  Also, use the Virtio collection from RedHat for 
>vionet and viostor.  We are currently using 0.1.171 without issue.  The ahci 
>>emulation in itself is extremely slow.  NVMe and virtio is really the only 
>way to go.

Is there anything else I can check here? I haven’t got round to testing 
networking yet but I’m using nvme for the disk.
It’s basically unusable and there is no way I could put anything production on 
it. Just highlighting an icon on the desktop takes several seconds.

Matt


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve issues on Dell C6220 node

[Matt Churchyard via freebsd-virtualization]

Hello,

I've recently got hold of some Dell C6220 systems (2 servers in a single 
chassis) while I was hoping to use for bhyve
The basic spec of a single server is as follows -

Xeon(R) CPU E5-2670 (VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID)
64GB DDR3
LSI MegaRaid 9265-8i
FreeBSD 12.1

I had a bit of trouble getting the disks configured and booting (**probably 
unrelated but see below for the details).

However, my main problem at the moment is that booting either 11.3 or 12.1 in a 
bhyve guest (using bhyveload) gets as far as starting to boot from the install 
ISO, then just hangs

"
Loading kernel...
/boot/kernel/kernel text=0x168fdf1 data=0x1d0a68+0x768d80 
syms=[0x8+0x178bc0+0x8+0x1969d5]
Loading configured modules...
can't find '/boot/entropy'
/
"

I get either a "|" or "/" character, then nothing else.

I've also tested a windows server 2016 guest, and while that will actually 
install via UEFI, it is noticeably slow - over a minute to boot to the login 
screen and everything crawls along

I'm at a loss at the moment. My test machine for years has been an old i3-2100 
system and that has always booted freebsd & windows guests fairly well

I'd be grateful if anyone has any ideas or insight, or whether I just need to 
scrap the whole idea and either switch to a completely different hypervisor 
(which I'd really rather not do as I know FreeBSD very well and was planning on 
making use of ZFS for backups/migration/etc) or find some different systems


**Regarding the disks, I didn't really want a raid controller and have ended up 
creating a jbod (RAID0) volume for each disk, which appears as mfidX. This 
seems to work fine, although at first I tried to create a single pool of 2 
mirrored pairs (4 disks). This failed to boot due to zfs block i/o errors on 
boot. In the end I had to create a separate boot mirror across 2 partitions, 
the create a second pool for actual data storage.

Regards,
Matt Churchyard

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: VM Switch broken after update on 20190507

[Matt Churchyard via freebsd-virtualization]

>I just updated my bhyve server running CURRENT up to r349133 and can't start 
>my network switch.  I suspect that this problem has a root cause in the kernel 
>configuration change made on 20190507 that created if_tuntap to replace the 
>tunnel and tap devices.  My vm switch has been in use for over a year and 
>worked fine.  When booting I get the message:

> /boot/kernel/if_tap.ko - unsupported file type.

> I can't remove or destroy my vm switch to create a new one.  I get the
> message:

> vm switch remove public re0 - unable to locate switch id vm destroy public - 
> failed to remove virtual switch

> My vm_bhyve port is version 1.3.0

> I tried to locate any config files that were generated by the vm switch 
> create utility.  I was not successful.  I wanted to see if I could just 
> manually edit something to change if_tap to if_tuntap.  What do I need to 
> make my vm switch work again?

Hi Tom,

I haven't tested this myself but there's a recent PR that was provided to fix 
this that hasn't made it into ports yet.

https://github.com/churchers/vm-bhyve/pull/305/files

Matt

>Tom

--
Public Keys:
PGP KeyID = 0x5F22FDC1
GnuPG KeyID = 0x620836CF
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: [vm-bhyve] Windows 2012 and 2016 servers guests would not stop

[Matt Churchyard via freebsd-virtualization]

Victor Sudakov wrote on 2019-04-22 19:43:
...
>> And the implementation is pretty brutal:
>> # 'vm stopall'
>> # stop all bhyve instances
>> # note this will also stop instances not started by vm-bhyve # 
>> core::stopall(){
>>  local _pids=$(pgrep -f 'bhyve:')
>>
>>  echo "Shutting down all bhyve virtual machines"
>>  killall bhyve
>>  sleep 1
>>  killall bhyve
>>  wait_for_pids ${_pids}
>> }

> yow.

>>
>> I wonder what the effect of the second kill is, that seems odd.
> 
> Indeed.

> the first killall will cause each client OS to see a soft shutdown signal. 
> the sleep 1 gives them some time to flush their buffers. the second killall 
> says, time's up, just stop.

Both of these killall commands send a soft shutdown signal and I've never seen 
an instance in my own testing where the second has caused an instant poweroff.
I've just double checked this, and a FreeBSD guest stopped with the existing 
code fully shuts itself down, ending with "acpi0: Powering system off"

The main reason for having both is that in my initial testing, I could not get 
Windows to respond to a single SIGINT. 100% of the time it would simply act 
like nothing had happened. Sending two however triggered a shutdown. I only 
have a single test machine though so maybe it's something strange about my 
particular system that no one else experiences...

Matt

> i think this is worse than brutal, it's wrong. consider freebsd's own work 
> flow when trying to comply with the first soft shutdown it got:

> https://github.com/freebsd/freebsd/blob/master/sbin/reboot/reboot.c#L220

> this has bitten me more than once, because using "pageins" as a proxy for "my 
> server processes are busy trying to synchronize their user mode state" is 
> inaccurate. i think _any_ continuing I/O should > be reason to wait the full 
> 60 seconds.

> and so i think the "sleep 1" above should be a "sleep 65".

> What is needed in vm-bhyve is the feature that if ACPI does not stop 
> the guest for a predefined period of time, the guest is powered off.

> i agree with this.

--
> P Vixie

___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor

[Matt Churchyard via freebsd-virtualization]

> > 
> > > 1. Does ARC actually cache zfs volumes (not files/datasets)?
> > 
> > Yes it does.
> 
> I find this distinction between volumes/files/etc and what is cached 
> causes confusion (as well as "volumes not datasets").
> 
> Both ZVOLs and Z file systems are types of dataset. A dataset stores 
> data in records (usually up to 128kb in size).  It's these records 
> that are cached (and that most ZFS functions such as 
> compression/raidz/zil/etc work with)
> 
> As far as the ZFS lower levels are concerned, there is no difference 
> between a volume and a file system.

>Thank you Matt, this was very instructive.

> > > 2. If ARC does cache volumes, does this cache make sense on a 
> > > hypervisor, because guest OSes will probably have their own disk cache 
> > > anyway.
> > 
> > IMHO not much, because the guest OS is relying on the fact that when 
> > it writes it’s own cached data out to „disk“, it will be committed 
> > to stable storage.
> 
> Maybe I've missed something but I don't quite get the link between 
> read cache (ARC) and guest writes here?

>Maybe there was a confusion between read and write caches, but my question 
>still stands:

>Does it make sense to cache the same data (for reading too) twice: one time in 
>host's RAM (ZFS ARC) and the other time in guest's RAM (whatever fs the guest 
>uses, all modern OSes have disk caches)?

>What do VMWare or VirtualBox do for this situation? Do they ever cache their 
>volumes in the hypervisor's RAM?

Virtualbox would be no different to bhyve in that it doesn't care what storage 
system you are using or how it is configured, that's up to the system admin.
I believe VMFS is more akin to other "traditional" file systems, and doesn't do 
RAM caching to anywhere near the extent of ZFS. I do think you can use 
SSD/NVMe/etc as cache in VMWare.

My initial instinct would be to keep cache on, but reduce the limit to allocate 
the majority of the RAM for guests. (I'd still want at least 4GB as an absolute 
minimum though, probably more on systems with 100GB+ total). Of course you 
could probably test with cache set to all/metadata and see what effect it has. 
Adding L2ARC may be useful if the main pool is spinning disks, but then I've 
heard there's a rule of thumb for requiring X amount of ARC for Y amount of 
L2ARC, but I'm not sure what that rule is.

I'd also be intrigued to know what the logic in FreeNAS is for it. It is simply 
a case of "(arc = total_ram - guest_allocated)"?
Is there a lower limit based on a percentage or total RAM, and/or a hard lower 
limit?

Matt

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: bhyve and vfs.zfs.arc_max, and zfs tuning for a hypervisor

[Matt Churchyard via freebsd-virtualization]

Patrick M. Hausen wrote:
> 
> > 1. Does ARC actually cache zfs volumes (not files/datasets)?
> 
> Yes it does.

I find this distinction between volumes/files/etc and what is cached causes 
confusion (as well as "volumes not datasets").

Both ZVOLs and Z file systems are types of dataset. A dataset stores data in 
records (usually up to 128kb in size).
It's these records that are cached (and that most ZFS functions such as 
compression/raidz/zil/etc work with)

As far as the ZFS lower levels are concerned, there is no difference between a 
volume and a file system.

> 
> > 2. If ARC does cache volumes, does this cache make sense on a 
> > hypervisor, because guest OSes will probably have their own disk cache 
> > anyway.
> 
> IMHO not much, because the guest OS is relying on the fact that when 
> it writes it’s own cached data out to „disk“, it will be committed to 
> stable storage.

Maybe I've missed something but I don't quite get the link between read cache 
(ARC) and guest writes here?

> This is an important point.

> > 3. Would it make sense to limit vfs.zfs.arc_max to 1/8 or even less 
> > of total RAM, so that most RAM is available to guest machines?
> 
> Yes if you build your own solution on plain FreeBSD. No if you are 
> running FreeNAS which already tries to autotune the ARC size according 
> to the memory committed to VMs.
> 
> > 4. What other zfs tuning measures can you suggest for a bhyve 
> > hypervisor?
> 
> e.g.
>   zfs set sync=always zfs/vm
> 
> if zfs/vm is the dataset under which you create the ZVOLs for your 
> emulated disks.

>Well, bhyve already has an option for this:

>The block-device-options are:

>nocache   Open the file with O_DIRECT.
>directOpen the file using O_SYNC.
>roForce the file to be opened read-only.

>I think something like
>"-s 4:0,virtio-blk,/dev/zvol/zroot/vm/mail/disk0,direct"
>would do the same?

> 
> I’m using this for all my VM „disks“ and have added a 16 GB SLOG 
> device to my spinning disk pool - seems to work great. This is on a home 
> system.

> Is SLOG also used by zfs volumes?

As above, the core of ZFS doesn't care what type of dataset it is working with. 
ARC/ZIL/etc all work exactly the same.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: NVMe and Bhyve

[Matt Churchyard via freebsd-virtualization]

Michael Reifenberger wrote:
> Hi,
> first I tried to install windows10 ltsc 2019 onto a nvme disk.
> This failed, the windows installer did not find a disk to install.
> 
> Then I tried the following setup:
> ...
> disk0_type="ahci-hd"
> disk0_dev="zvol"
> disk0_name="disk0"
> disk1_type="nvme"
> disk1_dev="zvol"
> disk1_name="disk1"
> disk1_type="ahci-cd"
> disk1_name="disk2.img"

>If this is exact copy/paste, shouldn't the 2 lines above use disk2 prefix 
>instead?

There's definitely an issue with the configuration file, but vm-bhyve retrieves 
config settings by looking through the file (which is loaded into a variable) 
line by line and returning the first match. As such, it will likely be ignoring 
the ahci-cd & disk2.img settings. Strange why the last disk is a CD and 
pointing at a .img file though.

More useful would be the actual bhyve command from /path/to/guest/vm-bhyve.log, 
and maybe also confirmation that the zvol path that appears in that command 
line correctly points to an existing zvol.

Matt

> ...
> 
> And Installed windows on disk0.
> Disk0 is found as expected, disk1 is not.
> 
> The devicemanager reports an error:
> https://imgur.com/a/zrHx23y
> 
> Shouldn't nvme be supported in behyve?
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: UEFI GOP: screen goes blank during boot after loader is finished

[Matt Churchyard]

-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of Warner Losh
Sent: 15 November 2018 02:57
To: Kyle Evans 
Cc: FreeBSD Current ; Rodney W. Grimes 
; freebsd-virtualization@freebsd.org
Subject: Re: UEFI GOP: screen goes blank during boot after loader is finished

On Wed, Nov 14, 2018 at 4:00 PM Kyle Evans  wrote:

> On Wed, Nov 14, 2018 at 4:55 PM Subbsd  wrote:
> >
> > On Thu, Nov 15, 2018 at 1:33 AM Kyle Evans  wrote:
> > >
> > > On Wed, Nov 14, 2018 at 4:30 PM Kyle Evans  wrote:
> > > >
> > > > On Wed, Nov 14, 2018 at 4:23 PM Subbsd  wrote:
> > > > >
> > > > > On Thu, Nov 15, 2018 at 1:05 AM Subbsd  wrote:
> > > > > >
> > > > > > On Thu, Nov 15, 2018 at 12:21 AM Rebecca Cran <
> rebe...@bluestop.org> wrote:
> > > > > > >
> > > > > > > On November 14, 2018 at 2:18:04 PM, Subbsd 
> > > > > > > (sub...@gmail.com)
> wrote:
> > > > > > >>
> > > > > > >>
> > > > > > >> My current host: FreeBSD 13.0-CURRENT r340319 and the 
> > > > > > >> problem
> is
> > > > > > >> still present.
> > > > > > >
> > > > > > > Rod was asking about the guest OS version, not the host though.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > I apologize, it seemed to me that I wrote earlier) Guest version:
> > > > > >
> https://download.freebsd.org/ftp/snapshots/amd64/amd64/ISO-IMAGES/13.0
> /FreeBSD-13.0-CURRENT-amd64-20181101-r339979-disc1.iso.xz
> > > > >
> > > > > Hm, it seems the problem is 'boot_serial' which is sets to YES 
> > > > > by
> default in gop
> > > > >
> > > > > set boot_serial=NO
> > > > > boot
> > > > >
> > > > > solve this issue
> > > >
> > > > Huh? This is perhaps going to be a stupid question, but where is 
> > > > boot_serial=YES getting set? Loader will not set it by itself 
> > > > and
> UEFI
> > > > doesn't respect /boot.config, so this must be explicitly set in 
> > > > /boot/loader.conf or /boot/defaults/loader.conf, but it's not 
> > > > clear
> to
> > > > me what's putting it there.
> > >
> > >
> http://src.illumos.org/source/xref/freebsd-head/usr.sbin/bhyveload/bhy
> veload.c#832
> > > is the only place I can see immediately that this might be 
> > > happening, but do UEFI boots go through bhyveload? I'm ignorant here.
> >
> > This is UEFI GOP methodvia bootrom/uefi-firmware, no bhyveload:
> >
> > bhyve -AHP -s 0:0,hostbridge -s 31:0,lpc -s 
> > 4:0,ahci-cd,/tmp/FreeBSD-13.0-CURRENT-amd64-20181101-r339979-disc1.i
> > so -c 1 -m 1024M -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait -l 
> > bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd freebsd1
> >
> > https://snag.gy/0MH7zU.jpg
> > https://snag.gy/kF5cxZ.jpg
> > https://snag.gy/htHMG0.jpg
> > https://snag.gy/vK1ALN.jpg
> > https://snag.gy/qKNPGU.jpg
>
> What does your /boot/loader.conf look like?
>

>What is the ConOut evifar look like? We set serial when the UEFI env says to 
>do  so.

>Warner

I can confirm that 11.2-REL boots fine but 12.0-BETA4 goes blank just after the 
loader, using the exact same bhyve configuration. (This is on an 11.2 host)
A few lines are output just before going blank but I can't catch what they say.

I'm using the official ISO files with no other configuration so it does seem 
that 12.0 will currently not boot under UEFI-GOP bhyve.

I can check the efivars if someone lets me know how.

Matt
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: bhyve VM using BHYVE_UEFI.fd gets EPT violation

[Matt Churchyard]

>Hi,

>I'm taking my first steps in using bhyve, and the first showstopper seems to 
>be that using '-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd' gets 
>me immediate vm exit with >exit_reason being 48 (EPT violation).

>Relevant part of dmesg:

FreeBSD 12.0-ALPHA9 r339291 GENERIC amd64 FreeBSD clang version 6.0.1 
(tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
WARNING: WITNESS option enabled, expect reduced performance.
VT(efifb): resolution 800x600
CPU: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz (2200.04-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4

Features=0xbfebfbff

Features2=0x7fbee3ff
  AMD Features=0x2c100800
  AMD Features2=0x1
  Structured Extended Features=0x281
  XSAVE Features=0x1
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics real memory  = 274877906944 
(262144 MB) avail memory = 267769249792 (255364 MB)

Simplified invocation line (I tried with a lot more options initially, but this 
seems to get the same):

# bhyve -c 2 -m 2G -l
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd test vm exit[0]
reason  VMX
rip 0xfff0
inst_length 3
status  0
exit_reason 48 (EPT violation)
qualification   0x0184
inst_type   0
inst_error  0
Abort trap

# bhyve -c 2 -m 2G -l
bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd test vm exit[0]
reason  VMX
rip 0xfff0
inst_length 3
status  0
exit_reason 48 (EPT violation)
qualification   0x0184
inst_type   0
inst_error  0
Abort trap

> Any hints?

The commands you're running aren't really very useful as you haven't provided a 
console device, or any disk/network/etc.
However the main issue appears to be a lack of an lpc device. I get the exact 
same error on a fully working system if I remove that device.

Note there is /usr/share/examples/bhyve/vmrun.sh, and tools in ports to handle 
running bhyve commands for you. (Unless you are intentionally trying to learn 
the raw bhyve command syntax of course)

Matt

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Problem installing Ubuntu with vm_byhve

[Matt Churchyard]

On 08/10/2018 20:28, Xavier Humbert wrote:
> On 08/10/2018 11:28, Matt Churchyard wrote:
>> # vm create -t ubuntu ubuntu-guest
> 
> Oops, forgot to select the template !
> 
> Works much better :-)
> 
> Thanks
> 

>Well it now boots and install, but when I reboot, I drop into grub.
>Managed it with this mantra :
>grub> set root=(hd0,gpt2)
>grub> set prefix=(hd0,gpt2)/boot/grub
>grub>insmod normal
>grub>normal

>How can I fix autoboot ?

Looks like it wants to boot of the second partition but vm-bhyve looks on 
partition 1 by default.

Try adding the following config -

grub_run_partition="2" 
- or -
grub_run_partition="gpt2" (not sure if just "2" will work or whether it needs 
the full "gpt2")

>The actual ubuntu.conf
>loader="grub"
>cpu=1
>memory=512M
>network0_type="virtio-net"
>network0_switch="public"
>disk0_type="virtio-blk"
>disk0_name="disk0.img"
>uuid="a68bdef0-cbc6-11e8-a293-d05099c11279"
>network0_mac="58:9c:fc:04:25:7a"

>Thanks

--
Xavier HUMBERT  
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Error installing Xubuntu under bhyve

[Matt Churchyard]

On 10/8/18 6:14 PM, D'Arcy Cain wrote:
> I am trying to install Xubuntu under bhyve with vm-bhyve.  The install

>I had a private email with some suggestions.  They didn't work but I thought I 
>should mention them here for discussion purposes.

> goes fine right up to near the end when it fails with a "Executing 
> 'grub-install /dev/vda' failed" error and the install fails.  A second 
> attempt says that there is a system already on the disk but it won't 
> boot up and leaves the console at a grub prompt.  I have grub2-bhyve 
> installed.  Here is my config:
> 
> guest="linux"
> uefi="yes"
> loader="grub"
> grub_run_partition="msdos1"

>He said that I don't need the grub lines if I am using uefi.  Those lines were 
>actually added in as a Hail Mary.  In or out made no difference.

Yes, all grub settings are irrelevant if you are using UEFI. With UEFI, the 
UEFI firmware is used instead of grub2-bhyve (even if the guest boots from UEFI 
into grub internally).
As such, the loader="grub" option isn't actually needed here either.
Also the "guest" configuration option was removed quite a while ago

I was able to install XUbuntu with a simple UEFI + graphics config and using 
the graphical installer via VNC -

uefi="yes"
graphics="yes"
xhci_mouse="yes"
cpu=1
memory=512M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"

> disk0_type="virtio-blk"

>He suggested ahci-hd.  It made no difference.

>He also suggested starting over with a new disk but that didn't help either.

>Still looking for suggestions.

-- 
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vex.net, VoIP: sip:da...@druid.net 
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: [vm-bhyve] Does anyone have a vm template for Linux Mint ?

[Matt Churchyard]

Josias L. Gonçalves wrote:
> > > I get the same blank rectangle with Ubuntu (vm install mint
> > > ubuntu-16.04-desktop-amd64.iso)
> > > So there must be something amiss in my setup.

> 
> 
> Verify if everything is installed:
> pkg install bhyve-firmware-1.0_1 bhyve-rc-3 grub2-bhyve-0.40_5
> libhyve-remote-0.1.4.2 uefi-edk2-bhyve-0.1,1 uefi-edk2-bhyve-csm-0.1,1

Well, not everything from your list. I have currently

>root@vas:~ # pkg info -a | egrep 'bhyve|uefi'
>grub2-bhyve-0.40_4 Grub-emu loader for bhyve
>uefi-edk2-bhyve-0.1,1  UEFI-EDK2 firmware for bhyve
>uefi-edk2-bhyve-csm-0.1,1  UEFI-EDK2 firmware for bhyve with CSM
>vm-bhyve-1.2.3 Management system for bhyve virtual machines
>root@vas:~ #

>bhyve-firmware: is a metaport for uefi-edk2-bhyve and uefi-edk2-bhyve-csm, I 
>already have them both installed. But no harm installing it also.

>bhyve-rc-3: "FreeBSD RC script for starting bhyve guests in tmux" - I 
>definitely don't need it because I use vm-bhyve for VM management.

>libhyve-remote-0.1.4.2: well, maybe this one is the culprit. We shall see 
>tonight if its presence makes any difference.

As you mention all the required firmware is installed by the bhyve-firmware 
port. You do not need to install the edk2 ports.
Grub2 is not required either if you're using UEFI, although you'll need it to 
run any guests with loader="grub" and there's no harm having it around if 
you're using bhyve.
I've never heard of libhyve-remote until now. This is an additional component 
(something to do with FreeNAS by the look of it) and has nothing to do with the 
actual running of bhyve or vm-bhyve.

Personally I've never had success with the CSM firmware (other that with 
SmartOS which was following instructions from the original bhyve devs). My 
understanding is that CSM is supposed to emulate a traditional BIOS, so 
anything that would boot on a BIOS should work (or at least try to boot). 
However I usually just get a blank screen and nothing else.

I have successfully** installed Ubuntu 16.04 (via VNC) using the following 
template, which is just a basic UEFI config with virtio network/disk.

uefi="yes"
graphics="yes"
xhci_mouse="yes"
cpu=2
memory=2G
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"

**I did have two fairly annoying issues getting 16.04 to work -

1) After installation it seemed to refuse to reboot without me removing the 
install CD, which I obviously couldn't do. I ended up powering off the system 
manually with "vm poweroff guest"

2) It then would not boot due to the fact that Ubuntu 16.04 puts the bootloader 
in a non-default location, then uses efivars to tell UEFI how to find it. 
Unfortunately the bhyve EFI firmware still does not save efivars, so any 
specific boot options set by a guest are lost on restart. I had to escape to 
the EFI shell and manually choose the EFI/ubuntu/grubx64.efi loader. See my 
post #11 on this forum post for more details - 
https://forums.freebsd.org/threads/how-to-install-a-ubuntu-guest-in-bhyve.66767/

Matt

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Problem installing Ubuntu with vm_byhve

[Matt Churchyard]

>Hi all,

>First, I must admit I'm not a byve guru, but I got a pfSense vm installed and 
>running, so the problem is not my host.

>I started with an old release to avoid URFI problems

>What I've done :
># vm create ubuntu
># vm install ubuntu ubuntu-12.04.5-server-amd64.iso Starting ubuntu
>  * found guest in /vms/ubuntu
>  * booting...
># vm console ubuntu
>Consoles: userboot

> ...

>my config file is straight from .templates :
># cat ubuntu/ubuntu.conf
>loader="bhyveload"
>cpu=1
>memory=256M
>network0_type="virtio-net"
>network0_switch="public"
>disk0_type="virtio-blk"
>disk0_name="disk0.img"

The main issue is that you're trying to use the freebsd loader (bhyveload) for 
Linux. You need to use grub or UEFI for Linux.

You could start by trying the Ubuntu template, although you may find you need 
to provide grub commands in order to get the guest to boot.
See https://github.com/churchers/vm-bhyve/wiki/Configuring-Grub-Guests for more 
info on booting with grub

# vm create -t ubuntu ubuntu-guest

If the guest won't boot without manually specifying Grub commands, you may 
actually find it easier to use a newer version of Linux and boot with UEFI.

Matt

>Thanks for any help

>Regards,

--
Xavier HUMBERT
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: New bhyve user

[Matt Churchyard]

-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of D'Arcy Cain
Sent: 28 September 2018 14:24
To: freebsd-virtualization@freebsd.org
Subject: New bhyve user

>Greetings.  I have just recently started using bhyve (previously a Xen user).  
>I am using vm-bhyve to manage it.  I have a few questions.

>First question, am I making the right choice by switching and, if so, is bhyve 
>the right choice to switch to?  I realize that that is an impossible question 
>but perhaps some pros and cons as well as war stories will help me.

It really depends on how well bhyve supports what you need to do. The main 
downside of bhyve at the moment is that it's very new and lacks many features 
and performance optimisations you may find in more established hypervisors. I 
can't really say yes you should use bhyve or no you should stick with Xen.

>I created a switch and clients using the examples on the vm-bhyve web site.  
>However, I could not get IP working until I put an IP address on the vm-public 
>interface.  I duplicated the address of the interface that it is >connected to 
>(re0 in my case) and used DHCP to assign addresses to the clients.  If this is 
>the correct way, shouldn't it have happened automatically?

No, you should not be duplicating IP addresses.

A virtual "switch" is really just an ethernet bridge. If you want guests to be 
on the same lan as the host, then you can just create a virtual switch and add 
your physical interface to it

# vm switch create public
# vm switch add public re0

Any guest that is connected to the vm-public switch will be bridged to re0, and 
as such to the network re0 is connected to. In that instance you would give 
guests IP addresses on the same range as re0 (or they could get addresses from 
your local DHCP server).

If you want to guests to have a separate network, then you can assign an 
address to the virtual switch (using a different address range to the host)

# vm switch create guests
# vm switch address guests 192.168.100.1/24

In this case you would assign guests address in that network, with 
192.168.100.1 as the gateway. (Alternatively you could install something to 
provide dhcp. There are guides on the vm-bhyve GitHub for using dnsmasq).
For this to work you would either need to configure the host to perform NAT, or 
configure the rest of your network to know that any traffic to 192.168.100.0/24 
should be routed to the bhyve host. (NAT is probably the easier option)

>In any case, I saw I see that it can be added at creation time but how do I 
>modify it later?  I saw "switch address a.b.c.d/xx|none" in the man page but 
>no way to specify which switch the address should be applied to.  I tried 
>>adding the switch name before and after the address but that gave me an error.

I've just tested the above address command and it seems to be working for me...

>I tried to boot into a Linux install but even though I set grahics to yes, it 
>doesn't seem to be serving VNC.  On the console I can only get into the live 
>CD.  How do I get it installed?

Did you have uefi="yes" in the configuration? Graphics are only available when 
using UEFI boot.

>I am thinking of creating a base install with various install options and then 
>copy that over to new installs as a starting point.  I was going to use rsync 
>with the -S option to copy over the file as sparse.
>Is there another way that is preferred?

For this sort of setup, ZFS is the obvious answer as you can use send/recv to 
duplicate guests (or even clone to create an instant copy without using 
additional disk space). If not using ZFS then rsync would be a reasonable 
option to create copies of guests.

>In Xen there is a maxvcpus which limit the number of CPUs but they could 
>baloon down if not busy so that other clients who are busy can use the CPUs.  
>In bhyve (at least in vm-bhyve) there is only a cpus line in the config.  >Is 
>this a minimum, maximum or is it a hard limit?

This is the number of virtual cpus that the guest will see. Remember that as 
far as the host is concerned, the guests are processes that are using 
resources, just like any other program. A guest that is not doing much will not 
being using much cpu time on the host, and the host will happily run other 
guests (or system processes) on the same physical cpus.

>That's it for now.  Thanks for any help.

>-- 
>D'Arcy J.M. Cain  |  Democracy is three wolves
>http://www.druid.net/darcy/|  and a sheep voting on
>+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
>IM: da...@vex.net, VoIP: sip:da...@druid.net 
>___
>freebsd-virtualization@freebsd.org mailing list 
>https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>To unsubscribe, send any mail to 
>"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list

RE: Checking bhyve supported features (sysctls)

[Matt Churchyard]

-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of Marcelo Araujo
Sent: 17 August 2018 10:14
To: Matt Churchyard 
Cc: freebsd-virtualization@freebsd.org
Subject: Re: Checking bhyve supported features (sysctls)

2018-08-17 16:54 GMT+08:00 Matt Churchyard :

>
>
>
>
> 2018-08-17 16:25 GMT+08:00 Matt Churchyard :
>
> -Original Message-
> From: owner-freebsd-virtualizat...@freebsd.org < 
> owner-freebsd-virtualizat...@freebsd.org> On Behalf Of Rodney W. 
> Grimes
> Sent: 16 August 2018 18:31
> To: Allan Jude 
> Cc: Matt Churchyard ; 
> freebsd-virtualization@ freebsd.org
> Subject: Re: Checking bhyve supported features (sysctls)
>
> > On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" <
> freebsd-...@pdx.rh.cn85.dnsmgr.net> wrote:
> > >>
> > >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> > >>
> > >> > > Hello,
> > >> > >
> > >> > > I'm looking for better ways to check for  bhyve support /
> > >available
> > >> > > features without trying to scan through dmesg output.
> > >> >
> > >> > >Yes, it would be very good to remove that, as it usually tries 
> > >> > >to grep a non-existent file /var/run/dmesg.boot that is not 
> > >> > >created until after vm_bhyve has been called from
> > >/usr/local/etc/rc.d
> > >> > >when you have things set to autostartup >in /etc/rc.conf
> > >> >
> > >> >
> > >> > >
> > >> > > I notice that the following 2 sysctl's appear to be set to 1 
> > >> > > as
> > >soon
> > >> > > as the vmm module is loaded
> > >> > >
> > >> > > hw.vmm.vmx.initialized: 1
> > >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> > >> > >
> > >> > > Will these be available on both Intel & AMD processors as a 
> > >> > > way to determine if the module has loaded successfully and 
> > >> > > can run
> > >guests?
> > >> > >
> > >> > > I also see the below sysctl related to iommu.
> > >> > >
> > >> > > hw.vmm.iommu.initialized
> > >> > >
> > >> > > Again, will this be set to 1 as soon as the module is loaded 
> > >> > > if iommu is supported, or only when it is used?
> > >> > > There also seems to be a vmm.amdvi.enable sysctl.
> > >> > > Would both these need checking or is vmm.iommu enough to 
> > >> > > determine support on any processor.
> > >> >
> > >> > >Probalby the safest way for a shell script to decide if bhyve 
> > >> > >is up and running is to stat /dev/vmm, if that exists then the
> > >modules
> > >> > >have loaded and initialized and bhyve should be ready to 
> > >> > >process
> > >guests.
> > >> >
> > >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> > >>
> > >> I'll investigate that, I was pretty sure that you should get this 
> > >> as soon as the vmm.ko module is finished initialzing, but you 
> > >> might be right in that it takes a first vm to cause its creation.
> > >> Confirmed, /dev/vmm does not exist until the first vm is created.
> > >>
> > >> >
> > >> > >sysctl's mentiond above would be a poor way to make this
> > >determination.
> > >> >
> > >> > It would be nice if sysctls were better documented.
> > >>
> > >> Agreed.
> > >>
> > >> > If vmx.initialized is set once vmm has successfully loaded, I 
> > >> > can't
> > >see a better way of checking for bhyve support (assuming it's not 
> > >Intel specific). This entry definitely exists and is set to 0 if 
> > >you load the module on a non-supported system, and set to 1 as soon 
> > >as vmm loads on my Intel test system.
> > >>
> > >> Given its undocumented status you would be relying on an 
> > >> undocumented feature that could change in either name or 
> > >> behavior, and that is not desirable.
> > >>
> > >> Let me see if I can come up with something else.
> > >
> > >I looked at the code for bhyvectl, bhyveload and byhve.  They do 
> > >not actually try t

RE: Checking bhyve supported features (sysctls)

[Matt Churchyard]

2018-08-17 16:25 GMT+08:00 Matt Churchyard 
mailto:matt.churchy...@userve.net>>:
-Original Message-
From: 
owner-freebsd-virtualizat...@freebsd.org<mailto:owner-freebsd-virtualizat...@freebsd.org>
 
mailto:owner-freebsd-virtualizat...@freebsd.org>>
 On Behalf Of Rodney W. Grimes
Sent: 16 August 2018 18:31
To: Allan Jude mailto:allanj...@freebsd.org>>
Cc: Matt Churchyard 
mailto:matt.churchy...@userve.net>>; 
freebsd-virtualization@freebsd.org<mailto:freebsd-virtualization@freebsd.org>
Subject: Re: Checking bhyve supported features (sysctls)

> On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" 
> mailto:freebsd-...@pdx.rh.cn85.dnsmgr.net>>
>  wrote:
> >>
> >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> >>
> >> > > Hello,
> >> > >
> >> > > I'm looking for better ways to check for  bhyve support /
> >available
> >> > > features without trying to scan through dmesg output.
> >> >
> >> > >Yes, it would be very good to remove that, as it usually tries
> >> > >to grep a non-existent file /var/run/dmesg.boot that is not
> >> > >created until after vm_bhyve has been called from
> >/usr/local/etc/rc.d
> >> > >when you have things set to autostartup >in /etc/rc.conf
> >> >
> >> >
> >> > >
> >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> >soon
> >> > > as the vmm module is loaded
> >> > >
> >> > > hw.vmm.vmx.initialized: 1
> >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> >> > >
> >> > > Will these be available on both Intel & AMD processors as a way
> >> > > to determine if the module has loaded successfully and can run
> >guests?
> >> > >
> >> > > I also see the below sysctl related to iommu.
> >> > >
> >> > > hw.vmm.iommu.initialized
> >> > >
> >> > > Again, will this be set to 1 as soon as the module is loaded if
> >> > > iommu is supported, or only when it is used?
> >> > > There also seems to be a vmm.amdvi.enable sysctl.
> >> > > Would both these need checking or is vmm.iommu enough to
> >> > > determine support on any processor.
> >> >
> >> > >Probalby the safest way for a shell script to decide if bhyve is
> >> > >up and running is to stat /dev/vmm, if that exists then the
> >modules
> >> > >have loaded and initialized and bhyve should be ready to process
> >guests.
> >> >
> >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> >>
> >> I'll investigate that, I was pretty sure that you should get this
> >> as soon as the vmm.ko module is finished initialzing, but you might
> >> be right in that it takes a first vm to cause its creation.
> >> Confirmed, /dev/vmm does not exist until the first vm is created.
> >>
> >> >
> >> > >sysctl's mentiond above would be a poor way to make this
> >determination.
> >> >
> >> > It would be nice if sysctls were better documented.
> >>
> >> Agreed.
> >>
> >> > If vmx.initialized is set once vmm has successfully loaded, I
> >> > can't
> >see a better way of checking for bhyve support (assuming it's not
> >Intel specific). This entry definitely exists and is set to 0 if you
> >load the module on a non-supported system, and set to 1 as soon as
> >vmm loads on my Intel test system.
> >>
> >> Given its undocumented status you would be relying on an
> >> undocumented feature that could change in either name or behavior,
> >> and that is not desirable.
> >>
> >> Let me see if I can come up with something else.
> >
> >I looked at the code for bhyvectl, bhyveload and byhve.  They do not
> >actually try to decide if vmm is supported or not, they simply
> >process the error from a vm_create() or vm_open() call and exit with
> >an error code if they can not handle it (some of the code can handle
> >a vm_create failure if infact we are trying to create a vm that
> >already exists).
> >
> >If you want to maintain full compatibility a similiar stratergy may
> >be in order.
> >
> >Why is it that vm-bhyve specifically needs to know if the kernel has
> >vmm support or not?
> >Cant it just be written to handle the errors returned if the
> >suppo

RE: Checking bhyve supported features (sysctls)

[Matt Churchyard]

-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of Rodney W. Grimes
Sent: 16 August 2018 18:31
To: Allan Jude 
Cc: Matt Churchyard ; 
freebsd-virtualization@freebsd.org
Subject: Re: Checking bhyve supported features (sysctls)

> On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" 
>  wrote:
> >> 
> >> Text manually wrapped to 80, any broken quoting is my fault - rwg
> >> 
> >> > > Hello,
> >> > > 
> >> > > I'm looking for better ways to check for  bhyve support /
> >available
> >> > > features without trying to scan through dmesg output.
> >> > 
> >> > >Yes, it would be very good to remove that, as it usually tries 
> >> > >to grep a non-existent file /var/run/dmesg.boot that is not 
> >> > >created until after vm_bhyve has been called from
> >/usr/local/etc/rc.d
> >> > >when you have things set to autostartup >in /etc/rc.conf
> >> > 
> >> > 
> >> > > 
> >> > > I notice that the following 2 sysctl's appear to be set to 1 as
> >soon
> >> > > as the vmm module is loaded
> >> > > 
> >> > > hw.vmm.vmx.initialized: 1
> >> > > hw.vmm.vmx.cap.unrestricted_guest: 1
> >> > > 
> >> > > Will these be available on both Intel & AMD processors as a way 
> >> > > to determine if the module has loaded successfully and can run
> >guests?
> >> > > 
> >> > > I also see the below sysctl related to iommu.
> >> > > 
> >> > > hw.vmm.iommu.initialized
> >> > > 
> >> > > Again, will this be set to 1 as soon as the module is loaded if 
> >> > > iommu is supported, or only when it is used?
> >> > > There also seems to be a vmm.amdvi.enable sysctl.
> >> > > Would both these need checking or is vmm.iommu enough to 
> >> > > determine support on any processor.
> >> > 
> >> > >Probalby the safest way for a shell script to decide if bhyve is 
> >> > >up and running is to stat /dev/vmm, if that exists then the
> >modules
> >> > >have loaded and initialized and bhyve should be ready to process
> >guests.
> >> > 
> >> > Hmm, I don't get /dev/vmm unless I actually have running guests.
> >> 
> >> I'll investigate that, I was pretty sure that you should get this 
> >> as soon as the vmm.ko module is finished initialzing, but you might 
> >> be right in that it takes a first vm to cause its creation.
> >> Confirmed, /dev/vmm does not exist until the first vm is created.
> >> 
> >> > 
> >> > >sysctl's mentiond above would be a poor way to make this
> >determination.
> >> > 
> >> > It would be nice if sysctls were better documented.
> >> 
> >> Agreed.
> >> 
> >> > If vmx.initialized is set once vmm has successfully loaded, I 
> >> > can't
> >see a better way of checking for bhyve support (assuming it's not 
> >Intel specific). This entry definitely exists and is set to 0 if you 
> >load the module on a non-supported system, and set to 1 as soon as 
> >vmm loads on my Intel test system.
> >> 
> >> Given its undocumented status you would be relying on an 
> >> undocumented feature that could change in either name or behavior, 
> >> and that is not desirable.
> >> 
> >> Let me see if I can come up with something else.
> >
> >I looked at the code for bhyvectl, bhyveload and byhve.  They do not 
> >actually try to decide if vmm is supported or not, they simply 
> >process the error from a vm_create() or vm_open() call and exit with 
> >an error code if they can not handle it (some of the code can handle 
> >a vm_create failure if infact we are trying to create a vm that 
> >already exists).
> >
> >If you want to maintain full compatibility a similiar stratergy may 
> >be in order.
> >
> >Why is it that vm-bhyve specifically needs to know if the kernel has 
> >vmm support or not?
> >Cant it just be written to handle the errors returned if the 
> >supported functions do not exist?
> 
> I think the question vm-bhyve wants to answer is: does the CPU have 
> the required features to run a multicore VM.

>Why does it need to know that?  If it tries to start a multicore/thread VM and 
>the system can not support it an error is returned and the bhyve command fails.

>Ac

Re: Checking bhyve supported features (sysctls)

[Matt Churchyard]

On 16 Aug 2018, at 19:55, Marcelo Araujo 
mailto:araujobsdp...@gmail.com>> wrote:



2018-08-17 0:53 GMT+08:00 Allan Jude 
mailto:allanj...@freebsd.org>>:
On August 16, 2018 5:28:05 PM GMT+01:00, "Rodney W. Grimes" 
mailto:freebsd-...@pdx.rh.cn85.dnsmgr.net>> 
wrote:
>>
>> Text manually wrapped to 80, any broken quoting is my fault - rwg
>>
>> > > Hello,
>> > >
>> > > I'm looking for better ways to check for  bhyve support /
>available
>> > > features without trying to scan through dmesg output.
>> >
>> > >Yes, it would be very good to remove that, as it usually tries
>> > >to grep a non-existent file /var/run/dmesg.boot that is not
>> > >created until after vm_bhyve has been called from
>/usr/local/etc/rc.d
>> > >when you have things set to autostartup >in /etc/rc.conf
>> >
>> >
>> > >
>> > > I notice that the following 2 sysctl's appear to be set to 1 as
>soon
>> > > as the vmm module is loaded
>> > >
>> > > hw.vmm.vmx.initialized: 1
>> > > hw.vmm.vmx.cap.unrestricted_guest: 1
>> > >
>> > > Will these be available on both Intel & AMD processors as a way
>> > > to determine if the module has loaded successfully and can run
>guests?
>> > >
>> > > I also see the below sysctl related to iommu.
>> > >
>> > > hw.vmm.iommu.initialized
>> > >
>> > > Again, will this be set to 1 as soon as the module is loaded if
>> > > iommu is supported, or only when it is used?
>> > > There also seems to be a vmm.amdvi.enable sysctl.
>> > > Would both these need checking or is vmm.iommu enough to
>> > > determine support on any processor.
>> >
>> > >Probalby the safest way for a shell script to decide if bhyve is
>> > >up and running is to stat /dev/vmm, if that exists then the
>modules
>> > >have loaded and initialized and bhyve should be ready to process
>guests.
>> >
>> > Hmm, I don't get /dev/vmm unless I actually have running guests.
>>
>> I'll investigate that, I was pretty sure that you should get this
>> as soon as the vmm.ko module is finished initialzing, but you might
>> be right in that it takes a first vm to cause its creation.
>> Confirmed, /dev/vmm does not exist until the first vm
>> is created.
>>
>> >
>> > >sysctl's mentiond above would be a poor way to make this
>determination.
>> >
>> > It would be nice if sysctls were better documented.
>>
>> Agreed.
>>
>> > If vmx.initialized is set once vmm has successfully loaded, I can't
>see a better way of checking for bhyve support (assuming it's not Intel
>specific). This entry definitely exists and is set to 0 if you load the
>module on a non-supported system, and set to 1 as soon as vmm loads on
>my Intel test system.
>>
>> Given its undocumented status you would be relying on an
>> undocumented feature that could change in either name or
>> behavior, and that is not desirable.
>>
>> Let me see if I can come up with something else.
>
>I looked at the code for bhyvectl, bhyveload and
>byhve.  They do not actually try to decide if vmm
>is supported or not, they simply process the error
>from a vm_create() or vm_open() call and exit
>with an error code if they can not handle it
>(some of the code can handle a vm_create failure
>if infact we are trying to create a vm that
>already exists).
>
>If you want to maintain full compatibility a similiar
>stratergy may be in order.
>
>Why is it that vm-bhyve specifically needs to know
>if the kernel has vmm support or not?
>Cant it just be written to handle the errors returned
>if the supported functions do not exist?

I think the question vm-bhyve wants to answer is: does the CPU have the 
required features to run a multicore VM.

These or similar sysctls do seem to be the correct way to communicate that 
support.


You are correct!

The question in case as I understood was about CPU feature supported, actually 
vmm(8) knows all this information! Some examples such like CPU with VMX 
unrestricted mode support (UG) that is necessary for guest VMs running with 
multiple vCPU or like VT-d necessary for PCI device passthrough.

I have a patch that exposes a sysctl saying what bhyve(8) is capable to run, 
however it needs to be polished a bit more to be more informative.
I think for third part software like vm-bhyve these information are crucial as 
these software can get advantage of these information prior to run a certain 
set that will end up in a fail because of a partial CPU support.

Best,


As mentioned in my first email, it does seem like some of these exist already 
in the way of
vmm.vmx.cap.* sysctls.

We could look at bhyve output and try to process that, but that seems more 
messy if there
are sysctls that expose support, especially as the vmm module does seem to know 
what
features the cpu/hardware supports. vm-bhyve has already forked into the 
background
by the time bhyve runs so can't easily provide feedback to the caller other 
than through
the log file.

We do also try and take action in some cases, such as reducing cpu count to 1 
if UG support
isn't found, rather than just having bhyve fail. (Of course you 

RE: Checking bhyve supported features (sysctls)

[Matt Churchyard]

> Hello,
> 
> I'm looking for better ways to check for bhyve support / available features 
> without trying to scan through dmesg output.

>Yes, it would be very good to remove that, as it usually tries to grep a 
>non-existent file /var/run/dmesg.boot that is not created until after vm_bhyve 
>has been called from /usr/local/etc/rc.d when you have things set to 
>autostartup >in /etc/rc.conf


> 
> I notice that the following 2 sysctl's appear to be set to 1 as soon 
> as the vmm module is loaded
> 
> hw.vmm.vmx.initialized: 1
> hw.vmm.vmx.cap.unrestricted_guest: 1
> 
> Will these be available on both Intel & AMD processors as a way to determine 
> if the module has loaded successfully and can run guests?
> 
> I also see the below sysctl related to iommu.
> 
> hw.vmm.iommu.initialized
> 
> Again, will this be set to 1 as soon as the module is loaded if iommu is 
> supported, or only when it is used?
> There also seems to be a vmm.amdvi.enable sysctl. Would both these need 
> checking or is vmm.iommu enough to determine support on any processor.

>Probalby the safest way for a shell script to decide if bhyve is up and 
>running is to stat /dev/vmm, if that exists then the modules have loaded and 
>initialized and bhyve should be ready to process guests.

Hmm, I don't get /dev/vmm unless I actually have running guests.

>sysctl's mentiond above would be a poor way to make this determination.

It would be nice if sysctls were better documented. If vmx.initialized is set 
once vmm has successfully loaded, I can't see a better way of checking for 
bhyve support (assuming it's not Intel specific). This entry definitely exists 
and is set to 0 if you load the module on a non-supported system, and set to 1 
as soon as vmm loads on my Intel test system.

Matt

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Checking bhyve supported features (sysctls)

[Matt Churchyard]

Hello,

I'm looking for better ways to check for bhyve support / available features 
without trying to scan through dmesg output.

I notice that the following 2 sysctl's appear to be set to 1 as soon as the vmm 
module is loaded

hw.vmm.vmx.initialized: 1
hw.vmm.vmx.cap.unrestricted_guest: 1

Will these be available on both Intel & AMD processors as a way to determine if 
the module has loaded successfully and can run guests?

I also see the below sysctl related to iommu.

hw.vmm.iommu.initialized

Again, will this be set to 1 as soon as the module is loaded if iommu is 
supported, or only when it is used?
There also seems to be a vmm.amdvi.enable sysctl. Would both these need 
checking or is vmm.iommu enough to determine support on any processor.

Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Curent Centos 7 and bhyve

[Matt Churchyard]

> > Rodney W. Grimes wrote:
> 
> [dd]
> 
> > > > > > 
> > > > > > Though it has a lot of features, it also has some short 
> > > > > > comings, like you can not spec a vm to be wired in memory, 
> > > > > > which IMHO is the only way to insure consistent VM performance.
> > > > > 
> > > > > Well, we have "bhyve_options" configuration option in the vm 
> > > > > config, why not put "-S" there, is that what you mean by 
> > > > > wiring the vm in memory?
> > > > 
> > > > I believe that fails as that only adds the -S to bhyve, and you 
> > > > must specify it both on bhyveload and bhyve for it to work.
> > > 
> > > I think it is totally doable becase vm-bhyve is nothing but a suit 
> > > of scripts. A PR with a feature request would be appropriate.
> > 
> > I made several attempts to contact the author at the email address 
> > provided at the git hub while making other bhyve changes to try and 
> > coordinate with him.  I got no response after 3 attempts,
> > so have stopped trying to contact them.   (This was while I was
> > adding the -c cpu topology modifications.)
> 
> > You can add yourself to
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230580
> > maybe something useful comes out of it.
> 
> I've already commented on the bug, although I'll reply here as well.
> If "-S" is found in bhyve_options it does currently affect both commands. I 
> have decided that a specific wired_memory option is useful though and will 
> add this to the next release.
> 
> The name limit has been increased to 32 since v1.2

>This is better, but still an artificial limit, the implementation of bhyve 
>allows this string to be any valid filename, the scripts should be designed to 
>allow for that as the valid limit.

As far as I'm aware bhyve is currently limited to 32 characters, although this 
obviously may change. We use the name in various situations (interface 
descriptions, console names, etc) so it's beneficial to make sure it's not 
excessively long, or contains characters that could cause problems when used in 
other places.

if(name == NULL || strlen(name) >= VM_MAX_NAMELEN)
   return (EINVAL);

#define VM_MAX_NAMELEN  32


>I modified the script so that the vm name is the last column, and removed the 
>length check all togeather, this allows for the string to be what ever length 
>and not mess with column widths.

>root@x230a:# vm list
>DATASTORE   LOADER  CPUMEMORYVNC  AUTOSTART
>STATENAME
>default bhyveload   1  128M  -No   
>Stopped  fb-bld-10-amd64
>default bhyveload   4  2048M -No   
>Stopped  fb-bld-11-amd64
>default bhyveload   4  1024M -No   
>Stopped  fb-bld-11-i386
>default bhyveload   1  128M  -No   
>Stopped  fb-bld-11.0-p1-amd64
>default bhyveload   1  128M  -No   
>Stopped  fb-bld-11.0-p1-i386
>default bhyveload   4  512M  -No   
>Stopped  fb-bld-11.1-amd64
>default bhyveload   4  512M  -No   
>Stopped  fb-bld-11.1-i386


>> I didn't realize the changes for cpu topology had actually made it 
>> into head, although I don't believe it's actually in a release yet?

>Yes, they are in head, the MFC has been delayed for other reasons.
>It is in the 12.0-ALPHA1 snapshot, and many before that.  

>> I will plan to support configuration and display of these before 12 
>> release.

>Thanks.  I think mostly just extract the NCPU's from the topology string.  The 
>code actually works now, but due to fixed column width assumptions the output 
>looks bad.

I've added some additional cpu_ settings to control the new cpu fields. "vm 
list" just shows the original cpu setting so this remains unmodified. "vm info" 
shows full settings.

> 
> Matt
> 
> > --
> > Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> > AS43859
> > ___
> > freebsd-virtualization@freebsd.org mailing list 
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to 
> > "freebsd-virtualization-unsubscr...@freebsd.org"
> 
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Curent Centos 7 and bhyve

[Matt Churchyard]

> Matt Churchyard wrote:

> wired_memory option has been added to next release.


> Thanks a lot, Matt! May I ask a question: how is the "-S" option supposed to 
> work if the loader is not bhyveload, but grub2-bhyve or UEFI?

In 1.2, if "-S" is found in bhyve_options, it adds this argument to the guest 
loader. This is done before choosing between bhyveload/grub2-bhyve and will be 
supplied to either.
In 1.3 (GitHub only), which has a new wired_memory option, you will need to use 
this, as the slightly hacky grep to scan bhyve_options for -S and push this 
through to the loader has been removed.

As far as I'm aware this is not an issue with UEFI as there is now just a 
single bhyve process (which gets all of bhyve_options) and no separate loader.

Matt

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Curent Centos 7 and bhyve

[Matt Churchyard]

-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of Victor Sudakov
Sent: 13 August 2018 10:53
To: Matt Churchyard 
Cc: Rodney W. Grimes ; 
freebsd-virtualization@freebsd.org
Subject: Re: Curent Centos 7 and bhyve

Matt Churchyard wrote:
> > Rodney W. Grimes wrote:
> 
> [dd]
> 
> 
> I've already commented on the bug, although I'll reply here as well.
> If "-S" is found in bhyve_options it does currently affect both 
> commands.

>Sorry, this is not working for me (vm-bhyve 1.1.8_2 from official packages):

>Aug 13 16:43:09: bhyve exited with status 1 Aug 13 16:43:09: destroying 
>network device tap0 Aug 13 16:43:09: stopped Aug 13 16:43:12: initialising Aug 
>13 16:43:12:  [loader: bhyveload] Aug 13 16:43:12:  [uefi: no] Aug 13 
>16:43:12:  [cpu: 1] Aug 13 16:43:12:  [memory: 1G] Aug 13 16:43:12:  
>[hostbridge: standard] Aug 13 16:43:12:  [com ports: com1] Aug 13 16:43:12:  
>[uuid: e148afdb-d4af-11e6-9a94-5404a6b49a66]
>Aug 13 16:43:12:  [utctime: yes]
>Aug 13 16:43:12:  [debug mode: no]
>Aug 13 16:43:12:  [primary disk: disk0.img] Aug 13 16:43:12:  [primary disk 
>dev: file] Aug 13 16:43:12: initialising network device tap0 Aug 13 16:43:12: 
>adding tap0 -> bridge0 (main) Aug 13 16:43:12: booting Aug 13 16:43:12: 
>bhyveload -c /dev/nmdm0A -m 1G -e autoboot_delay=3 -d /d02/vm/fido/disk0.img 
>fido Aug 13 16:43:17:  [bhyve options: -c 1 -m 1G -AHP -S -U 
>e148afdb-d4af-11e6-9a94-5404a6b49a66 -u] Aug 13 16:43:17:  [bhyve devices: -s 
>0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/d02/vm/fido/disk0.img -s 
>5:0,virtio-net,tap0,mac=58:9c:fc:01:f6:ac]
>Aug 13 16:43:17:  [bhyve console: -l com1,/dev/nmdm0A] Aug 13 16:43:17: 
>starting bhyve (run 1) Aug 13 16:43:17: bhyve exited with status 1 Aug 13 
>16:43:17: destroying network device tap0 Aug 13 16:43:17: stopped

Sorry, it looks like the change didn't make it into 1.1. You'd need to upgrade 
to 1.2, which is in ports but looks like it hasn't made it into pkg yet. 
portsmon.freebsd.org is still broken also so I can't easily confirm whether 
it's made it into any of the quarterly/latest pkg repos. (It was in ports over 
a month ago so should at least be in latest)

Matt

>Here is my complete config:

>guest="freebsd"
>loader="bhyveload"
>cpu=1
>memory=1G
>network0_type="virtio-net"
>network0_switch="main"
>disk0_type="virtio-blk"
>disk0_name="disk0.img"
>utctime="yes"
>uuid="e148afdb-d4af-11e6-9a94-5404a6b49a66"
>network0_mac="58:9c:fc:01:f6:ac"
>bhyve_options="-S"


>--
>Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
>AS43859
___
freebsd-virtualization@freebsd.org mailing list 
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Curent Centos 7 and bhyve

[Matt Churchyard]

> Rodney W. Grimes wrote:

[dd]

> > > > > 
> > > > > Though it has a lot of features, it also has some short 
> > > > > comings, like you can not spec a vm to be wired in memory, 
> > > > > which IMHO is the only way to insure consistent VM performance.
> > > > 
> > > > Well, we have "bhyve_options" configuration option in the vm 
> > > > config, why not put "-S" there, is that what you mean by wiring 
> > > > the vm in memory?
> > > 
> > > I believe that fails as that only adds the -S to bhyve, and you 
> > > must specify it both on bhyveload and bhyve for it to work.
> > 
> > I think it is totally doable becase vm-bhyve is nothing but a suit 
> > of scripts. A PR with a feature request would be appropriate.
> 
> I made several attempts to contact the author at the email address 
> provided at the git hub while making other bhyve changes to try and 
> coordinate with him.  I got no response after 3 attempts,
> so have stopped trying to contact them.   (This was while I was
> adding the -c cpu topology modifications.)

> You can add yourself to
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230580
> maybe something useful comes out of it.

I've already commented on the bug, although I'll reply here as well.
If "-S" is found in bhyve_options it does currently affect both commands. I 
have decided that a specific wired_memory option is useful though and will add 
this to the next release.

The name limit has been increased to 32 since v1.2

I didn't realize the changes for cpu topology had actually made it into head, 
although I don't believe it's actually in a release yet? I will plan to support 
configuration and display of these before 12 release.

Matt

> --
> Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
> AS43859
> ___
> freebsd-virtualization@freebsd.org mailing list 
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve server 2016 lockup

[Matt Churchyard]

Hello,

I have a server 2016 virtual machine which I only created for testing and don't 
use much. Booting it up recently I've found that it locks up consistently 
within a few minutes of booting. What can I do to try and figure out what's 
causing the problem?

Host was recently updated to 11.2
# uname -a
FreeBSD dev.--- 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r335510: Fri Jun 22 
04:32:14 UTC 2018 r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC 
 amd64

Bhyve command is as follows
# bhyve -c 2 -m 2G -Huwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
  -U 841e8764-75f4-11e8-b2e3-50e549369bc6 -l com1,stdio
  -s 0,hostbridge -s 31,lpc -s 3:0,ahci-cd,/vm/.config/null.iso -s 
4:0,ahci-hd,/vm/w2016/disk0.img -s 5:0,e1000,tap1,mac=58:9c:fc:08:8e:70
  -s 6:0,fbuf,tcp=0.0.0.0:5900 -s 7:0,xhci,tablet w2016

Top output -
  PID USERNAMETHR PRI NICE   SIZERES STATE   C   TIMEWCPU COMMAND
29796 root 22  200  2122M   944M kqread  0  24:59 113.13% bhyve

Regards,
Matt Churchyard

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Issue encountered booting FreeBSD STABLE and CURRENT snapshots with EFI

[Matt Churchyard]

-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
 On Behalf Of Kyle Evans
Sent: 23 March 2018 13:06
To: Joe Maloney 
Cc: Warner Losh ; freebsd-virtualization@freebsd.org
Subject: Re: Issue encountered booting FreeBSD STABLE and CURRENT snapshots 
with EFI

On Fri, Mar 23, 2018 at 3:56 AM, Joe Maloney  wrote:
> We narrowed the issue down to how vm-bhyve attaches a null.iso when 
> starting the VM.
>

>What exactly are the contents of this null.iso? It sounds like we're just 
>ultimately choosing the wrong partition to boot with the null.iso attached. 
>I'd be interested to see if a loader.efi built with >D13784 [1] applied 
>resolves this both replacing /boot/loader.efi initially then while installed 
>on the ESP.

>Do note that I'm pretty sure we still had some small outstanding issues with 
>D13784's loader.efi being installed as /boot/loader.efi (with console 
>handling), so it may not be a good long-term solution, >but this should 
>hopefully land in the not-so-distant future.

>[1] https://reviews.freebsd.org/D13784

The original reason for null.iso was due to notes on the bhyve wiki
https://wiki.freebsd.org/bhyve/Windows

" The install.iso is only required for the first boot of Windows Server and 
must be removed from the bhyve command after the first boot. Desktop editions 
of Windows require that a null install.iso file remains and it can be created 
with touch install.iso"

At the time the UEFI support was primarily used for loading Windows guests, and 
some desktop versions apparently needed an empty iso file to boot. I was not 
able to verify which guests needed this file and so it was supplied to all, as 
it seemed to have no ill effects on the server versions of Windows I was 
testing with.

As mentioned above is this more an issue with the boot process? If bhyve at 
some point allows an "empty" cd device to be attached, with the ability to 
insert an iso on the fly, like many other hypervisors do, would that trigger 
the same issue?

Matt

> For now a hack like so to vm-bhyve can get around the issue:
>
> https://github.com/araujobsd/vm-bhyve/commit/29db2d6c6ec4a29578dc11190
> 3107f25a78cdf82
>
> This may simply be an issue with vm-bhyve attaching an invalid iso 
> image to the VM, and I would conclude it is odd that it does so.  Or 
> there may still be an issue that affects even the latest 03-22-18 
> snapshots for example if other media is present when booting with 
> bhyve, or natively.  I would need to do some more testing at a later 
> date to confirm but just wanted to pass along what was discovered to be the 
> root cause thus far.
>
> On Friday, March 16, 2018, Marcelo Araujo  wrote:
>>
>> 2018-03-16 9:55 GMT+08:00 Kyle Evans :
>>
>> > On Thu, Mar 15, 2018 at 8:46 PM, Marcelo Araujo 
>> > 
>> > wrote:
>> > >
>> > >
>> > > 2018-03-16 7:07 GMT+08:00 Kyle Evans :
>> > >>
>> > >> On Thu, Mar 15, 2018 at 5:01 PM, Kyle Evans 
>> > >> wrote:
>> > >> > On Thu, Mar 15, 2018 at 4:09 PM, Peter Grehan 
>> > >> > 
>> > >> > wrote:
>> > >> >>> I believe the problem may have been introduced with this commit:
>> > >> >>> > >
>> > >> >>> https://svnweb.freebsd.org/base/stable/?view=log=329
>> > >> >>> 114
>> > >> >>
>> > >> >>  Any chance of being able to work out where in that list of 
>> > >> >> commits
>> > in
>> > >> >> CURRENT the loader stopped working ?
>> > >> >>
>> > >> >
>> > >> > Indeed- if you could work out the exact commit in that range 
>> > >> > from head that caused it, I wouldn't think it to be a tough 
>> > >> > fix. After tonight I'm out until Sunday, but should have time 
>> > >> > Sunday or Monday to try and diagnose it further.
>> > >>
>> > >> Can one of you try this with boot1.efi+loader.efi built from 
>> > >> today's head stand/? I'm not sure what I'm expecting here since 
>> > >> these are among my first times trying bhyve, but this is what 
>> > >> I'm seeing now (vs. from the mentioned head snapshot where I 
>> > >> noted similar behavior as originally mentioned):
>> > >>
>> > >> 1.) Get to loader.efi, menu is good
>> > >> 2.) Break into loader prompt
>> > >> 3.) `lsdev`- pager is restricted to the line the prompt is on, 
>> > >> so the output is useless
>> > >> 4.) `boot`
>> > >> 5.) "Unhandled ps2 mouse command 0xe1"
>> > >>
>> > >> At this point, the boot looks screwed until I VNC into it- it 
>> > >> booted fine here, but the console stopped working after the kernel 
>> > >> handoff.
>> > >>
>> > >> Thanks,
>> > >>
>> > >> Kyle Evans
>> > >> ___
>> > >> freebsd-virtualization@freebsd.org mailing list 
>> > >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualizatio
>> > >> n To unsubscribe, send any mail to 
>> > >> 

Re: Multiple bhyve Guests, Single bridge/tap?

[Matt Churchyard]

As mentioned a bridge is the virtual equivalent of a switch. It only really
makes sense to have more than one bridge if you have more than one
interface on your guest(s), and want to connect those interfaces to
separate networks. (Or you want some guests on a different network,
possibly bridged to a different physical interface).

If you want to provide complete network separation between guests, it's
much easier to just use the 'private' option to ifconfig when bridging the
guest's tap interface. Any bridge member set to private can not talk to any
other private bridge member. Of course this is only really applicable in
multi-tenant situations like Aryeh says. If they are all your own guests,
the fact that they can see each other on the network should hopefully be a
non-issue.

Matt

On Thu, 29 Dec 2016 at 15:26, Aryeh Friedman 
wrote:

> On Thu, Dec 29, 2016 at 10:19 AM, Vincent Olivier  wrote:
>
>
>
> > Hi!
>
> >
>
> > > Use the same bridge but a different tap (each tap represents the
> virtual
>
> > equivalent of a NIC where the bridge is the virtual equivalent of a hub)
>
> >
>
> >
>
> > Thanks! This is very clear. For extra isolation, could I use a new bridge
>
> > too or is that useless?
>
> >
>
>
>
> Yes but it only makes sense in a multi-tenant (aka cloud provider) setup
>
> because any attacker on a VM should be assumed to able to get into the host
>
> due to knowing your password (which typically is not all that different on
>
> the two machines unless you randomly generated it).
>
>
>
> --
>
> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org
>
> ___
>
> freebsd-virtualization@freebsd.org mailing list
>
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Windows 2016

[Matt Churchyard via freebsd-virtualization]

On Tue, Nov 1, 2016 at 8:58 AM, Matt Churchyard via freebsd-virtualization 
<freebsd-virtualization@freebsd.org<mailto:freebsd-virtualization@freebsd.org>> 
wrote:
On Tue, Nov 01, 2016 at 03:16:12PM +0100, Daniel Tihanyi wrote:
> On Tue, 2016-11-01 at 08:02 -0600, The Doctor wrote:
> > On Tue, Nov 01, 2016 at 09:02:57AM +, Matt Churchyard wrote:
> > >
> > > >
> > > > I was able to install the Windows 2016 using the GUI method.
> > > >
> > > > One problem.
> > > >
> > > > It is assigned an IP but
> > > >
> > > > the netmask and the default route is not showing up.
> > > >
> > > > Also??The virtual Windows 2016 box does not see a network
> > > > interface.
> > > This makes very little sense. You usually specify the netmask and
> > > default route when you assign Windows an IP address.
> > > And how have you assigned the guest an IP address if it has no
> > > network interface?
> > >
> > The taps are allocated an IP address.
> >
> > I fully concur.
> >
> > In the install at one point, I declare that the IP address, netmask
> > and gateway are 'declared' at the 'MB ' BIOS interface.
> >
> >
> > >
> > > >
> > > > What must I do to convince this box that is it on a switch
> > > On the bhyve host it should be configured just like any other
> > > bhyve virtual machine. The guest should have a virtio-net device,
> > > which is linked to a tap interface on the host. That tap interface
> > > should be bridged with whichever physical network adapter you want
> > > the guest connected to.
> > >
> > > However, I think I mentioned in a previous message that Windows
> > > does not have the virtio-net drivers by default. You need to boot
> > > the guest with the virtio-net driver ISO attached and install the
> > > driver. You should see the interface in Device Manager flagged as
> > > not installed/working.
> > >
> >
> > That is correct .
> >
> > I do have an iso of the MB drivers,
> >
> > namely a Supermicro??X10DRW-i MB?
> >
> > and?
> >
> > from Intel I did obtain the .exe file for the i350GbE drivers.
> > I script a 2 phase installation as follows:
> >
> > 1)
> >
> >
> > /usr/sbin/bhyve -c 2 -m 4G -w -H -s 3,ahci-cd,./.iso/14393.0.160715-
> > 1616.RS1_RELEASE_SERVER_EVAL_X64FRE_EN-US.ISO -s 4,ahci-
> > hd,windows2016.img -s 5,virtio-net,tap15 -s
> > 29,fbuf,tcp=0.0.0.0:5900<http://0.0.0.0:5900>,w=800,h=600,wait -s 
> > 30,xhci,tablet -s
> > 31,lpc -l com1,stdio -l
> > bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
> > windows2016guest
> >
> > 2)
> >
> >
> > /usr/sbin/bhyve -c 2 -m 4G -w -H -s
> > 3,ahci-cd,./.iso/null-install.iso -s 4,ahci-hd,windows2016.img -s
> > 5,virtio-net,tap15 -s 
> > 29,fbuf,tcp=0.0.0.0:5900<http://0.0.0.0:5900>,w=1024,h=768,wait -s
> > 30,xhci,tablet -s 31,lpc -l com1,/dev/nmdm15A -l
> > bootrom,/usr/local/share/uefi- firmware/BHYVE_UEFI.fd
> > windows2016guest &
> >
> > As I said, I have picked up an ISO from the MB manufacturer
> >
> > let's call it manufacturer.iso .
> >
> > Where do I place said iso in the 'phase' of things?
>
> You use virtio network adapters, you have to install the virtio
> drivers and NOT the drivers for the physical interface. Then configure
> the IP Address, gateway, etc. in Windows.

>All right, let me repeat what I am up aginst.

>I do set the IP configuration in the virtual BIOS.

>However when the Win2016 VM boots it sees

>1) the localhost

>2) its assigned IP address

>3) A yellow triangle where the ethernet adaptor is.
I'm not sure what people find so difficult about this - Just treat it like it 
was a real Windows machine

1) Do not assign the guest's IP address on the host
2) Do not set the IP in the guest BIOS (I'm not even sure what you mean by this)

As you have found, the virtio-net network adapter has a yellow triangle in 
Windows, because it is not installed. You need to run bhyve but replace the 
Windows ISO with the virtio-net driver ISO, which you can download off the 
Internet; Then install the network driver through VNC. The guest has a 
"virtio-net" interface which is created by bhyve - This has nothing to do with 
your motherboard and you do not need to install any of the motherboard or 
physical interface drivers in the guest.

Once you have the virtio driver installed, you will have a "Redhat VirtIO" 
network adapter in Windows, and y

RE: Windows 2016

[Matt Churchyard via freebsd-virtualization]

On Tue, Nov 01, 2016 at 03:16:12PM +0100, Daniel Tihanyi wrote:
> On Tue, 2016-11-01 at 08:02 -0600, The Doctor wrote:
> > On Tue, Nov 01, 2016 at 09:02:57AM +0000, Matt Churchyard wrote:
> > > 
> > > > 
> > > > I was able to install the Windows 2016 using the GUI method.
> > > > 
> > > > One problem.
> > > > 
> > > > It is assigned an IP but
> > > > 
> > > > the netmask and the default route is not showing up.
> > > > 
> > > > Also??The virtual Windows 2016 box does not see a network 
> > > > interface.
> > > This makes very little sense. You usually specify the netmask and 
> > > default route when you assign Windows an IP address.
> > > And how have you assigned the guest an IP address if it has no 
> > > network interface?
> > > 
> > The taps are allocated an IP address.
> > 
> > I fully concur.
> > 
> > In the install at one point, I declare that the IP address, netmask 
> > and gateway are 'declared' at the 'MB ' BIOS interface.
> > 
> > 
> > > 
> > > > 
> > > > What must I do to convince this box that is it on a switch
> > > On the bhyve host it should be configured just like any other 
> > > bhyve virtual machine. The guest should have a virtio-net device, 
> > > which is linked to a tap interface on the host. That tap interface 
> > > should be bridged with whichever physical network adapter you want 
> > > the guest connected to.
> > > 
> > > However, I think I mentioned in a previous message that Windows 
> > > does not have the virtio-net drivers by default. You need to boot 
> > > the guest with the virtio-net driver ISO attached and install the 
> > > driver. You should see the interface in Device Manager flagged as 
> > > not installed/working.
> > > 
> > 
> > That is correct .
> > 
> > I do have an iso of the MB drivers,
> > 
> > namely a Supermicro??X10DRW-i MB?
> > 
> > and?
> > 
> > from Intel I did obtain the .exe file for the i350GbE drivers.
> > I script a 2 phase installation as follows:
> > 
> > 1)
> > 
> > 
> > /usr/sbin/bhyve -c 2 -m 4G -w -H -s 3,ahci-cd,./.iso/14393.0.160715- 
> > 1616.RS1_RELEASE_SERVER_EVAL_X64FRE_EN-US.ISO -s 4,ahci- 
> > hd,windows2016.img -s 5,virtio-net,tap15 -s 
> > 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait -s 30,xhci,tablet -s 
> > 31,lpc -l com1,stdio -l 
> > bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
> > windows2016guest
> > 
> > 2)
> > 
> > 
> > /usr/sbin/bhyve -c 2 -m 4G -w -H -s 
> > 3,ahci-cd,./.iso/null-install.iso -s 4,ahci-hd,windows2016.img -s 
> > 5,virtio-net,tap15 -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait -s 
> > 30,xhci,tablet -s 31,lpc -l com1,/dev/nmdm15A -l 
> > bootrom,/usr/local/share/uefi- firmware/BHYVE_UEFI.fd 
> > windows2016guest &
> > 
> > As I said, I have picked up an ISO from the MB manufacturer
> > 
> > let's call it manufacturer.iso .
> > 
> > Where do I place said iso in the 'phase' of things?
> 
> You use virtio network adapters, you have to install the virtio 
> drivers and NOT the drivers for the physical interface. Then configure 
> the IP Address, gateway, etc. in Windows.

>All right, let me repeat what I am up aginst.

>I do set the IP configuration in the virtual BIOS.

>However when the Win2016 VM boots it sees

>1) the localhost

>2) its assigned IP address

>3) A yellow triangle where the ethernet adaptor is.

I'm not sure what people find so difficult about this - Just treat it like it 
was a real Windows machine

1) Do not assign the guest's IP address on the host
2) Do not set the IP in the guest BIOS (I'm not even sure what you mean by this)

As you have found, the virtio-net network adapter has a yellow triangle in 
Windows, because it is not installed. You need to run bhyve but replace the 
Windows ISO with the virtio-net driver ISO, which you can download off the 
Internet; Then install the network driver through VNC. The guest has a 
"virtio-net" interface which is created by bhyve - This has nothing to do with 
your motherboard and you do not need to install any of the motherboard or 
physical interface drivers in the guest.

Once you have the virtio driver installed, you will have a "Redhat VirtIO" 
network adapter in Windows, and you can assign an IP/Netmask/Gateway to this 
just as you normally would in Windows.

Forget about the Linux machines. They have the virtio-net driver built in, so 
you can just configure eth0 out-of-the-box like normal. If you've configured 
their IP addresses on the bhyve host then that's not really the correct way to 
configure things, even if it doesn't actually break anything; The IP address 
should be configured inside the guest OS.

Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Windows 2016

[Matt Churchyard via freebsd-virtualization]

>I was able to install the Windows 2016 using the GUI method.

>One proble.

>It is assigned an IP but

>the netmask and the default route is not showing up.

>Also  The virtual Windows 2016 box does not see a network interface.

This makes very little sense. You usually specify the netmask and default route 
when you assign Windows an IP address.
And how have you assigned the guest an IP address if it has no network 
interface?

>What must I do to convince this box that is it on a switch

On the bhyve host it should be configured just like any other bhyve virtual 
machine. The guest should have a virtio-net device, which is linked to a tap 
interface on the host. That tap interface should be bridged with whichever 
physical network adapter you want the guest connected to.

However, I think I mentioned in a previous message that Windows does not have 
the virtio-net drivers by default. You need to boot the guest with the 
virtio-net driver ISO attached and install the driver. You should see the 
interface in Device Manager flagged as not installed/working.


Just one other thing to add to this. I'm not sure if this is the case here, but 
it's something I've seen a few times now.
When configuring IP addresses, etc for a guest, this should be done -inside- 
the guest.
Do not assign a guest's IP address to the host, or to the tap interface.
Bhyve works just like any other hypervisor. IP settings are done in the guest 
as if it was a real machine, the host just acts like a switch.


Matt

>192.168.0.60

>Thar is handling the default routing to the interent.

>For argument sake  ,

>the Windows server is IP as 192.168.0.68


>I have an ubuntu up at 192.168.0.57
>a Centos at 192.168.0.54
>and
>Fedora at 192.168.0.53

>all running successfully.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Windows 2016

[Matt Churchyard via freebsd-virtualization]

>I was able to install the Windows 2016 using the GUI method.

>One proble.

>It is assigned an IP but

>the netmask and the default route is not showing up.

>Also  The virtual Windows 2016 box does not see a network interface.

This makes very little sense. You usually specify the netmask and default route 
when you assign Windows an IP address.
And how have you assigned the guest an IP address if it has no network 
interface?

>What must I do to convince this box that is it on a switch

On the bhyve host it should be configured just like any other bhyve virtual 
machine. The guest should have a virtio-net device, which is linked to a tap 
interface on the host. That tap interface should be bridged with whichever 
physical network adapter you want the guest connected to.

However, I think I mentioned in a previous message that Windows does not have 
the virtio-net drivers by default. You need to boot the guest with the 
virtio-net driver ISO attached and install the driver. You should see the 
interface in Device Manager flagged as not installed/working.

Matt

>192.168.0.60

>Thar is handling the default routing to the interent.

>For argument sake  ,

>the Windows server is IP as 192.168.0.68


>I have an ubuntu up at 192.168.0.57
>a Centos at 192.168.0.54
>and
>Fedora at 192.168.0.53

>all running successfully.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Installing SQL Server on Windows 2012 bhyve guests

[Matt Churchyard via freebsd-virtualization]

-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
[mailto:owner-freebsd-virtualizat...@freebsd.org] On Behalf Of Peter Grehan
Sent: 25 October 2016 19:19
To: Randy Terbush
Cc: freebsd-virtualization@freebsd.org
Subject: Re: Installing SQL Server on Windows 2012 bhyve guests

>Hi Randy,

> I've followed the info provided here 
> https://wiki.freebsd.org/bhyve/Windows
> to create a Windows Server 2012 image that I successfully installed on 
> the bhyve hypervisor. After getting a working Win2012 image running, 
> applying updates, etc. I attempted to install SQL Server 2014. In the 
> late stages of that install, the process attempts to start the SQL server 
> engine and fails.

>  There have been some reports of this previously. What might be happening is 
> that SQL Server has stricter requirements on the underlying block >size than 
> NTFS itself has.

 > Since you are using a file-backed image, the reported block size from the 
 > emulated storage controller may be 8KB or even higher, depending on >the 
 > underlying filesystem type.

>  A suggestion is to force the block size to 4KB (has to be done during 
> install as well), and if the problem persists, try 512 bytes. This is done by 
> using >the 'sectorsize=' parameter to the disk configuration e.g.

>  -s 4,ahci-hd,/path/to/disk.img,sectorsize=4096

Just to add, if you're using vm-bhyve you should be able to add this with the 
diskX_opts template/config option

disk0_opts="sectorsize=4096"

I may adjust the sample templates to include a sectorsize option for Windows 
guests, although I'm not 100% clear on what exact OS versions or applications 
require specific settings. I do know that SQL Server has been reported to 
suddenly fall over unless a valid block size is used (although I believe it did 
install and run), but again I'm not sure exactly what the "correct" settings 
are.

Also note that you shouldn't need any grub_* template options if you're using 
UEFI. Those options only apply to grub-bhyve.

Matt

>later,

>Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Windows 2016 Server

[Matt Churchyard via freebsd-virtualization]

>On Mon, Oct 24, 2016 at 08:07:13AM +0000, Matt Churchyard wrote:
> On Fri, Oct 21, 2016 at 03:21:10PM -0700, Peter Grehan wrote:
> > >> [Windows could not parse or process unattend answer file 
> > >> [D:\autounattend.xml] for pass [windowsPE]. The answer file is 
> > >> invalid.]
> > >
> > > Something is wrong with your autounattend.xml file.
> > 
> >   Still working on the 2k16 unattend file. Unfortunately things have 
> > changed from TP5 in a non-obvious way :(
> > 
> >   A workaround is to do a GUI install.
> > 
> > later,
> > 
> > Peter.
> 
> > Please explain what you mean by a GUI install.
> 
> Not sure if anyone's replied directly to you -
> 
> 1) Make sure you are running at least 11-RELEASE
> 
> 2) Use an "off-the-shelf" Windows install CD
> 
> 3) Add the following to your bhyve command
> 
> -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait
> -s 30,xhci,tablet
> 
> 4) Run bhyve and then use a VNC client to connect and install using 
> the normal Windows GUI
>

>Well all right did follow https://wiki.freebsd.org/bhyve/UEFI


>My next question is after the initial install is done, how do I followup on 
>the next step , i.e. to see if the VM is booting.

As long as you have the fbuf device specified you can connect to the IP address 
of the bhyve host using VNC to view the "physical" guest console and watch it 
boot. Usually with Windows you'd then use the VNC console to log in and set up 
permanent RDP access.

One other issue is that Windows doesn't support the virtio-net network device 
by default.  The easiest way to get this working is to boot the guest with the 
virtio driver ISO attached instead of the Windows install disk (once Windows is 
installed and working). You can then install the driver for the network 
interface from the CD using the VNC console.

Depending on what you are doing you may find it easier to use something like 
iohyve/chyves/vm-bhyve/vmrc that handles all the raw bhyve commands for you.

Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Windows 2016 Server

[Matt Churchyard via freebsd-virtualization]

On Fri, Oct 21, 2016 at 03:21:10PM -0700, Peter Grehan wrote:
> >> [Windows could not parse or process unattend answer file 
> >> [D:\autounattend.xml] for pass [windowsPE]. The answer file is 
> >> invalid.]
> >
> > Something is wrong with your autounattend.xml file.
> 
>   Still working on the 2k16 unattend file. Unfortunately things have 
> changed from TP5 in a non-obvious way :(
> 
>   A workaround is to do a GUI install.
> 
> later,
> 
> Peter.

> Please explain what you mean by a GUI install.

Not sure if anyone's replied directly to you -

1) Make sure you are running at least 11-RELEASE

2) Use an "off-the-shelf" Windows install CD

3) Add the following to your bhyve command

-s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait
-s 30,xhci,tablet

4) Run bhyve and then use a VNC client to connect and install using the normal 
Windows GUI

Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Bhyve tests and findings

[Matt Churchyard via freebsd-virtualization]

> [..]
> 
> You just need the vm name (win) on the end:
> 
> # bhyve [options] vm-name
> 

>Sorry, I was being stupid. With the vm name on the end, it just exits with 
>error code 1 and without printing anytning.
>I have recompiled bhyve itself and the libvmmapi, maybe it's not enough? The 
>kernel module? It still remains that from 10.3

Definitely sounds like something not right with bhyve; It shouldn't exit like 
that.
You can try building vmm from 11, although I'm not really a kernel dev and 
can't help much with trying to merge 11 code into 10.

Matt

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Bhyve tests and findings

[Matt Churchyard via freebsd-virtualization]

> > > > > > 
> > > > > > uefi="yes"
> > > > > > graphics="yes"
> > > > > 
> > > > > Dear Colleagues,
> > > > > 
> > > > > Can I enjoy and test all those nice "UEFI-GOP" features on a
> > > > > 10.3-RELEASE system, or do I have to install CURRENT for that?
> > > > 
> > > > AFAIK you can checkout usr.sbin/bhyve from the 11 branch and compile it
> > > > on 10.3.
> > > 
> > > I'm not sure how to compile something from a different branch without
> > > compiling the whole world. make what?
> > 
> > For bhyve it's easy. Check out the sources, go to usr.sbin/bhyve and run
> > 
> > make depend; make obj; make; make install. 
> 
> >Unfortunately, after all my efforts, with libvmmapi and bhyve compiled from
> >CURRENT sources:
> 
> >Aug 09 12:46:52: warning; UEFI graphics is only available in FreeBSD 11 and 
> >newer
> >Aug 09 12:46:52: booting
> 
> >Bummer!
> 
> It's possible that you do have a bhyve system with graphics support. 
> Unfortunately vm-bhyve isn't aimed at users who have manually merged changes, 
> and will blindly tell anyone that isn't running 11+ that they don't have 
> graphics support. 
> 
> You have a couple of options -
> 
> 1) Run bhyve manually rather than using vm-bhyve
> 2) Upgrade to the 11 beta
> 3) Modify vm-bhyve to not automatically skip graphics support if the version 
> is less than 11 

>Aha! I've commented out the version check from vm-run, and now when
>trying to install, I get:

>авг 09 17:01:05: initialising
>авг 09 17:01:05:  [loader: none]
>авг 09 17:01:05:  [uefi: yes]
>авг 09 17:01:05:  [cpu: 1]
>авг 09 17:01:05:  [memory: 1G]
>авг 09 17:01:05:  [hostbridge: standard]
>авг 09 17:01:05:  [com ports: com1]
>авг 09 17:01:05:  [uuid: 093f9134-5e18-11e6-9502-5404a6b49a66]
>авг 09 17:01:05:  [utctime: no]
>авг 09 17:01:05:  [debug mode: no]
>авг 09 17:01:05:  [primary disk: disk0.img]
>авг 09 17:01:05:  [primary disk dev: file]
>авг 09 17:01:05: generated static mac 58:9c:fc:03:77:3b (based on 
>'win:0:1470736865:0')
>авг 09 17:01:05: initialising network device tap5
>авг 09 17:01:05: adding tap5 -> bridge1 (isolated)
>авг 09 17:01:05: dynamically allocated port 5900 for vnc connections
>авг 09 17:01:05: booting
>авг 09 17:01:05:  [bhyve options: -c 1 -m 1G -Hwl 
>bootrom,/d02/vm/.config/BHYVE_UEFI.fd -U 093f9134-5e18-11e6-9502-5404a6b49a66]
>авг 09 17:01:05:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 
>4:0,ahci-hd,/d02/vm/win/disk0.img -s 5:0,virtio-net,tap5,mac=58:9c:fc:03:77:3b 
>-s >6:0,fbuf,tcp=0.0.0.0:5900,wait]
>авг 09 17:01:05:  [bhyve console: -l com1,/dev/nmdm0A]
>авг 09 17:01:05:  [bhyve iso device: -s 
>3:0,ahci-cd,/d02/vm/.iso/bhyve_win7_x64.iso]
>авг 09 17:01:05: starting bhyve (run 1)
>авг 09 17:01:05: bhyve exited with status 1
>авг 09 17:01:05: destroying network device tap5
>авг 09 17:01:05: stopped

Looks like bhyve is exiting immediately. Easiest thing is to probably add 
debug="yes" to  the guest config file then try and run it again. Any output 
from the bhyve command will be written to bhyve.log (note that's a different 
file to vm-bhyve.log).

Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Bhyve tests and findings

[Matt Churchyard via freebsd-virtualization]

>Lars Engels wrote:
> > > > > 
> > > > > uefi="yes"
> > > > > graphics="yes"
> > > > 
> > > > Dear Colleagues,
> > > > 
> > > > Can I enjoy and test all those nice "UEFI-GOP" features on a
> > > > 10.3-RELEASE system, or do I have to install CURRENT for that?
> > > 
> > > AFAIK you can checkout usr.sbin/bhyve from the 11 branch and compile it
> > > on 10.3.
> > 
> > I'm not sure how to compile something from a different branch without
> > compiling the whole world. make what?
> 
> For bhyve it's easy. Check out the sources, go to usr.sbin/bhyve and run
> 
> make depend; make obj; make; make install. 

>Unfortunately, after all my efforts, with libvmmapi and bhyve compiled from
>CURRENT sources:

>Aug 09 12:46:52: warning; UEFI graphics is only available in FreeBSD 11 and 
>newer
>Aug 09 12:46:52: booting

>Bummer!

It's possible that you do have a bhyve system with graphics support. 
Unfortunately vm-bhyve isn't aimed at users who have manually merged changes, 
and will blindly tell anyone that isn't running 11+ that they don't have 
graphics support. 

You have a couple of options -

1) Run bhyve manually rather than using vm-bhyve
2) Upgrade to the 11 beta
3) Modify vm-bhyve to not automatically skip graphics support if the version is 
less than 11 

Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: A couple of newbie questions

[Matt Churchyard via freebsd-virtualization]

Colleagues,

>I like bhyve very much, and have sucessfully run FreeBSD and Ubuntu
>16.04 server in FreeBSD 10.3 bhyve, with vm-bhyve as a shell.
>Now I am trying to boot Windows 7 but have not succeeded so far. 

>However there are things I don't quite understand. A couple of questions, if 
>you allow.

>1. Why is it that for some guest systems, there are two stages: first 
>bhyveload or grub2-bhyve and then bhyve itself. And for UEFI systems there is 
>>only one stage.  Does it mean that eventually bhyveload and grub2-bhyve will 
>become totally obsolete and the one-stage VM startup procedure >will become 
>the universal method?

Originally the quickest and easiest way to get guests running on bhyve was to 
load the guest into memory "manually", then run it. This started with just 
bhyveload to load FreeBSD guests, then expanded to supporting various other 
guests with grub-bhyve. UEFI was required to get Windows running (and makes 
bhyve act a bit more like other hypervisors) but took a lot of effort.

UEFI does seem to be the best way to run most guests. Linux guests have always 
been a bit finicky unless they have grub2 installed, and the ability to get 
remote access to the console makes things like a bhyve web frontend feasible.

Bhyveload is still a pretty quick and useful way to run FreeBSD guests though, 
and I don't think that or grub-bhyve will go anywhere.

>2. All this fbuf/VNC stuff looks cool, but I don't quite understand. You can 
>see the guest OS's console in VNC, like the Windows desktop, or only >the UEFI 
>shell, and then you have to access the guest OS via RDP/ssh etc ?

With the frame buffer enabled you see the full guest OS in vnc, same as you 
would in Virtualbox/VMWare/etc.

Matt

>TIA for explanations.

>--
>Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
>sip:suda...@sibptus.tomsk.ru

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Bhyve tests and findings

[Matt Churchyard via freebsd-virtualization]

> 
> uefi="yes"
> graphics="yes"

> Dear Colleagues,

> Can I enjoy and test all those nice "UEFI-GOP" features on a 10.3-RELEASE 
> system, or do I have to install CURRENT for that?

I believe you should also be able to use the 11 beta.

Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

FW: Bhyve tests and findings

[Matt Churchyard via freebsd-virtualization]

(sent this to the wrong list...)

-Original Message-
From: Matt Churchyard 
Sent: 04 August 2016 12:57
To: 'Michael Reifenberger'
Cc: freebsd...@freebsd.org
Subject: RE: Bhyve tests and findings

>Hi,
>after waiting for UEFI-GOP and using bhyve (with vm-bhyve as a convenient 
>tool) an a new E3-1225 v5 based system, the following are my findings >so far.
>(BTW:
>   Currently my rational for running bhyve is twofold:
>   - Run a Windows 8.1 or 10 instance for accessing different remote locations 
> via different VPN solutions
>   - Run Centos7/RHEL7 instances with SAP Systems on it
>)

>But first a huge Thanks to all who worked on bhyve and made it usable in its 
>current state.
>That alone is quite impressive.

>What works so far:
>- Windows 8, 8.1 and 10 installs and runs in graphical mode flawlessly.
>- Centos7 installs and runs too using the UEFI-GOP Image (Yeah, no more 
>Grub fiddling :-)
>- I was able to graphically Restore/Reconfigure a Acronis Windows-Backup into 
>a Bhyve instance
>   using the Acronis Restore-CD (Converting a BIOS Win8.1 to UEFI 
>Win8.1)


> What doesn't:
>- Only vnclient from FreeBSD can connect to the bhyve VNC Server.
>   I havn't found any vncviewer running on Windows which where able to 
>work (tried UltraVNC, RealVNC, ...)

I've been using TightVNC on Windows (first free client I came across) since GOP 
support came out and have not had any problems with it.

> - in VNC only most basic Keys work most special characters like (*\@) 
> (and of course no german localization)  but at least a usual US-kbd would be 
> helpful.

It's not clear whether you're saying that those special character have a 
problem or not?
I'm in the UK and these keys seem to work fine in the VNC client I'm using.

 >  (Is there a way to debug the keystrokes or duplicate a localized VNC kbd 
 > from some VNC server)
>- For the SAP-Systems it seems that only 4 disks get used when the disk type 
>is virtio-blk.
>   (Is this intentionally or a feature of vm-bhyve? How to provide more 
>disks)

I know that the UEFI firmware only connects slots 3/4/5/6, but I don't know if 
some guests can support non-boot disks on other slots. In my bhyve manager I 
used to limit Windows to 3 disks (+1 cd), however I have tested 8 disks in 
Server2012 with a FreeBSD 12 host, which allows up to 32 disks per ahci 
controller.

>- It seems to miss a way to add an ISO CD/DVD without booting from it 
>automatically.
>   Also ISO's seem to miss a hot-plug feature (f.e. for inserting driver CD's 
> after installation.

I've just tried adding a bootable CD to bhyve as a second device and yes, it 
does appear to try and boot from it even though the hdd is bootable.
I'm sure I've seen a UEFI cli at some point but I don't know if there's any way 
to configure boot order (devs?) Obviously hot-plug CD would be nice and I 
expect is a feature that would come eventually once the basics are finished.

>Some additional questions:
>- Can one over-provisioning/ballooning guest memory's ?
>- Is it (speed-wise) better to use ZFS-zvol's or files in regular 
>ZFS-directories?
>- Are the virtio-blk or ahci-hd disks having the same overhead?
>- Can ahci-hd be used paravirtualized in Centos?

These are probably all questions for the devs...

Matt

>Thanks in advance!

>All in all it looks quite promising!

>Greetings
>---
Mike
>
>Gruß
>---
>Michael Reifenberger

>___
>freebsd-virtualization@freebsd.org mailing list 
>https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>To unsubscribe, send any mail to 
>"freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Understanding Bhyve shutdown

[Matt Churchyard via freebsd-virtualization]

As I understand it

1) shutdown from guest
2) 'kill ' -> pressing the power button once.
3) --force-poweroff -> holding power button in
4) --force-reset -> pressing the reset button
5) bhyvectl --destroy -> same as 3? (although vmm is destroyed as well)

1 or 2 are obviously preferred. I've found however that some guests don't 
respond that well to the apci event. CentOS 6 seems to need apci installing and 
enabling?!, and my Win2012R2 machine will only shutdown if I send the signal 
twice.

The rest are all hard shutdowns that should not be seen as a way to stop the 
guest, just a last resort if it can't be stopped correctly. I don't know the 
practical difference between poweroff vs just destroy.

I see no reason why the documentation suggests reboot rather than shutdown. 
Bhyve exits either way and the only difference is the exit code. When using one 
of the bhyve management tools that support reboots (such as 
vmrun.sh/vm-bhyve/iohyve) however, a "restart" exit code (0) will cause the 
guest to restart automatically; If a guest is locked up for example, a 
--force-reset will cause it to immediately reboot.

>From a host, the only safe choice is 'kill ' (possibly multiple times) 
>and wait for bhyve process to (hopefully) exit. If that doesn't work either 
>the acpi issue needs fixing or ideally the guest needs to be stopped using its 
>built-in shutdown function.

The devs will have to confirm why they're implemented the way they are. My 
instinct is that bhyvectl works on the vmm device, and reset/poweroff are just 
functions that are possible via that interface, and so have been exposed. The 
ACPI shutdown event likely needs to be initiated by the bhyve process itself, 
hence using a signal to it. /end-speculation

I think most users will want to use a bhyve tool so the raw specifics of the 
bhyve/bhyvectl commands are glossed over, although that doesn't mean the 
handbook documentation of the base commands shouldn't be as complete/correct as 
possible of course.

Matt

-Original Message-
From: owner-freebsd-virtualizat...@freebsd.org 
[mailto:owner-freebsd-virtualizat...@freebsd.org] On Behalf Of Roman Bogorodskiy
Sent: 13 April 2016 11:55
To: freebsd-virtualization@FreeBSD.org
Subject: Understanding Bhyve shutdown

Hi,

I was trying to get better understanding of how to properly shutdown VMs in 
bhyve, but unfortunately the documentation does not provide much details on 
that.

Specifically, handbook [I] suggests to reboot a machine and then run bhyvectl 
--destroy on it. 

The bhyvectl(8) manpage mentions the '--force-reset' and '--force-poweroff' 
switches, but does not give details on those. 

I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html

I tried all the options I know and wrote down the results. I also have some 
questions, hopefully you'll be able to answer some of them.

1. bhyvectl --vm=$name --destroy

 * looks like hard poweroff in the guest
 * the corresponding bhyve(8) process goes away
 * /dev/vmm/ entry goes away

In my experience, it's a dangerous way to shutdown a VM because sometimes it 
appears it damages the image and VM fails to boot with something like this:

---
Starting devd.
mode = 0100600, inum = 170269, fs = /
panic: ffs_valloc: dup alloc
cpuid = 0
KDB: stack backtrace:
#0 0x80984e30 at kdb_backtrace+0x60
#1 0x809489e6 at vpanic+0x126
#2 0x809488b3 at panic+0x43
#3 0x80b74a6e at ffs_valloc+0x84e
#4 0x80bb60ad at ufs_makeinode+0x7d
#5 0x80bb24fd at ufs_create+0x2d
#6 0x80e71841 at VOP_CREATE_APV+0xa1
#7 0x809cd9e6 at uipc_bindat+0x346
#8 0x809c5488 at kern_bindat+0x108
#9 0x809c52a7 at sys_bind+0x77
#10 0x80d4b3f7 at amd64_syscall+0x357
#11 0x80d30adb at Xfast_syscall+0xfb
Uptime: 3s

Dump failed. Partition too small.
---

2. kill -SIGTERM $bhyve_pid

If guest supports ACPI shutdown:

 * guest shuts down cleanly
 * the corresponding bhyve(8) process terminates
 * /dev/vmm entry is still here, need bhyvectl --destroy for complete
   cleanup

If guest does not support ACPI shutdown (such as doing sysctl
hw.acpi.power_button_state=NONE):

 * Nothing happens

 Q1: Is there a way to know if a guest reacted to power button but
 waiting for the bhyve process to terminate?   
 Q2: Why it's not done via bhyvectl (it seems that it's easier for users
 + don't have to overload a useful SIGTERM signal)

3. bhyvectl --vm=$name --force-poweroff

 * looks like hard poweroff in the guest
 * the corresponding bhyve(8) process goes away
 * /dev/vmm entry is still here, need bhyvectl --destroy for complete
   cleanup

Q: what's the practical difference with just doing --destroy right away?

4. bhyvectl --vm=$name --force-reset

Looks very similar to item #3 with just different exit code (reboot
appears to be using 0, while shutdown and halt use 1 and 2).   

Q: what's the practical use of it?

Would greatly appreciate if somebody could provide more details 

vm-bhyve port upgrade

[Matt Churchyard via freebsd-virtualization]

Hello,

For anyone interested I have submitted a PR to update the version of vm-bhyve 
in the ports tree.
Primarily this fixes the off-putting, but completely benign error printed when 
users run 'vm init' (the very first thing to run...)
I've no idea if I've got the diff format right though.

Also adds various small fixes, and the following changes since the last ports 
version:

Command to rename a guest
Configuration options for utctime, hostbridge, disk options, debug mode, custom 
grub commands, virtual random device
Snapshot and rollback commands when using ZFS
Allows use of custom bridges and/or tap devices in addition to the normal 
automated networking
Ability to specify a custom path for disk devices
Guests can now automatically attach correctly to virtual switches if the real 
interface(s) (and thus the bridge) are using jumbo frames
Template options to specify zfs dataset/zvol properties to apply when creating 
a guest (most useful for zvol volblocksize)
New 'info' commands showing detailed guest/switch details including disk & 
network usage
No longer replaces dnsmasq.conf, just provides a sample config for the user to 
apply if they want dhcp on a nat-enabled virtual switch.

Matt (churchers)
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

bhyve uefi error please help me!

[Matt Churchyard via freebsd-virtualization]

> Hello!
> not workink bhyve and windows please help me:

> bhyve
>  -c 2
>  -s 0,hostbridge
>  -s 3,ahci-hd,/images/win.img
>  -s 4,ahci-cd,/images/win_repack.iso
>  -s 10,virtio-net,tap0
>  -s 31,lpc
>  -l com1,/dev/nmdm0A
>  -l com2,/dev/nmdm1A
>  -l bootrom,/path/to/BHYVE_UEFI.fd
>  -m 2G -H -w
>  windows

> bhyve: invalid lpc device configuration 'bootrom,BHYVE_UEFI_20151002.fd'

> https://people.freebsd.org/~grehan/bhyve_uefi/windows_install.txt

Is this on FreeBSD CURRENT (FreeBSD-11)?
Other people on this list will know for certain but that error suggests to me 
that your version of bhyve might not have bootrom support.

Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Illumos boot

[Matt Churchyard via freebsd-virtualization]

> Hi,

>> ..
> /pci@0,0/pci8086,2821@4/disk@0,0 (sd1) online
> NOTICE: vioif0: Got MAC address from host: e4:94:1:0:ff:ff
> pseudo-device: tun0
> tun0 is /pseudo/tun@0
> pseudo-device: lx_systrace0
> lx_systrace0 is /pseudo/lx_systrace@0
> 
> panic[cpu0]/thread=ff0002566c40: BAD TRAP: type=d (#gp General 
> protection) rp=ff00025664c0 addr=20
> 
> sched: #gp General protection
> addr=0x20
> pid=0, pc=0xf80d375a, sp=0xff00025665b0, eflags=0x10282
> cr0: 8005003b cr4: 
> 406b8
> cr2: fed3b5accr3: 1dc0cr8: c
> 
>rdi: 7f1a90ff00c3 rsi: ff00c33321f8 rdx: ff00c37f5828
>rcx: ff00c3868603  r8: ff00ca7aa600  r9: 2ba6
>rax:0 rbx: ff00c30d0ef0 rbp: ff00025665c0
>r10: fb8554c4 r11:1 r12:   1f
>r13: ff00c319f880 r14:   10 r15:   20
>fsb:0 gsb: fbc326a0  ds:   4b
> es:   4b  fs:0  gs:  1c3
>trp:d err:0 rip: f80d375a
> cs:   30 rfl:10282 rsp: ff00025665b0
> ss:   38
> 
> This is the log of the bhyve options used (apart from 1 cpu, 1G ram)
> 
> Oct 13 14:22:43:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 
> 4:0,ahci-hd,/data/vm/smartos/disk0.img -s 7:0,virtio-net,tap1] Oct 13 
> 14:22:43:  [bhyve console: -l com1,/dev/nmdm1A -l com2,/dev/nmdm2A] 
> Oct 13 14:22:43:  [bhyve options: -Hw -l 
> bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd]
> Oct 13 14:22:43:  [bhyve iso device: -s 
> 3:0,ahci-cd,/data/vm/.iso/smartos-latest.iso]

> Ouch, even with the additional verbosity, the output isn't very insightful.  
> All I can glean is that you are reasonably far along in the boot > process.  
> You could try to run with KMDB (-k boot option) and do some disassembly 
> around that program counter to see if any specific module is > implicated.

> Alternately, I'd omit the network device and see how far that gets you.

It does appear that I can get as far as a login console without the virtio-net 
device

Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Illumos boot

[Matt Churchyard via freebsd-virtualization]

>> -s 0,hostbridge

> Things should work if you leave out the hostbridge. The PCIe
> capability that is tacked on to this will make Illumos use MSI/MSIx for
> the virtio adapter which apparently hits a bug in the driver. Without
> it, the virtio driver will fall back to legacy interrupts. This also
> means that the virtio adapter will be confined to slots 3/4/5/6.

> [root@smartos ~]# echo ::interrupts | mdb -k
> IRQ  Vect IPL BusTrg Type   CPU Share APIC/INT# ISR(s)
> 10x40 5   ISAEdg Fixed  0   1 0x0/0x1   i8042_intr
> 30xb1 12  ISAEdg Fixed  0   1 0x0/0x3   asyintr
> 40xb0 12  ISAEdg Fixed  0   1 0x0/0x4   asyintr
> 90x81 9   PCILvl Fixed  1   1 0x0/0x9   acpi_wrapper_isr
> 12   0x41 5   ISAEdg Fixed  1   1 0x0/0xc   i8042_intr
> 16   0x42 5   PCILvl Fixed  1   1 0x0/0x10  ahci_intr
> 17   0x43 5   PCILvl Fixed  0   1 0x0/0x11  ahci_intr
> 18   0x60 6   PCILvl Fixed  1   1 0x0/0x12  virtio_intx_dispatch
> 160  0xa0 0  Edg IPIall 0 - poke_cpu
> 208  0xd0 14 Edg IPIall 1 - kcpc_hw_overflow_intr
> 209  0xd1 14 Edg IPIall 1 - cbe_fire
> 210  0xd3 14 Edg IPIall 1 - cbe_fire
> 240  0xe0 15 Edg IPIall 1 - xc_serv
> 241  0xe1 15 Edg IPIall 1 - apic_error_intr


>   (I believe Andriy Gapon (cc'd) has a fix for this in Illumos)

> later,

> Peter.

Thanks Peter. I can't believe how many times I checked you Illumos instructions 
making sure there were no obvious differences, and didn't notice the hostbridge 
was missing. I think I just had it in my mind that the hostbridge was a basic 
required device.

Regards
Matt
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

RE: Illumos boot

[Matt Churchyard via freebsd-virtualization]

>Hi,

>Please see inline.

>On Oct 13, 2015, at 7:17 AM, Matt Churchyard via freebsd-virtualization 
><freebsd-virtualization@freebsd.org> wrote:

> In my quest to continue expanding guest support in my vm-bhyve utility (See 
> https://github.com/churchers/vm-bhyve :) ), I've found the Windows support 
> pretty solid once I got clear on the slot requirements. I'm now trying an OS 
> that requires CSM (Illumos) but unfortunately I'm currently struggling to get 
> it to boot up correctly.
> 
> Here's an example of the command I'm generating at the moment (This is 
> running on an Intel Core-i3):
> 
> bhyve -c 2 -m 2G -s 0,hostbridge -s 31,lpc \
>  -s 3,ahci-cd,/data/vm/.iso/smartos-latest.iso \
>  -s 4:0,ahci-hd,/data/vm/smartos/disk0.img \
>  -s 5:0,virtio-net,tap0 \
>  -l com1,stdio -l com2,/dev/nmdm2A \
>  -H -l bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd \
>  smartos
> 
> I have com1 set to stdio so I can easily watch the output as it runs.
> It tends to get as far as "Legacy INT19 Boot...", then fall over.
> Depending on whether I put the network interface directly in the slot after 
> the HDD, I seem to get different errors -
> 
> slot 3 - cd
> slot 4 - hdd
> slot 5 - virtio-net
> 
> panic[cpu0]/thread=ff01457cdb40: BAD TRAP: type=e (#pf Page fault) 
> rp=ff0004a69a60 addr=40 occurred in module "genunix" due to a NULL 
> pointer dereference
> 
> slot 3 - cd
> slot 4 - hdd
> slot 7 - virtio-net
> 
> panic[cpu1]/thread=ff0004002c40: BAD TRAP: type=d (#gp General 
> protection) rp=ff0004002740 addr=0
> 
> On com2 I see the boot menu, then one and a half lines of dots. The second 
> line of dots stops about 2/3 of the way across.

>Have you tried booting illumos in verbose mode - edit the grub command line 
>and provide '-v'.  That may give you a better backtrace than a >program 
>counter.

This is what I get from a verbose boot:

Bhyve-HandleProtocol: Copying DevPath: 
PciRoot(0x0)/Pci(0x3,0x0)/Sata(0x0,0x0,0x0) [32]
Legacy INT19 Boot...
cpu0: x86 (chipid 0x0 GenuineIntel 206A7 family 6 model 42 step 7 clock 3109 
MHz)
cpu0: Intel(r) Core(tm) i3-2100 CPU @ 3.10GHz
pseudo-device: stmf_sbd0
stmf_sbd0 is /pseudo/stmf_sbd@0
pseudo-device: lofi0
lofi0 is /pseudo/lofi@0
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
acpinex0 at root
acpinex0 is /fw
iscsi0 at root
iscsi0 is /iscsi
xsvc0 at root: space 0 offset 0
xsvc0 is /xsvc@0,0
acpinex: sb@0, acpinex1
acpinex1 is /fw/sb@0
pseudo-device: pseudo1
pseudo1 is /pseudo/zconsnex@1
pseudo-device: pseudo2
pseudo2 is /pseudo/zfdnex@2
/pci@0,0/pci8086,2821@3 :
SATA CD/DVD (ATAPI) device at port 0
model BHYVE SATA DVD ROM
firmware 001
serial number BHYVE-EA14-A68A-54FA
supported features:
 DMA
SATA Gen3 signaling speed (6.0Gbps)
pseudo-device: llc10
llc10 is /pseudo/llc1@0
pseudo-device: power0
power0 is /pseudo/power@0
pseudo-device: ramdisk1024
ramdisk1024 is /pseudo/ramdisk@1024
pseudo-device: ucode0
ucode0 is /pseudo/ucode@0
pseudo-device: zfs0
zfs0 is /pseudo/zfs@0
pseudo-device: srn0
srn0 is /pseudo/srn@0
pseudo-device: dtrace0
dtrace0 is /pseudo/dtrace@0
pseudo-device: dcpc0
dcpc0 is /pseudo/dcpc@0
pseudo-device: fasttrap0
fasttrap0 is /pseudo/fasttrap@0
pseudo-device: fbt0
fbt0 is /pseudo/fbt@0
pseudo-device: profile0
profile0 is /pseudo/profile@0
pseudo-device: lockstat0
lockstat0 is /pseudo/lockstat@0
pseudo-device: sdt0
sdt0 is /pseudo/sdt@0
pseudo-device: systrace0
systrace0 is /pseudo/systrace@0
pseudo-device: ipd0
ipd0 is /pseudo/ipd@0
pseudo-device: stmf0
stmf0 is /pseudo/stmf@0
sd0 at ahci0: target 0 lun 0
sd0 is /pci@0,0/pci8086,2821@3/cdrom@0,0
pseudo-device: fssnap0
fssnap0 is /pseudo/fssnap@0
/pci@0,0/pci8086,2821@3/cdrom@0,0 (sd0) online
/pci@0,0/pci8086,2821@4 :
SATA disk device at port 0
model BHYVE SATA DISK
firmware 001
serial number BHYVE-3083-1AF1-1754
supported features:
 48-bit LBA, DMA, Native Command Queueing
SATA Gen3 signaling speed (6.0Gbps)
Supported queue depth 32
capacity = 62914560 sectors
WARNING: kvm: no hardware support

pseudo-device: pool0
pool0 is /pseudo/pool@0
pseudo-device: bpf0
bpf0 is /pseudo/bpf@0
sd1 at ahci1: target 0 lun 0
sd1 is /pci@0,0/pci8086,2821@4/disk@0,0
pseudo-device: pm0
pm0 is /pseudo/pm@0
pseudo-device: nsmb0
nsmb0 is /pseudo/nsmb@0
pseudo-device: tap0
tap0 is /pseudo/tap@0
/pci@0,0/pci8086,2821@4/disk@0,0 (sd1) online
NOTICE: vioif0: Got MAC address from host: e4:94:1:0:ff:ff
pseudo-device: tun0
tun0 is /pseudo/tun@0
pseudo-device: lx_systrace0
lx_systrace0 is /pseudo/lx_systrace@0

panic[cpu0]/thread=ff0002566c40: BAD TRAP: type=d (#gp General protection) 
rp=ff00025664c0 addr=20

sched: #gp General protection
addr=0x20
pid=0, pc=0xf80d37

RE: Options for zfs inside a VM backed by zfs on the host

[Matt Churchyard via freebsd-virtualization]

 On Wed, Aug 26, 2015 at 11:10:44PM -0700, Marcus Reid wrote:
 On Wed, Aug 26, 2015 at 05:25:52PM -0400, Vick Khera wrote:
   Opinions? Preferably well-reasoned ones. :)
   
  However, having the ARC eating up lots of memory twice seems pretty 
  bletcherous.  You can probably do some tuning to reduce that, but I 
  never liked tuning the ARC much.

 I just realized that you can turn primarycache off per-dataset.  Does it make 
 more sense to turn primarycache=none on the zvol on the host, or  on the 
 datasets in the vm?  I'm thinking on the host, but it might be worth 
 experimenting.

I'd be very wary of disabling ARC on the main host, it can have pretty serious 
side effects. It could possibly be useful in the guest though, as data should 
be cached already by ARC on the host, you're just going through an extra step 
of reading through the virtual disk driver, and into host ARC, instead of 
directly from the guest memory. Would need testing to know what performance was 
like and if there are any side effects.

I do agree that it doesn't seem unnecessary to have any redundancy in the guest 
if the host pool is redundant. Save for any glaring bugs in the virtual disk 
emulation, you wouldn't expect to get errors on the guest pool if the host pool 
is already checksumming the data.

It's also worth testing with guest ARC enabled but just limited to a fairly 
small size, so you're not disabling it entirely, but doing at little 
double-caching as possible.

ZFS features seems perfect for virtual hosts, although it's not ideal that you 
have to give up a big chunk of host RAM for ARC. You may also find that you 
need to limit host ARC, then only use MAX_RAM - MY_ARC_LIMIT for guests. 
Otherwise you'll have ZFS and VMs fighting for memory and enough of us have 
seen what shouldn't, but unfortunately does happen in that situation.

Matt
-

 Marcus
 ___
 freebsd-virtualization@freebsd.org mailing list 
 https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org