Re: bhyve current windows status

[Peter Grehan]

Hi Matt,

I'm after some general information on the current status/best practises 
for Windows on bhyve. Not entirely the correct place for this but then 
at the moment no-one else seems to really know the answers. Maybe I can 
help some of the other people who are just as unclear as me on what is 
actually the best information at this point.


 This is as good a place as any.

What are the current recommended devices/options for Windows (2019 
server in my case) - especially with ZFS. Should I be specifying a 
512/4096 sector/block size via bhyve and/or zfs? I assume nvme & 
virtio-net are the current best options but is there a preferred virtio 
driver version. Are any of the other virtio drivers of any use to be 
installed or just the network drivers?


 nvme - yes.

 I'll leave the sector/block size issues to others. I don't touch any 
of those params but don't use enough Windows apps to make a qualified call.


 No need for other virtio drivers. For virtio-net, the recommendation 
is to use the latest one.


Are there any known problems with applications like AD/Exchange? I know 
that SQL 2012 had massive storage overhead issues on ZFS due to 512 byte 
writes, but I'm not sure if that still affects newer versions or other 
applications?


 As above, I'll leave it up to others to chime in here.

The system I am currently using is a Xeon E5-2670, which I know was 
terrible before the TPR commit. My test system seems to run reasonably 
on 12.2 (although I'd be intruiged to compare against ESXi if I had the 
time), but do you think I would expect to see any significant gains by 
using a CPU with APICv? (not that I expect anyone has done any 
benchmarking of this)


 It's been a long while since I've benchmarked APICv, and have never 
benched it on Windows, but my expectation is it won't make a lot of 
difference unless you have a very i/o intensive workload.


Are there any other changes in being worked on that are likely to have 
an impact on support or performance?


 No. The main focus for Windows guests right now is GPU passthru.

I believe quite a bit of work is 
being done on the UEFI firmware but I expect that doesn't really affect 
much other than the boot process. I'm sure I saw reference to the devs 
having regular bhyve calls, but I have little idea what is currently 
being worked on.


 You can always ask here. For interactive response, there's the bhyve 
office hours which you are most welcome to participate in:

   https://wiki.freebsd.org/bhyve/OfficeHours

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: regression in virtio-9p option parsing

[Peter Grehan]

Hi Roman,


diff --git a/usr.sbin/bhyve/pci_virtio_9p.c b/usr.sbin/bhyve/pci_virtio_9p.c
index e27159eb22cb..830e13878a71 100644
--- a/usr.sbin/bhyve/pci_virtio_9p.c
+++ b/usr.sbin/bhyve/pci_virtio_9p.c
@@ -232,7 +232,7 @@ pci_vt9p_notify(void *vsc, struct vqueue_info *vq)
  static int
  pci_vt9p_legacy_config(nvlist_t *nvl, const char *opts)
  {
-   char *sharename, *tofree, *token, *tokens;
+   char *sharename = NULL, *tofree, *token, *tokens;

 if (opts == NULL)
 return (0);


 This looks fine to commit.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: FreeBSD13/12 oddity

[Peter Grehan]

I was testing AlmaLinux on bhyve. > > works on FreeBSD 13> > but is crashes on 
12 upon install.

 I tried on CURRENT and it also works (with UEFI).

 Were you using UEFI on 12 ? Same options as on 13 ?

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Extra dependency of the uefi-edk2-bhyve-csm firmware port

[Peter Grehan]

dependency on gcc48 ?

PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254257


 Being fixed by bcran in https://reviews.freebsd.org/D29254

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: About nested virtualization for bhyve

[Peter Grehan]

Hi,


Does bhyve support the following schemes?

Scheme A:
Layer 0 - bhyve on FreeBSD Host
Layer 1 - FreeBSD guest
Layer 2 - bhyve guest

Scheme B:
Layer 0 - bhyve on FreeBSD Host
Layer 1 - Windows guest
Layer 2 - Hyper-V guest

 ...

What about on a bhyve host? For example, any configruation in
/boot/loader.conf or /etc/sysctl.conf to enable it?


 Those configurations aren't yet supported: bhyve will have to emulate 
VT-x/SVM functionality.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: What happened to hw.vmm.topology.cores_per_package?

[Peter Grehan]

Hi Patrick,


What happened to those tunables?


 They've been rolled into a per-VM parameter. From the man page:

   [-c [[cpus=]numcpus][,sockets=n][,cores=n][,threads=n]]

 ... and there's more description of the '-c' parameter there.

https://www.freebsd.org/cgi/man.cgi?query=bhyve=FreeBSD+12.2-RELEASE+and+Ports

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve on 14-CURRENT crashes trying to boot Windows 10 20H2 DVD

[Peter Grehan]

I've been testing some UEFI firmware changes, but even with the 0.2  > (2014.SP1) port version, I can't boot from the Windows 10 20H2 DVD > 

without causing bhyve to crash. The console output is:

The command line I'm using is:

bhyve -AH -s 0:0,hostbridge -s 31:0,lpc -c 1 -m 8G -l 


 No need for the 'A' option: ACPI tables are generated from UEFI. 
However, that should be harmless.


bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -l com1,stdio -s 
3:0,ahci-hd,bhyve-windows-10-20h2.img -s 
29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait -s 
7,ahci-cd,en_windows_10_business_editions_version_20h2_updated_jan_2021_x64_dvd_533a330d.iso 
guest


 You likely need a '-w' option to ignore unimplemented MSRs.

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: UEFI -l com1, stdio floods console/Windows Server SAC with "rdmsr to register 0x198 on vcpu..."

[Peter Grehan]

Can the "rdmsr to register 0x198 on vcpu..." and similar messages on a 
-l com1,stdio console when using UEFI firmware be redirected or 
suppressed? The -w flag does not silence them, assuming, perhaps 
incorrectly, they are unsupported MSRs.


If redirected, should these be logged?


 Please file a PR (https://www.freebsd.org/support/bugreports.html)

 A simple workaround is to use an nmdm or pty device for com1 output.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [bhyve][arm64] Number of allowed vcpus

[Peter Grehan]

Hi Andrei,


I've summed up your proposals and sketched and algorithm:
N - the number of vcpus from the '-c' parameter
X - the number of vcpus from the device tree

if there isn't the parameter -> start the vm with X vcpus
if there is the parameter --> if N <= X -> start the vm with N parameters
|-> if N > X -> throw an error

I believe this gives the opportunity to rebuild the device tree and use all
vcpus (first branch)
or let the device tree as it is (more vcpus) but use only a subset of them.
Is this alright?


 If you are going to have the ability to rebuild the device tree, you 
might as well do it every time.


 If a user hasn't specified how many vCPUs on the command line, it's 
safe to assume a default of 1.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [bhyve][arm64] Number of allowed vcpus

[Peter Grehan]

Hi Andrei,


While working on enabling multiple virtual cpus for the arm64 virtual
machine
I used the bhyve's "-c N" parameter to create N cpus (it gets only the
cpus, in contrast to the amd64). If N is smaller than the number of cores
described in the device tree it will enable only N and if it's larger it
will enable only the number of cores from the device tree.

I don't know if it is the best approach. Another way would be to enable all
the cpus from the device tree, no more, no less.

Do you have any suggestions?


 My suggestion to generate the device tree based on VM configuration. 
bhyve on amd64 already does this for ACPI tables.


 (I do understand that this is more difficult for bhyveload on ARM64 
since the device tree has to be known at load time, and bhyveload is 
only given a subset of parameters: however it may be possible to reserve 
space at a fixed location and then have bhyve copy a generated table 
into that region).


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve office hours

[Peter Grehan]

  John Baldwin and myself will be doing a FreeBSD "office hours" 
streaming this Wednesday, Dec 16, at 1PM PST (21:00 UTC).


  All are welcome. There will be some reports on recent work, though 
much of the session will be an interactive Q


 Meeting notes and pdfs for the short talks up at


https://wiki.freebsd.org/action/fullsearch/bhyve/OfficeHours/16December2020

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Bhyve office hours

[Peter Grehan]

Hi all,

 John Baldwin and myself will be doing a FreeBSD "office hours" 
streaming this Wednesday, Dec 16, at 1PM PST (21:00 UTC).


 All are welcome. There will be some reports on recent work, though 
much of the session will be an interactive Q


 Hope to see you there.

 https://live.freebsd.org/FreeBSD/officehours/

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [arm64][bhyve] Hypervisor implementation proposal

[Peter Grehan]

Hi Andrei,
I have updated the revision by removing the armv7 support and host 
platform support.


 Thanks for the update: that's going to make the review a lot easier.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [arm64][bhyve] Hypervisor implementation proposal

[Peter Grehan]

Hi Andrei,


The UPB team made a revision for the bhyve on arm/arm64 on phabricator[1].

Any feedback is greatly appreciated.

[1] https://reviews.freebsd.org/D26976 


 Thanks for putting up the review: that is a sizeable amount of work.

 I'd suggest removing 32-bit host support from the review to reduce the 
size. Note that Linux/KVM has dropped 32-bit host support (see the 
thread at 
https://patchwork.kernel.org/project/kvm/cover/20200210141324.21090-1-...@kernel.org/), 
and the same reasons would apply to FreeBSD.

   If there is interest it could be resurrected at a later time.

 The MI/MD split could be done independently of this. It has been 
talked about for a long while, so I'll commit to doing this in the 
existing amd64 codebase. The ARM work could then be rebased into that.


 Also, any host platform support, such as the ARM simulator or 
EspressoBin, could be split out since it's independent of the hypervisor 
portion.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: HD audio problem on FreeBSD 12.1 bhyve VM

[Peter Grehan]

Hi Ali,


It doesn't work for me... See below.

...

I'm booting with the following script.

VM=freebsd12.1
ifconfig tap1 up
bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \
-s 2:0,virtio-net,tap1 -s 3:0,virtio-blk,./disk.img \
-c 1 -m 4096M \
-s 19,xhci,tablet \
-s 11,fbuf,tcp=0.0.0.0:5920,w=1280,h=1024,wait \
-s 20,hda,play=/dev/dsp0,rec=/dev/dsp0 \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
$VM
bhyvectl --destroy --vm=$VM


 Does removing the 'rec=/dev/dsp0' make any difference ?


Audio on the Thinkpad host is:

grehan@carbon-usb:/images $ cat /dev/sndstat
Installed devices:
pcm0:  (play/rec) default
pcm1:  (rec)
pcm2:  (play)



I have the issue on my daily driver, a Thinkpad T430. But also tested on
my x220, same issue, only noise. However, sound works perfectly fine
on OpenSUSE guest.


 What does 'cat /dev/sndstat' show on your 430/x220 ?

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve: vnc limited to 1920x1080?

[Peter Grehan]

I used the following option for bhyve command: > > -s 29,fbuf,rfb=192.168.1.6:8001,wait,w=2560,h=1440> > ...and got 
rather confusing error message:> > Invalid fbuf emulation option "w"> 
fbuf: {wait,}{vga=on|io|off,}rfb=:port{,w=width}{,h=height}> device 
emulation initialization error: Inappropriate ioctl for device> > Had to 
check the man page/source to find the hardcoded 1920x1080 > limitation 
-- is there any technical reason for it, or is it just "ought > to be 
enough for anybody"?
 The framebuffer is 16MB, and the max of 1920x1200 was chosen as a 
compromise between a common resolution that would fit in that, the 
amount of CPU used in the tile-based motion detection, and the amount of 
bandwidth required to transmit full-screen's worth of data at 30 fps.


 The original intent was the VNC console would only be used for setup, 
and after that a more efficient remote desktop would be set up on the guest.


 (More details at 
https://people.freebsd.org/~grehan/talks/bhyve_bsdtw_bhyve_graphics.pdf)


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: HD audio problem on FreeBSD 12.1 bhyve VM

[Peter Grehan]

Hi Ali,


I tested the following combinations:

1) 12.1 guest on 12.1 host (with the HD audio from 13-current).
2) Same 12.1 guest on 13-current host.


 2) worked ok for me, at least with very limited testing and a 
simplistic config


On the guest,

root@:~ # uname -a
FreeBSD  12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

root@:~ # cat /dev/sndstat
Installed devices:
pcm0:  (play) default
No devices installed from userspace.

root@:~ # sysctl dev.hdac.0.pindump=1
dev.hdac.0.pindump: 0hdaa0: Dumping AFG pins:
hdaa0: nid   0xas seq device   conn  jackloccolor   misc
hdaa0:  3 1010 1  0  Line-out  Jack  Unknown 0x00   Black   0
hdaa0: Caps:OUT  Sense: 0x8000 (connected)
hdaa0: NumGPIO=0 NumGPO=0 NumGPI=0 GPIWake=0 GPIUnsol=0

cat test.wav > /dev/dsp worked the same in the guest as on the host

The guest was configured with "-s 20,hda,play=/dev/dsp"

Audio on the Thinkpad host is:

grehan@carbon-usb:/images $ cat /dev/sndstat
Installed devices:
pcm0:  (play/rec) default
pcm1:  (rec)
pcm2:  (play)

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: HD audio problem on FreeBSD 12.1 bhyve VM

[Peter Grehan]

Hi Ali,


However, on FreeBSD 12.1 only noise comes out. Using /dev/dsp4 (which is a USB 
sound
device), audio works fine, but not on /dev/dsp0, the default sound device
on my system.

...

Note that: The above problem is also reproducible on 13-Current 20200917.


 Just to confirm: that was a 12.1 guest on a 13-current host ? (or a 
13-current guest ?)


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: grub-bhyve device map file usage

[Peter Grehan]

Hi Chuck,


I'm working on an update to grub-bhyve and wanted to know if people's
map files differ from:
(hd0) /some/path/to/the/disk.img
Primarily, I'm interested if the map files use a device other than
'hd', but I'd also be curious about use cases involving more than one
entry. TIA!


 I use 'cd' as well as 'hd', and often have files with multiple entries 
for those e.g. "debian.map" that has a cd/hd for each version I was 
testing. The -r parameter was used to pick which one would be the 
bootable filesystem image.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Nested Virtualization Under KVM Hypervisor

[Peter Grehan]

Hi Pete,

Ah I see - once I realized we had a newer kernel I didn't focus too much 
on the first bug - here's the MSR's



MSR 0x480: 0x00981000 0x11e57ed0
MSR 0x482: 0xfff9fffe 0x0401e172
MSR 0x48b: 0x03fb 0x
MSR 0x48c: 0x0f01 0x04114040


 This looks very similar to comment 14 in the bug report
 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203994#c14

 INS/OUTS isn't marked as supported, and single-context INVEPT is 
missing. As mentioned in comment 17, these are simple to work around but 
other issues started to show up, indicating that other problems are at play.


 Given kernel 4.15 looks like it could be from Ubuntu 16-LTS, I'd 
recommend moving to 18-LTS if that is possible, since that kernel 
version is known to work.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Nested Virtualization Under KVM Hypervisor

[Peter Grehan]

Hi Pete,

Sep 10 03:25:01 bells kernel: vmx_init: processor does not support 
desired basic capabilities


 ^^^  This is what's causing the issues, and is covered in the first 
bug report I mentioned 
(https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203994)


 Would you be able to dump the MSRs for the FreeBSD guest as mentioned 
in the bug report ?



I also received this input from the ISP, they are running this kernel:
  4.15.0-47-generic #50~16.04.1-Ubuntu SMP


 I'm now wondering if this is too old a kernel :(


reading through that ticket i set this in loader.conf:
hw.vmm.vmx.use_apic_vid=0

and verified via sysctl:
$ sysctl hw.vmm.vmx.cap.virtual_interrupt_delivery
hw.vmm.vmx.cap.virtual_interrupt_delivery: 0
$

since you mention this is a possible bug with KVM is there some useful 
information i can relay to my ISP (they are a BSD friendly vendor) that 
they can try on their end as a workaround, or perhaps something they 
ping the linux/kvm community about?


 As above, this isn't the issue for this particular case: it manifests 
as a hypervisor guest immediately exiting when started.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Nested Virtualization Under KVM Hypervisor

[Peter Grehan]

Hi Pete,


   VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
 That's just a warning: bhyve will attempt to enable VT-x on module 
load, and will only fail if BIOS has locked the disable.


 Do you know if the vmm module fails to load ? (there will be an error 
in syslog "vmx_init: VMX operation disabled by BIOS")



Hypervisor: Origin = "KVMKVMKVM"
real memory  = 34359738368 (32768 MB)
avail memory = 33365536768 (31819 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: 

They did verify that VT-x pass through *is* enabled as mentioned above 
(they shared with me the config file snippet on their end to verify), 
and they did extensive testing on their end and were not able to get 
FreeBSD to detect VT-x as being enabled.


I pinged them asking for more info on which Linux kernel their 
hypervisors are running to see if I can help debug this on my end.  I 
have two questions:


1) is anyone else doing nested virtualization under KVM hypervisors.  i 
am confident that this is possible, but am asking for confirmation on 
this :)


 Yes: A bug report 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203994

  ... had the resolution that a kernel > 4.10 has to be used, and

 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=246168
  ... requires posted interrupts to be disabled.

2) Are there any other requirements that FreeBSD requires to run Bhyve 
nested under KVM?


 No - nested has been enabled by default with KVM for quite a while now.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve/e1000: Assertion failed: (dtype == ntype), function e82545_transmit with win10 guest

[Peter Grehan]

This appears to be an MP issue: if the number of vCPUs in the win10 
guest is set to 1, I'm unable to trigger this. However, it happens 
almost immediately with vCPUs > 1.


Even with a single vCPU there looks to be some false triggering of the 
transmit process: still debugging that.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve/e1000: Assertion failed: (dtype == ntype), function e82545_transmit with win10 guest

[Peter Grehan]

I'm getting the following periodically when Windows 10 (2004) guest is  > booting:> > Assertion failed: (dtype == ntype), function 
e82545_transmit, file > /usr/src/usr.sbin/bhyve/pci_e82545.c, line 
1139.> Abort trap> > It's especially unhelpful when booting the guest 
first time after > installation as it completely breaks it and requires 
reinstall.

 I was able to reproduce this. Looking at it now.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

rdtscp support (was Re: bhyve guest illegal instruction)

[Peter Grehan]

Hi Chuck,


12              time1 = __builtin_ia32_rdtscp();


 rdtscp shouldn't be used without checking that it's available via 
CPUID first, but as you mentioned the feature is available on the host, 
just hidden from the guest.


This same program works on the FreeBSD 12-stable machine hosting the VM 
as well as another bare-metal Linux host. Poking around in the vmm code, 
I found

                         /*
                          * Hide rdtscp/ia32_tsc_aux until we know how
                          * to deal with them.
                          */
                         regs[3] &= ~AMDID_RDTSCP;
                         break;
in sys/amd64/vmm/x86.c which I _think_ is relevant because lscpu doesn't 
show the rdtscp flag. If this is the root cause, what would need to be 
done to implement this?


 At a quick glance, if the feature is available on the host you'd need to
  - expose it via CPUID
  - save/restore the TSC_AUX MSR, but using the VMCS MSR h/w 
save/restore mechanism that will have to be resurrected. (this avoids 
any preemption issues,even at NMI level).

  - set the "enable RDTSCP" VM-execution control to one in the VMCS

 That being said, I've heard anecdotally that rdtscp results in 
VM-exits on other hypervisors so there may be reason to emulate it 
rather than allow a pass-thru. More investigation may be required.


later,

Peter.


___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve, Windows guest and trim support

[Peter Grehan]

Hi Andrea,

Where would I download a fresher driver? Do I need RedHad's 
virtio-win-latest? Is it in there? Or else, what?


 The link in the PR pointed to


https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.185-1/

 (currently an alias of 
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/)


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve, Windows guest and trim support

[Peter Grehan]

Hi Andrea,


Thanks for this hint, I can now confirm this also works with virtio-blk
(applied Allan's patch to 12.1). REFER for the zvol backing my Windows
Server 2016 dropped from 315G to 72.5G.


Hmm...
I applied the same patch and, still using virtio-blk, when trying to 
retrim, Windows says "Optimize-Volume : The volume optimization 
operation requested is not supported by the hardware backing the volume."

Seems like the guest doesn't think TRIM support is available.
Do I have to set anything?

I have the following in the machine config (I'm using vm-bhyve):
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="zvol"
disk0_opts="sectorsize=512"


 You may need a recent Windows viostor driver. A working configuration 
was reported in

https://github.com/virtio-win/kvm-guest-drivers-windows/issues/392

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [Bug 246121] [bhyve][PATCH] Append Keyboard Layout specified option for using VNC.

[Peter Grehan]

Hi Henrik,


When I wrote my Video BIOS code one year ago, I noticed that bhyve is
sending updates continuously from rfb_wr_thr(), without any requests.
On the other hand, the RFB protocol specification says that "an update
is only sent from the server to the client in response to an explicit
request from the client", so bhyve seems to technically break this.


 So do the clients that don't send any UpdateRequest messages at all.

 Spec compliance in the VNC world is a very loose definition :(


I tested my code with TigerVNC, and at one point I ended up with a very
long delay for keyboard events. From my diary at the time (2019-05-02):

     "In any case, I finally found the cause of the long input queue.
  TigerVNC sends a new update request for each display update it
  gets, and it only expects to get one, but bhyve sent a separate
  update for each rectangle, so as soon as display updates started,
  the input buffer filled up with update requests."


 Ok, that's an optimization that could be done: I think there's enough 
info to do that in an update.



By then, I had probably already changed the rfb.c code a lot, but any
delay for TigerVNC could be due to a similar issue. I guess different
VNC clients would react differently to unexpected update messages.


 And the majority do.


Anyway, I wanted to mention it before you rewrite the RFB code again.
My patch is still in limbo at https://www.gulbra.net/freebsd-bhyve/,
and the changes in rfb.c are those that are most likely to conflict
with other modifications. It would be better for me if I don't have to
do the same debugging all over again if I finally return to this code.


 Thanks; I 'll have a look at it.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [Bug 245392] bhyve: PCI passthru built-in rtl8168 to OpenBSD no packets (AMD Ryzen 3 1200)

[Peter Grehan]

Hi Anatoli,


Sorry, didn't get any reply to the details provided. Don't know if it's
insufficient or the issue is so complex that nobody wants to investigate
it further.

Please let me know what's the status and what are the options/directions
to investigate it further.


 Debugging this might need some DTrace on the FreeBSD host side, and 
maybe a custom OpenBSD kernel build to see what's happening there. This 
could be time-consuming :( Get in touch direct if you're ok to go down 
this path.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [Bug 245392] bhyve PCI passthru built-in rtl8168 to OpenBSD no packets (AMD Ryzen 3 1200)

[Peter Grehan]

Hi Anatoli,

I just submitted a bug as to not send here all the attachments.


 Saw that: thanks.


Is this something known? Or maybe I'm missing some configuration option?
Should I submit more details?


 That should be enough detail. I'll follow-up in the bug.
.
later,

Peter
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: grub-bhyve

[Peter Grehan]

Grub-bhyve is using Gcc 4.8.5 to copile with. > > I recall compiling with gcc 9.0 with a few updates needed to the> 

code.  Has anyone else tried this?
 The ports version is using gcc 9 so no issues there.

 The grub2-bhyve github README has just been updated to remove the 
versions from gcc/gdb - the most recent versions of those are fine to use.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: USB device passthrough implementation

[Peter Grehan]

Hi Daniel,

* We need to disconnect the USB device from host: we are currently not very
familiar with the kernel API to perform this


 See usbconfig(8) and the detach_kernel_driver command. It's probably 
fine to request a user to issue that command, similar to how external 
commands are required to detach a PCI driver before using it for 
passthru in bhyve.



* We need to emulate a USB device controller: we are currently looking into
pci_xhci.c and usb_mouse.c to reuse some of the code from there and we
would like to know if there is some sort of in depth documentation
regarding these.


 pci_xhci.c is an implementation of a USB XHCI host controller so 
should be the place to start: it either has the functionality that is 
needed, or requires slight modifications.


 (Lots of info and pointer to the spec at 
https://en.wikipedia.org/wiki/Extensible_Host_Controller_Interface)


 usb_mouse.c is just an example of a USB device interface, that in turn 
passes through VNC mouse-movement events to the guest o/s.



* We need to pass the USB device to the guest through an emulated port of
the device controller: we know that project ACRN implements USB device
passthrough and we are trying to figure out how they do it.


 ACRN uses libusb as the interface to USB devices on the host (see 
devicemodel/hw/platform/usb_pmapper.c).


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Passthru a VF to Windows guest cause BSOD, chelsio T520BT, SR-IOV

[Peter Grehan]

When I pass the same VF1 to Windows 10 guest, ver 1909, the guest got
BSOD during installation. I also try to add VF1 post installation.
also got BSOD.
the message is IRQL_NOT_LESS_OR_EQUAL.
I also try Windows server 2012, during installation of Chelsio Driver, BSOD too.

Any suggestions? driver issue or bhyve issue?


 Can you try disabling APIC acceleration (which should be available on 
that Xeon E5) ? This can be done on the fly using kenv and 
unloading/reloading vmm.ko (or loader.conf and rebooting)


kenv hw.vmm.vmx.use_tpr_shadowing=0
kldunload vmm
kldload vmm

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: [GPU pass-through] no compatible bridge window for claimed BAR

[Peter Grehan]

Hi Alex,


dmesg | grep "no compatible bridge window"

pci :00:01.0: can't claim BAR 1 [mem 0xd0-0xd00fff 64bit
pref]: no compatible bridge window

 ...>  From what I can read from all the info above,

somehow the requested memory space for BAR1 for that device is
0xd0-0xd00fff which is out of addressable space on the system.


 Yep, that's the issue, and it's a bhyve bug - there is no check to see 
if the 64-bit window is within the addressable range of the processor.


 A quick fix is to change the constant for that range in pci_emul.c

#define PCI_EMUL_MEMBASE64  0xD0UL
#define PCI_EMUL_MEMLIMIT64 0xFDUL

  .. to a value that is within the address bits of the CPU, but also 
above guest DRAM.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Patch for Nominal Media Rotation Rate field of AHCI disk.

[Peter Grehan]

Hi Qian,


I have a patch for Nominal Media Rotation Rate field.
currently bhyve will not report this field.


 Nice !

 I was thinking another way to do this would be to change the device 
emulation name for this i.e. "ata-ssd" instead of "ata-hd" with an 
additional parameter.


 That way some additional parameters and fields could be changed, such 
as the name presented in the identify device data.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: System (ps/top) & bhyvectl memory usage differences

[Peter Grehan]

Hi Matt,


Is there a reason why the resident memory used by a bhyve guest is quite different 
when comparing ps/top & bhyvectl?
Does bhyvectl take in account something in kernel space that top/ps doesn't see?

USER   PID %CPU %MEM VSZRSS TT  STAT STARTED  TIME COMMAND
root 12670  0.1  1.4 2120984 951764  1  SC+  22Jan20 157:29.22 bhyve: smtp-a 
(bhyve)

# bhyvectl --get-stats --vm=smtp-a | grep Res
Resident memory 1439875072

1.4G vs 925M


 They could both be wrong, but not by a huge amount :(

 "top" shows the memory used by the bhyve process, which includes usage 
by the hypervisor (which admittedly should be very small). However, the 
guest usage is pages that have been faulted into the address space of 
the bhyve process as a result of accesses by the hypervisor (e.g. 
filling in network/disk buffers for i/o).


 The bhyvectl stats are the pages that have been faulted into the EPT 
paging structures as a result of guest access (code/data access). This 
will generally be more than that reported by top, since the guest will 
be getting execution faults that the bhyve process won't.


 Over time they should slowly converge.

 A fully accurate count would have to walk the bhyve/guest portion of 
the address space and the EPT paging structures, to avoid 
double-counting. There are probably better ways to do it: certainly an 
interesting design topic.



I have a guest with 2G memory allocated, and dmesg lists 2048MB real memory. The 
real & avail figures are also quite close which resembles the output I 
generally expect on real hardware.

Hypervisor: Origin = "bhyve bhyve "
real memory  = 2147483648 (2048 MB)
avail memory = 2043318272 (1948 MB)

However, I have a guest with 5G allocated, and get the following in dmesg -

Hypervisor: Origin = "bhyve bhyve "
real memory  = 6442450944 (6144 MB)
avail memory = 5141663744 (4903 MB)

bhyveload -m 5G ...
bhyve -c 2 -m 5G -AHP ...

I haven't tested where it seems to change. My only theory is that it could 
possibly be something to do with crossing the old 32bit limit?


 bhyve inserts a 1G region just below 4G for 32-bit PCI address space. 
The 'real memory' printed by FreeBSD is just the highest physical 
address (5G guest total is 3G below 4G, plus 2G above, giving 6G as the 
highest phys addr).


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Ubuntu 18.04 and bhyve

[Peter Grehan]

Has anybody managed to run Ubuntu 18.04 under bhyve?

I was unable to get it working with churchers vm-bhyve tool.


 You either need the latest UEFI port, or run it under grub-bhyve.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Centos7 uefi boot problem with bhyve after update

[Peter Grehan]

Hi Mike,


the fault here could be that of bootrom not reading the files it should


 That is exactly the issue. The current UEFI code does not save 
non-volatile variables to persistent storage. Guest o/s's are 
increasingly writing their efi loaders to non-standard locations and 
using nv vars to direct UEFI to boot from these locations.


 I'm (very slowly) merging a fix from Leon for this in both UEFI and 
bhyve which directs nv var writes to a file on the host, providing a 
persistent store.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Greetings from SmartOS

[Peter Grehan]

Hi Patrick,

 Good to see you on the list :)


Now that the project is stabilizing on our end, we would like to reach
out about upstreaming fixes or feature work back into FreeBSD.  One
small fix to posted interrupt handling on Intel
(https://smartos.org/bugview/OS-6829) might be a good starting point
for us to walk through the process of review and upstreaming.


 For that one and changes to the base system, I think you've already 
discovered Phabricator (https://wiki.freebsd.org/Phabricator)


 I should also mention, for bhyve/UEFI, a git pull request against 
https://github.com/freebsd/uefi-edk2/tree/bhyve/UDK2014.SP1 is the best 
way to upstream changes.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve pci passthrough usb controller

[Peter Grehan]

I surely will try this out, but what about device 4/0/0 showing the
same issue. Both tests where performed separately showing the exact
same result, of course I'd be happy if either one of them worked.


 I think what is happening here is that Windows is enabling legacy 
interrupts on the device, and that happens to be on the same shared IRQ 
line that the ATA controller is on, resulting in spurious interrupts.


 The dmesg for the system may be able to confirm this, since it should 
list the legacy IRQs that have been assigned.


 bhyve PCI passthru relies not only on the device supporting MSI/MSI-x 
interrupts, but the guest using those.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issue encountered booting FreeBSD STABLE and CURRENT snapshots with EFI

[Peter Grehan]

Hi Kyle,


FYI- I've created PR #2 [1] against the freebsd/uefi-edk2 repository
and have confirmed that this fixes the broken-looking firmware along
with the booting problem experienced in this thread.

[1] https://github.com/freebsd/uefi-edk2/pull/2


 Thanks - merged. I've asked Fabian to update the UEFI ports.

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issue encountered booting FreeBSD STABLE and CURRENT snapshots with EFI

[Peter Grehan]

I believe the problem may have been introduced with this commit: > > 
https://svnweb.freebsd.org/base/stable/?view=log=329114
 Any chance of being able to work out where in that list of commits in 
CURRENT the loader stopped working ?


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Call for testing bhyve cpu topology additions

[Peter Grehan]

I shall iterate again, this change makes no change to what the guests
sees if you use the old method sysctl hw.vmm.topology.cores_per_package
or hw.vmm.topology.threads_per_core to set these values, it is
purely a interface enhancement that makes these tuneables easy
to access from userland bhyve(8).


 Those sysctls were an undocumented workaround with no error checking.

 You are making this a documented part of bhyve,


A guest can not tell the diffence in what way these are set.
If hw.vmm.topology.* needs testing thats not on me, that is
an existing problem, and one that has existed for far too long.


 Ah, no, the testing is on you, not the user community.


   You can easily download an eval of Windows 10 to try this out with.
You do not (and have never) required ATA support to run Windows on bhyve.


I have made a call for testing, whats your issue?
Are you trying to force me to do that testing?


 At a minimum, you are expected to test changes that you expect to commit.


And I consider this change rather trivial and with near 0 risk,


 You've never made a commit to bhyve but somehow feel qualified to make 
risk assessments.



   And why the rush ? I'm yet to understand what the urgency for this
work is ? Who is demanding it ?


The users have been wanting this for well over a year, it was
a frequently requested item when I wrote it.  It is long overdue.


 Right, so 3 weeks for MFC is perfectly acceptable in that case.


You seem to be raising a bar higher than any other part of
FreeBSD for this rather simple user command enhancement,
can I ask what your objection is?


 The fact that you seem to think testing this is someone else's 
problem, in a subsystem where rigorous testing is a requirement.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Call for testing bhyve cpu topology additions

[Peter Grehan]

   And as mentioned in the review, I'd also like to see your Windows
desktop guest test results with this change.


I do not run any windows in bhyve as bhyve can not run the
windows I use due to missing/broken ATA support.
The person I was helping with bhyve windows regression tests has
become unavaliable.


 I'm staggered by this. The *only reason* for having this change is to 
get past the 1/2 socket limitation when running Windows desktop guests. 
It has no impact on any other guests, whatsoever.


 You can easily download an eval of Windows 10 to try this out with. 
You do not (and have never) required ATA support to run Windows on bhyve.



   Standard MFC time is 3 weeks.


Can you point to this some place?  My understanding is that
MFC is at the discretion of the committer, and the only
thing the big list of rules says:


 This is project culture, which you should have seen observing MFC 
times for commits.


 And why the rush ? I'm yet to understand what the urgency for this 
work is ? Who is demanding it ?


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Call for testing bhyve cpu topology additions

[Peter Grehan]

I would like to ask that if people can test this and provide
feedback that they do so.


 And as mentioned in the review, I'd also like to see your Windows 
desktop guest test results with this change.



If I can get some significant test results back I plan to commit
D9930 to ^head and merge it back to stable/11 3 days later.


 Standard MFC time is 3 weeks.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[grehan (Peter Grehan)]

grehan added a comment.


  So a comment on this: in general, api's are not added to FreeBSD that don't 
have base-system clients that use them, or for a good reason. I think this 
falls into the latter but I'd like to cut it down a bit.
  
  - can the get_unrestricted_guest() routine be removed ? There is an error 
return on the set, which seems like it can be used to determine if unrestricted 
mode is not available (e.g. that's how bhyve uses the ioctl).
  - is there a need for vcpu_reset() ? The BSP should be initialized to 
power-on state.. That could be a bug in bhyve and better to have it fixed there.

REVISION DETAIL
  https://reviews.freebsd.org/D14473

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, rgrimes, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: rumpkernel and bhyve: triple faults

[Peter Grehan]

Hi Fabian,


For a page-fault, the virtual address that resulted in the fault
will

be in the CR2 register.


I don’t see a CR2 register in the output of bhyvectl --get-all, I
was  looking for that too.


 Oops, I'll add that to bhyvectl.


I’m pretty sure it’s tooling that’s displaying something off, since
hopper is showing me this as

0x00102a56 cmpword [0x2], 0x0

Which is very similar to what r2 is giving me:

;-- cons_init:
0x00102a50  53 push rbx; /arch/x86:43
0x00102a51  e8ea0a call sym.hypervisor_detect  ; /arch/x86:47
0x00102a56  66833da4d5ef.  cmp word [0x0002], 0; /arch/x86:62


 This is reading the 16-bit value from memory location 0x2. Hard to see 
why this would generate a page-fault - the zero page is often mapped 
read-only. Perhaps rumpkernel doesn't have a mapping for it, but then, 
the offset for the access would be incorrect (maybe a linking issue with 
the location of variables ?).



Maybe I’m off with my analysis of the actual fault here, but how I understand
the source (assuming compilers work as I would expect, which is not always true)
the values here are initialised from values in the bios data area (which is
zeroed out on bhyve):


 It shouldn't matter that those were zero. Loading them into a memory 
location shouldn't be a problem.



Here’s my full output from bhyvectl --get-all:

ID  Length  Name
0   128MB   sysmem
Address Length  Segment Offset  Prot  Flags
0   128MB   sysmem  0   RWX
efer[0] 0x0500


 Ok, the guest is in 64-bit mode - the LMA bit is set. This implies 
that rumpkernel has set up it's own mappings, since the multiboot loader 
entered the guest in flat 32-bit mode.



cr0[0]  0x80010031


 Paging is enabled (bit 31) as expected.

later,

PEter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: rumpkernel and bhyve: triple faults

[Peter Grehan]

Hi Fabian,


657   0 350887309700 vm testing[0]: handled exception vmexit at 0x102a56
656   0 350887309570 vm testing[0]: Exception 14 pending
655   0 350887309442 vm testing[0]: Setting intr_shadow to 0 succeeded
654   0 350887305126 vm testing[0]: Reflecting exception 14/0 into the 
guest
653   0 350887302436 vm testing[0]: vm_exit_intinfo: info1(0x8b0e)
652   0 350887248280 vm testing[0]: Resume execution at 0x102a56
651   0 350887184160 vm testing[0]: vm_entry_intinfo: info1(0), 
info2(0x8b0e), retinfo(0x8b0e)
650   0 350887184040 vm testing[0]: Exception 14 delivered: 0x8b0e
649   0 350887182668 vm testing[0]: handled exception vmexit at 0x102a56



 Exception 14 is a page fault (SDM Vol3 ch 6.15). The exception type is 
"fault" which means it is delivered at the address it was detected at.


 This cascaded very quickly into a triple-fault, so it looks like it 
could possibly be an issue with the stack. One debug tool you do have is 
to get a register dump on exit, with 'bhyvectl --get-all --vm=name>'.


 For a page-fault, the virtual address that resulted in the fault will 
be in the CR2 register.


 From the code at the faulting address:

> 00102a50 :
>102a50:   push   rbx
>102a51:   call   103540 
>102a56:   cmpWORD PTR [rip-0x102a5c],0x0# 2 



 It's using RIP-relative addressing here, but objdump seems to think 
this may be an offset in the current_lwp structure - is it possible that 
may have an uninitialized value ?


 (I don't believe this has anything to do with VGA).

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Ryzen lockup on bhyve was (Re: new Ryzen lockup issue ?)

[Peter Grehan]

On the Epyc it seemed to take a longer, but eventually the box locked
up and the watchdog ended up rebooting the server. I started 3, vCPU
guests all with separate CPU's pinned. It lasted 9hrs.


The Ryzen board also locked up with just 2 VMs running, and all vCPUs
pinned to separate cores.


 Thanks for that Mike - it rules out any issues related to vCPUs moving 
around on physical cores.


 I'd like to now see if there's a way to get this to happen quicker - 
perhaps running the test with increasing numbers of guest vCPUs ? From 
what I've seen, the network test only really uses 2 vCPUs so it's 
probably ok to even go to oversubscribing.


 I'll get this going on my Ryzen.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Ryzen lockup on bhyve was (Re: new Ryzen lockup issue ?)

[Peter Grehan]

Hi Mike,


  I could reproduce this on a Ryzen 1700 with HT enabled - it took ~700
seconds, and the entire machine locked up.

  An interesting point is a single VM was able to run to completion.

  I'll keep testing with various combinations of numbers of VMs,
different vCPUs per VM, and pinning guest vCPUs.


 3 4*vCPU guests locked the machine up in ~170 secs, 4 guests ran to 
completion :(



Not sure if its related to core count or not, but it takes several
instances on the Epyc to lock up and takes a lot longer. I enabled IOMMU
on it as well, and it's dmesg shows below.  But the lockups are still
there with or without it enabled.


 Would you be able to try on the Epyc with vCPUs pinned, and all guest 
vCPUs on different physical CPUs ?


 i.e. for a configuration that will lock up, run the same configuration 
but with vCPUs pinned. For example, a 4 vCPU guest that will have vCPUs 
0-3 pinned to host physical CPUs 7-10 would look like:


 -c 4 -p 0:7 -p 1:8 -p 2:9 -p 3:10

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Migrate from real server to virtual

[Peter Grehan]

But I think the bigger problem is that this is an MBR disk with Windows 7 in 
it, I
do not believe that bhyve can run Windows 7 at this time.


 Bhyve has been able to run Win7 since 2015, though x64 only and in 
UEFI mode.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Migrate from real server to virtual

[Peter Grehan]

Have you tried using the CSM UEFI firmware that came in same bundle as the
one above? It's worth a try.


 Unfortunately that is known not to work with Windows :(

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve and mmap(2)

[Peter Grehan]

Hi Fabian,


Would be possible to use mmap(2) around libvmmapi to map a file
(MAP_PRIVATE) and or shared memory segments (MAP_SHARED) into guest
physical address space?


 It currently isn't - doing that will just insert the mappings into the 
user process's view, and not into guest address space.


 Bhyve currently uses a separate vmspace for guest address space. While 
this simplifies the EPT code internally, it also prevents direct 
manipulation of the address space using standard system calls.


 A possible solution to this would be to have the ability to apply 
address-space manipulation calls to an arbitrary address space - this 
was brought up in Linux a long while back: https://lwn.net/Articles/19047/


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Ryzen lockup on bhyve was (Re: new Ryzen lockup issue ?)

[Peter Grehan]

Hi Mike,

 (moving this to -virtualization)


Actually I can confirm the same sort of hard lockup happens on my Epyc
board with RELENG11.  It also happens in current. I will file a PR and
post on freebsd-current in case someone has any suggestions on how to
try and figure out whats going on.


 I could reproduce this on a Ryzen 1700 with HT enabled - it took ~700 
seconds, and the entire machine locked up.


 An interesting point is a single VM was able to run to completion.

 I'll keep testing with various combinations of numbers of VMs, 
different vCPUs per VM, and pinning guest vCPUs.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D14473: userboot: add callbacks to set unrestricted guest mode

[grehan (Peter Grehan)]

grehan accepted this revision.
grehan added a comment.


  You may be able to get away with just the "set" call. Given that all h/w that 
is supported by bhyve, with the exception of only Nehalem systems, support 
this, you may want to use just the error from the set call to handle this 
low-probability error.
  
  Note that AMD SVM has always supported this mode. "Unrestricted Guest" is an 
Intel-only feature, but given that the name is baked into a bhyve API already, 
there doesn't seem to be any harm in keeping it going.

REPOSITORY
  rS FreeBSD src repository

REVISION DETAIL
  https://reviews.freebsd.org/D14473

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: fabian.freyer_physik.tu-berlin.de, imp, #bhyve, grehan
Cc: grehan, imp, freebsd-virtualization-list, #contributor_reviews_base
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: VMX exit reason=33 and general userboot.so questions

[Peter Grehan]

Hi Fabian,

>exit_reason 33

 From the Intel SDM, vol 3B Appendix C, this error is "VM-entry failure 
due to invalid guest state".


 These errors can be difficult to debug given the large amount of guest 
state involved :(


 However, looking at the state from your dump:

> tr desc[0]  0x/0x/0x

 I believe you will have to set this. Here's the comment and relevant 
code fragment from grub2-bhyve 
grub-core/kern/emu/bhyve_hostif.c:grub_emu_bhyve_boot32()


  /*
   * XXX TR is pointing to null selector even though we set the
   * TSS segment to be usable with a base address and limit of 0.
   * Has to be 8b or vmenter will fail
   */
  desc_access = 0x008b;
  assert(vm_set_desc(bhyve_ctx, 0, VM_REG_GUEST_TR, 0x1000, 0x67,
 desc_access) == 0);

 grub2-bhyve has been able to load/boot multiboot images, so I suspect 
the register settings in grub_emu_bhyve_boot32() are a good place to 
start from.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve and contention

[Peter Grehan]

2.
In the following context, the server is the same but this time all five
guests have -c 4 per guest, so bhyve is asking 12 more cores than that
existing in hardware. Does the guest fail to load, do either guest or
server crash?


The is core over commit, very common in the virtualization world,
bhyve does its best effort to give the guests cores as needed.


 To add to what Rod said - bhyve uses a thread for each vCPU. It's up 
to the FreeBSD scheduler to determine where/when these threads run.


 It is possible for a guest to fail, for example if a spinlock times 
out due to vCPUs not being able to run to release a lock. This is a 
general problem with virtualization and can occur even on VMWare ESXi 
with heavy oversubscription.


 That being said, there is certainly scope to provide more information 
to the FreeBSD scheduler so it can make better decisions when running VM 
guests.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve passthrough problems (update)

[Peter Grehan]

Hi Mario,


Well, now, even without setting up passthrough, the VM doesn't start
(which it did, before I svned up /usr/src).

...

working:   321652
not working (current): 328163

...

Any updates on this issue?


 Can you verify if the issue still occurrs if you destroy the guest 
before re-invoking it ? (i.e. bhyvectl --destroy --vm=WIN-764 )


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D13995: NVMe controller emulator for bhyve.

[grehan (Peter Grehan)]

grehan abandoned this revision.
grehan added a comment.


  Closing this revision - moving to https://reviews.freebsd.org/D14022

REVISION DETAIL
  https://reviews.freebsd.org/D13995

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: grehan, trasz, imp, sux2mfgj_gmail.com
Cc: chuck, seanc, rgrimes, cem, freebsd-virtualization-list
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D13995: NVMe controller emulator for bhyve.

[grehan (Peter Grehan)]

grehan commandeered this revision.
grehan edited reviewers, added: sux2mfgj_gmail.com; removed: grehan.

REVISION DETAIL
  https://reviews.freebsd.org/D13995

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: grehan, trasz, imp, sux2mfgj_gmail.com
Cc: chuck, seanc, rgrimes, cem, freebsd-virtualization-list
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D13995: NVMe controller emulator for bhyve.

[grehan (Peter Grehan)]

grehan added a comment.


  Yep, that's correct. The code in the new review is based on this code so it's 
really a continuation of the effort.

REVISION DETAIL
  https://reviews.freebsd.org/D13995

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: sux2mfgj_gmail.com, grehan, trasz, imp
Cc: chuck, seanc, rgrimes, cem, freebsd-virtualization-list
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D13995: NVMe controller emulator for bhyve.

[grehan (Peter Grehan)]

grehan requested changes to this revision.
grehan added a comment.
This revision now requires changes to proceed.


  Chuck - the review for this is now in https://reviews.freebsd.org/D14022, 
where I believe your comments have already been addressed.

REVISION DETAIL
  https://reviews.freebsd.org/D13995

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: sux2mfgj_gmail.com, grehan, trasz, imp
Cc: chuck, seanc, rgrimes, cem, freebsd-virtualization-list
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D13995: NVMe controller emulator for bhyve.

[grehan (Peter Grehan)]

grehan added a comment.


  The updated version of this code is at https://reviews.freebsd.org/D14022
  
  I've moved the reviewer/subscriber list over as well.

REVISION DETAIL
  https://reviews.freebsd.org/D13995

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: sux2mfgj_gmail.com, grehan, trasz, imp
Cc: seanc, rgrimes, cem, freebsd-virtualization-list, chuck
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D13995: NVMe controller emulator for bhyve.

[grehan (Peter Grehan)]

grehan added a comment.


  Leon Dang has also been working on NVMe emulation, and his version works with 
Linux, Windows and UEFI boot. I'll post that code for review since it is a bit 
more recent and tested. In the meantime, that version can be seen at 
www.freebsd.org/~grehan/pci_nvme.c
  
  (There is also a version of UEFI with NVMe support compiled in at 
www.freebsd.org/~grehan/BHYVE_NVMe.fd.xz )

REVISION DETAIL
  https://reviews.freebsd.org/D13995

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: sux2mfgj_gmail.com, grehan, trasz, imp
Cc: rgrimes, cem, freebsd-virtualization-list, chuck
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve passthrough problems

[Peter Grehan]

Hi Mario,


I'm trying to passthrough a firewire pci card to a windows 7 64 guest.


 First, you'll need to verify if that card supports MSI or MSI-x since 
that is required by bhyve for passthru support. This can be seen with 
'pciconf -lbc' on the FreeBSD host.



VM exits with:

Assertion failed: (mr->name == memp->name), function
unregister_mem, file /usr/src/usr.sbin/bhyve/mem.c, line 265.
Abort trap


 This is bug in bhyve when running Windows with PCI devices that have 
more than 1 memory BAR (e.g. Intel igb ethernet devices), due to the 
order that Windows determines BAR sizes. I've been meaning to fix this 
for a long time so will try and get to it.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve/amd: interrupt delivered when it shouldn't be?

[Peter Grehan]

Hi Andriy,


The hardware is AMD.


 Ryzen ?


But what I see suggests that at this point a Local APIC timer interrupt gets
delivered to the thread.  And that causes all the mess as the thread holding the
spinlock gets preempted.

Does this ring a bell to anyone?


 I have seen something similar to this after about ~20 mins when doing 
a current -j 16 buildworld in a guest, with the symptom being a spinlock 
timeout, with one vCPU spinning in


smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x352/frame 
0xfe02c80098d0

smp_masked_invlpg() at smp_masked_invlpg+0x4c/frame 0xfe02c8009900
pmap_invalidate_page() at pmap_invalidate_page+0x191/frame 
0xfe02c8009950

pmap_ts_referenced() at pmap_ts_referenced+0x7b3/frame 0xfe02c8009a00
vm_pageout() at vm_pageout+0xe04/frame 0xfe02c8009a70

... and all the others eventually spinning on that held lock.

However, NMIs are still able to get through (the post-panic ddb NMI IPI) 
so the VM isn't completely locked up - either an interrupt is missed, or 
a write isn't seen by the vCPU issuing the tlb shootdown.



Is there any suspect code?


 Not sure yet, but the interrupt-injection path could do with a close 
inspection.



It seems that we set v_intr_masking bit, so the rFLAGS / eFLAGS should be
completely virtualized.  So, maybe a hardware issue?


 Hard to say. Running with all vCPUs pinned makes the problem go away, 
but that could just mean the issue is isolated to when vCPUs migrate.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Recovering an ZFS vm

[Peter Grehan]

Hi Randy,


I have a Centos vm that has suddenly stopped booting. At the console, grub
tells me the following if I attempt to list any of the available partitions.

error: not a correct XFS inode.
error: not a correct XFS inode.
error: not a correct XFS inode.
error: not a correct XFS inode.
error: not a correct XFS inode.
Filesystem type xfs, UUID 7652ffda-f7c5-408a-b0ce-b554b66fc2e5 - Partition
start at 2048 - Total size 2097152 sectors
grub>

Is there an easy way to recover this? This has happened more than once.
Just so happens there is something on this image I would like to have
access to...


 Looks like the grub partition was upgraded to the version of XFS that 
has the CRC feature enabled (7.2 ?). Unfortunately this feature is not 
understood by grub-bhyve :(


 One way to recover the disk is to create a new VM with the most recent 
CentOS, but using UEFI for the bootloader. Then, add this disk to the 
guest, and from within the guest I think you can run an XFS utility that 
will disable the use of CRCs on that partition.


 The proper fix would be for grub-bhyve to be updated to the latest 
version of grub2, though a workaround is to create guests with UEFI and 
not use grub-bhyve.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve. Unable install Windows 7/Server 2008R2

[Peter Grehan]

Hi,


  bhyve -c 2 -s 0,hostbridge -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0


 For win7/2k8*, the sector size presented to the guest has to be forced 
to 512 bytes i.e. for the ahci-cd slot, the config would look like:


 -s 3,ahci-hd,/dev/zvol/my_zroot/VM/img/win7/disk0,sectorsize=512

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve uses all available memory during IO-intensive operations

[Peter Grehan]

bargs  -A -H -P -S


 The -S flag to bhyve wires guest memory so it won't be swapped out.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Onboard USB controller passthrough

[Peter Grehan]

The controllers looks like this

# pciconf -l -BbcV ppt2@pci0:0:26:0
ppt2@pci0:0:26:0:   class=0x0c0320 card=0x72708086 chip=0x8d2d8086
rev=0x05 hdr=0x00
 bar   [10] = type Memory, range 32, base 0xc6134000, size 1024, enabled




 cap 01[50] = powerspec 2  supports D0 D3  current D0
 cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
 cap 13[98] = PCI Advanced Features: FLR TP


 More importantly, EHCI controllers don't support MSI/MSI-x interrupts 
which is currently required for bhyve PCI-passthru.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve and backed-up dd disk image

[Peter Grehan]

Hi,


Would I be able to spin up in bhyve a disk image that had been
obtained from the disk of a bare-metal standard freebsd installation?


 In theory, yes. There will always be some issues with differences in 
devices (e.g. using igb0 on the host, but vtnet0 in the guest).


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve detaches my hard disk

[Peter Grehan]

Hi Roberto,


Attached to this email is the script that I am using to launch the bhyve
machine and the output
from the console at the moment of the crash, the output of pciconf -lv and
uname -a.


 The attachment didn't have the bhyve launch script. Would you be able 
to resend that, along with a dmesg of the host system ?


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve passthrough fails

[Peter Grehan]

Hi Rod,


Hi Farhan,

bhyve: PCI device at 2/0/0 is not using the ppt(4) driver.


   Just noticed this myself: looks like a regression :( I'll see if I can
work it out and get a fix in.


What is the state of bhyve in 10.4?  Do we need to fire off some
quick tests


 Always worth some quick tests before a release.

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve passthrough fails

[Peter Grehan]

Hi Farhan,

bhyve: PCI device at 2/0/0 is not using the ppt(4) driver.


 Just noticed this myself: looks like a regression :( I'll see if I can 
work it out and get a fix in.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: openbsd 6.0 or 6.1 guest on freebsd-12 (current) host

[Peter Grehan]

There is a bug in the early Intel Skylake CPU series.  The Silicon
Debug (SDBG) flag is turned on as a default and this causes the
OpenBSD 6.1 .iso to refuse installation.


 The '-w' option for bhyve can be used to work around this.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Xeon E5 (PowerEdge 320) bailing out

[Peter Grehan]

Hi,


Trying to run Bhyve on a PowerEdge 320 system which DOES have a POPCNT
instruction which shows support for VT-x support via EPT but I still get
the this error there

sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 2048M -t tap0 -d guest.img -i
-I FreeBSD-10.3-RELEASE-amd64-bootonly.iso fbsd10
Launching virtual machine "fbsd10" ...
vm_create: Device not configured


 You may want to check your BIOS to make sure that VT-x is enabled 
(/var/log/messages may also have a message from the bhyve kernel module 
about this)



dmesg:
vmx_init: processor does not support VMX operation

cat /var/run/dmesg.boot | grep -i popcnt
   Features2=0xffba2203
   Features2=0xffba2203

Isn't this processor supposed to support Bhyve (Xeon E5)??


 A quick bit of searching shows the R320 shipped with a Xeon E5-2400 
(aka Ivy Bridge) which will run bhyve fine.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: My USB card is a bhyve killer

[Peter Grehan]

Hi,


Any further thoughts on this?


 Seems a bit of a mystery. The PCI dump looked fine.


I would like to return this card back to Amazon in the next few days. So
if anybody needs any more debug info please let me know.


 I'll contact you off-list.


I do plan to
retain the other NEC uPD720201 chipset based card since it works within
bhyve. It seems something is broken in FreeBSD with respect to PCIe USB
cards with all the boot freezes i have had.

>

Is there any old version of FreeBSD which i can try to figure out if
this is a regression?


 It might be worth trying 11.0 since that was pre-capsicum.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: My USB card is a bhyve killer

[Peter Grehan]

Hi,


I have a PCIe USB card which is exported to the bhyve vm using
passthrough. I have tried to use the card in Ubuntu, Debian 8 and Centos
7. Each time the OS has given a segmentation fault on bootup and killed
the bhyve session. This is what i get in the bhyve-log:


 Is stderr logged anywhere ? That may have additional error output from 
the bhyve passthru code.



The USB card works fine on the host and is detected by the xhci driver.
The card is "Inateck Superspeed 4 Ports PCI-E to USB 3.0 Expansion Card
- Interface USB 3.0 4-Port" (https://www.amazon.com/gp/product/B00B6ZCNGM)

ppt3@pci0:129:0:0:  class=0x0c0330 card=0x11001b73 chip=0x11001b73
rev=0x10 hdr=0x00
 vendor = 'Fresco Logic'
 device = 'FL1100 USB 3.0 Host Controller'
 class  = serial bus
 subclass   = USB


 Would you be able to post the output of

pciconf -l -BbcV pci0:129:0:0

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve Broken: whose fault (AMD, FreeBSD, ZFS ...?)

[Peter Grehan]

On 7/21/17 5:54 PM, Peter Grehan wrote:

remember that the quoted problem is spit out by the guest, not the host.

That said, the 'top' line on the frozen bhyve was:

29380 root  22  200  1060M   928M kqread  5 218:32 399.30% 
bhyve


... indicating that the bhyve had almost all it's memory... and the 
system had also 500M free when I checked it.


  Are you using a "-j" option to buildworld in the guest ? You'd need at 
least 1G for each vCPU when doing parallel builds (and preferably a bit 
more) or you will swap heavily.


 I should add: I'm not saying this will fix the problem, but if it is 
an issue with the guest swapping, giving the guest more RAM is a workaround.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve Broken: whose fault (AMD, FreeBSD, ZFS ...?)

[Peter Grehan]

oh ... and ... the console spit out:

swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4522, size: 8192

(swap is on a separate zVol).


 Ok - you may have hit a separate issue. Is ZFS ARC limited on your 
setup ? If bhyve and ZFS (and other consumers) end up fighting for 
memory, everyone loses :( A general rule of thumb is to limit ARC to 
less than the bhyve VM usage + a few additional gig for the base system.


 The FreeBSD default of giving ZFS all memory minus 1GB doesn't work 
too well when running VMs.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve Broken: whose fault (AMD, FreeBSD, ZFS ...?)

[Peter Grehan]

Hi,


What should be next steps here?  This is repeatable.  The host is stable
(it can makeworld -j32 in about 25 minutes ... so it's hardware seems
good).  Is this an AMD bug?  Is it bad to use ZFS ZVols?


 ZVols are fine. Is the guest panic a spinlock timeout ?

 I believe this is a bug in bhyve/SVM. It appears somewhat related to 
processor speed (I can't repro on a 2.3GHz 8 CPU Opteron 6320, but can 
hit it after 15 mins or so on a Ryzen 1700, with/without SMT).


 Anish and I are currently chasing this and have repros. An experiment 
you could try is to run with the vCPUs pinned i.e. for a 4 vCPU guest, 
add the options "-p 0:1 -p 1:2 -p 2:3 -p 3:4".


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: VNC stuff

[Peter Grehan]

Hi Rod,


r302504 is not likely to be MFC'ed at this time, this is the e1000 driver
and man page.  This code has known issues at this time and is an action
item list on the bhyve developers work list.


 The e1000 driver was MFC'd 11 months ago. The issues are currently 
only with Windows - it works fine with Linux/*BSD/Solaris. I'm hoping 
that the Windows issue can be fixed before 11.1.



Peter, could you please do an RFa to re@ to have R31 MFC'ed?  We
have 3 days left before 11.1B2.  If you want me to follow up on that
just send me back a direct mentor approval and I'll do the RE and
MFC work to get this in tree.


 I'll get in touch offline on this.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: VNC stuff

[Peter Grehan]

Hi Rod,


Do we want the VNC/Auth fix (once cleaned up for Coverty issues) in
the 11.1 release?

Doesnt this allow a lot more clients to connect to the fb?

Or is it only a few clients that are being pendandic about
using a questionable authentication means?


 The main one is the OSX built-in client where vnc-auth is mandatory, 
but that also requires rfb version 3.3 support in the bhyve server 
(admittedly minor, but probably not in time for 11.1).


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Windows Server 2016 as guest on bhyve 4k emulated drive

[Peter Grehan]

Hi Artur,


I can not explain how Windows comes to 8192 bytes.


 As explained in Jeremya F's email, that's the zvol default record size.



about 512E drives. According to the article, the 512E drives should not
be used, because this configuration affects both the lifetime and the
performance of the drives.


 Bhyve presents a virtual drive so this doesn't apply.


(https://msdn.microsoft.com/en-us/windows/compatibility/advanced-format-disk-compatibility-update?f=255=-2147217396)

For this reason, I started using 4K emulated drive, both logical and
physical (sectorsize=4096/4096), but Windows can not be installed on a
4K emulated drive.


 I'd recommend sticking with "sectorsize=512/4096" for now.

 The 4096/4096 appears to be a bhyve bug that needs to be fixed, 
especially since the NVMe emulation GSoC project has been approved*, and 
that mandates 4K block size.


later,

Peter.

* https://summerofcode.withgoogle.com/projects/#5252121027936256
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Windows Server 2016 as guest on bhyve 4k emulated drive

[Peter Grehan]

Hi Artur,


if the option 'sectorsize=4096/4096' for the emulated drive specified, the
installation process of "Windows Server 2016" completes the first stage
('Copying Windows files', 'Getting files ready for installation',
'Installing features' etc.) and fails after rebooting to the second stage.

The second stage ends with the following message:
Boot Failed. Windows Boot Manager
Boot Failed. EFI DVD/CDROM
Boot Failed. EFI DVD/CDROM 1
Boot Failed. EFI Hard Drive

I suppose that Windows Boot Manager has problems when booting the 4k
emulated drive.

Does anyone have any idea what can be the cause of this?


 I think it may be an issue with the AHCI driver in UEFI not returning 
the correct block size - will look into it further.


 I did verify that setting 'sectorsize=512/4096' works fine with 2k16.

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

[Differential] D10540: vmx: explicit checking for necessary invvpid/invept types

[grehan (Peter Grehan)]

grehan added inline comments.

INLINE COMMENTS

> ept.c:68
> + (((cap) & INVEPT_REQUIRED_TYPES_MASK) == INVEPT_REQUIRED_TYPES_MASK)
>  
>  #define  EPT_PWLEVELS4   /* page walk levels */

Might be worth splitting these out into individual tests, since I have a change 
that will look at the EPT capabilities individually (for PR 203994, bhyve as a 
KVM guest. Jan - judging from your work in 
https://patchwork.kernel.org/project/kvm/list/?submitter=170053 you are 
intimately familiar with this issue :)

REPOSITORY
  rS FreeBSD src repository

REVISION DETAIL
  https://reviews.freebsd.org/D10540

EMAIL PREFERENCES
  https://reviews.freebsd.org/settings/panel/emailpreferences/

To: jan.dakinevich_gmail.com, grehan, neel
Cc: novel, anish, imp, freebsd-virtualization-list
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: dos in bhyve

[Peter Grehan]

Hi Andriy,


I wonder if anyone was able to run any DOS variant in bhyve.


 I've run vm86 mode in Linux but that doesn't really count :)


I tried FreeDOS using BHYVE_UEFI_CSM firmware, but it seems to just hang.


 Try looking at the CSM/BIOS debug output that is available on the 
bvmcons debug port (use bhyve's "-b" option, though you'll want to send 
serial port output to somewhere other than stdout (e.g an nmdm port).


 That will at least show what BIOS calls are being issued, and if any 
are unimplemented.



I also tried adding a frame buffer device, it seems that there is some reaction
to key presses (RET) sent via VNC, but the screen stays black, so it's
impossible to tell if there are any prompts and what happens.


 The VGA renderer hasn't been hooked up in this mode yet (fbuf,vga=on). 
I can work with you offline to get that going.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve fbuf and keys on the numeric part of the keyboard

[Peter Grehan]

Hi Christian,


After having used bhyve for servers for quite a while, I have just
started using it with the graphical console using vnc a few days ago.

This stuff is awesome. Having a native solution for virtualization on
FreeBSD is great.

However, I had some trouble with keyboard mappings and the vnc based
console. I guess that having a non-US keyboard is what causes problems.


 Yep, that's correct. The VNC client is also part of this, since that's 
where the mapping from the native keyboard scancodes to the VNC protocol 
occurs.



The first thing I always do after installing a new operating system is
fetching my ssh public keys via https. I had a hard time finding out how
to type '/' and I never found a way to type '-'.

I know that this stuff is hard to fix. What I am hoping is, that it
could be easy to add the '/*-+' keys on the numeric part of the keyboard
to the mapping. Those are often helpful and I think they map the same
way on all keyboards.


 I think that works Ok with a US keyboard and VNCviewer, but will retry 
to make sure.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: OpenBSD 6.1-snapshot no longer works in bhyve

[Peter Grehan]

Hi Jason,


FYI, 6.1-release and 6.1-snapshots will not boot at this stage.  I have my
new production kit arriving so I can test on different hardware to
determine if this is a CPU rev. issue but thought I'd put it here to get
the thought processes working on it.  I am running 11.0-release p8


 You can use bhyve's "-w" option to workaround this.

 I'll cherrypick the hardened BSD commit which fixes this in the meantime

https://github.com/HardenedBSD/hardenedBSD/commit/cc91b57f4d1dabddfbf8b1e7655bc19908f24f78

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Host filesystem in Bhyve

[Peter Grehan]

Hi Jakub,


We've been using FreeNAS 10 (now Corral) as our testbed for virtio-9p
for over a year now and I think at this point it's stable enough to
be upstreamed. :-) Please expect phabricator diffs in the upcoming
weeks!


 Great to hear ;)

 So... any news on a FreeBSD 9p client ?

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Host filesystem in Bhyve

[Peter Grehan]

Hi Rajil,


It seems that FreeNAS 10 is able to expose the host filesystem to bhyve
sessions.
Is there any plan to bring that feature into FreeBSD?


 I'd like to see it happen but it's really up to the iX guys who wrote 
that code to get it in.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Ubuntu UEFI Won't Reboot

[Peter Grehan]

Hi Vincent,


So after a minute or two, I get the UEFI Shell and when I exit, I get
something that appears to be the bhyve BIOS (?) when I go from there
to the Boot Maintenance Manager, and then « Boot From File » I can
select the block device, then , then , then grubx64.efi
and then I get the Grub menu from which I can select Ubuntu, which in
turn boots the OS just fine. Now, the question is, can I configure
bhyve to look for that file instead of whatever it is currently
looking for and not finding? Or do I have to modify the ESP?


 This is a known issue in that UEFI nvvars aren't written to permanent 
storage (e.g. see 
https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-September/004808.html). 
A number of guest o/s's use nvvars to handle boot order/non-standard 
boot loader names etc, and expect changes to these to be persistent.


 There is a fix that just needs to be comitted (to both bhyve and 
UEFI/bhyve). Hope to get to that soon.


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve /Xen

[Peter Grehan]

Hi Dustin,


I'm running 2016 (both Standard and Essentials) under 12-CURRENT
bhyve without any obvious issues other than it's acting a lot slower
than it should be. However I'm not entirely sure that it's a bhyve
issue.


 Any perf issues in particular ? network/disk ?

 I have noticed networking performance on the same h/w as other guests 
with virtio isn't so good (e.g. iperf3 results).


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve /Xen

[Peter Grehan]

Hi,


1)  What do I need to send a bhyve crashing report?


 This list is fine, or you can file a PR against FreeBSD (you need to 
create an account for that, https://bugs.freebsd.org/bugzilla/enter_bug.cgi)



Seems that yes 6 Vms are running and are crashing at the same time.

2)  Windows 2016 and bhyve


Why can Widnows 2016 rpc not work with bhyve?


 Can you describe the case that isn't working ? (to get a repro)


3)  Bhyve to Xen .

Is there an easy way to migrate from bhyve to Xen on FreeBSD 11 ?


 You should be able to boot a guest disk image created for bhyve under Xen.

later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

[Peter Grehan]

Hi,


Does bhyve not execute peripheral cards' option ROMs?


 Not yet.


I guess it doesn't. This could explain a lot of strange
behaviour seen resulting from running in a VM.


 Yes.


How does UEFI work in this regard? My guess is that cards
have to explicitly support the new boot method (UEFI)?


 Yes - an additional section in the option ROM is needed, but as 
mentioned in an earlier email, that support is now widespread thanks to 
Windows.



So passthrough with newer cards may be easier? This could
explain why the newer RX 480 worked right away, and the
older Quadro 2000 (and a lot of other nVidia cards without
manufacturer's support for VMs) had no chance -- UEFI cards
are somehow more "autonomous".


 Possibly, though it might also be the card itself not requiring as 
much initialization from the option ROM.



It all is just speculation on my side, I know nothing about
this UEFI stuff.

Could you summarize in couple sentences what's the deal between
bhyve and UEFI (if there is any), or future plans?


 UEFI is the ROM firmware for bhyve (and most modern PCs). bhyve has a 
custom build of the standard Intel EDK2 distribution:

   https://github.com/freebsd/uefi-edk2/tree/bhyve/UDK2014.SP1

 The changes are to support running as a hypervisor guest, where a lot 
of what is in a normal boot ROM isn't required (e.g. DRAM controller 
setup, CPU microcode update), and it also contains drivers for device 
emulations supported by bhyve.


 Currently, the ability to process an option ROM has been disabled.

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

[Peter Grehan]

Hi,


  That is extremely likely. bhyve itself doesn't have a BIOS, though
bhyve/UEFI could be modified to handle options ROMs (see
http://awilliam.github.io/presentations/KVM-Forum-2014/#/)


Hm, interesting. I wonder if a card that's not designed for use
with UEFI is destined not to work well/at_all with bhyve...
I'll read the presentation later.


 I think in general almost all cards have UEFI ROM support these days 
since it has been mandated by Microsoft. However, as Rod mentioned, the 
bhyve UEFI implementation does not run PCI device option ROMs.


 (see 
http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html)



-GPU UUID:
GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2
+GPU UUID: Unknown Error


  That implies some type of h/w access isn't working, either MMIO
registers or response from a DMA command.


I have a feeling it's something to do with DMA that's
not getting configured correctly for data transfers,
and returns wrong data (or good data to wrong location).


 Yes.


  A general issue with PCI passthrough is that often MMIO from the
guest works, since that is just VT-x remapping, but DMA doesn't work
due to issues with IOMMU programming (or incorrect mappings being
used). This gives a device that partially works in that registers can
be read, but data transfer doesn't work.


Didn't we verify that the BARs are programmed correctly?


 The BAR values you see are fictional and are created by bhyve. The 
actual physical BAR values are those set up by the host BIOS. bhyve uses 
EPT mappings to translate between the 'fake' value and the real value.



So you're saying that bhyve has a bug in that it doesn't
program the IOMMU right to match guest's memory-mapped
address regions to host's addresses?


 There isn't a known bug, but the 64-bit BAR region hasn't been tested 
for a long time so it's possible there is an issue with it.



BTW, is it [generally] safe to decrease the BAR base address further?
My workstation has a CPU with just 36 address bits...

  Yes. The only potential conflict is with the top of guest RAM, and 36
bits is a lot of RAM :)


64G of RAM isn't that much these days, how incredible is that :)
But you're saying there's nothing else inbetween the top of
guest's RAM and the BAR base? In that case it's nothing to
worry about at all, as a guest will always have less RAM that
the host's CPU can address.


 Right - the 64-bit PCI decode region would be set dynamically based on 
the phys address bits, rather than being a hard-coded value.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Help getting vbox .vdi imares to run under Freenas

[Peter Grehan]

Hi Nils,


can I do this inside of a jail? I'm on FreeNAS, so I don't want to (and
can't) install and compile a lot of stuff in the main FN host. From what
I've managed to read up on, jails don't get the full networking, but at
the grub stage I should be OK, correct?


 I've not tried it but I believe so. The /dev/vmm device file would 
have to be made available in the jail.


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Help getting vbox .vdi imares to run under Freenas

[Peter Grehan]

Hi Nils,


On 2017-01-08 19:04, Peter Grehan wrote:


There's some additional info at
https://github.com/pr1ntf/iohyve/issues/228 -

grub> ls (hd0) (cd0) (host) grub> ls (hd0) Device hd0: No known
filesystem detected - Total size 16777216 sectors

... so it looks like grub isn't able to auto-detect the
partitions.


Anything that I can do to debug that?
https://github.com/grehan-freebsd/grub2-bhyve doesn't appear very
active... would runningit under gdb make sense, or are there any
other diagnostics that you can think of? All the diagnostics I've
done have said that the image looks good (extracting MBR, looking at
it with fdisk -l under Linux, etc.) Thanks Nils


 Running it under gdb is probably the best (since it's a curses app, so 
you will have to use the techniques in 
http://www.dirac.org/linux/gdb/07-Debugging_Ncurses_Programs.php)


later,

Peter.

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

[Peter Grehan]

Hi,


BTW, is it [generally] safe to decrease the BAR base address further?

> My workstation has a CPU with just 36 address bits...

 Yes. The only potential conflict is with the top of guest RAM, and 36 
bits is a lot of RAM :)


later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

[Peter Grehan]

Hi,


There doesn't seem to be support for CPUID 0x4001 in bhyve either.

What is it supposed to do?


As far as I can tell it's the Hypervisor extension flags list. The lack
of these extensions/optimisations might explain why your FreeBSD VM runs
slow but their presence also causes the nVidia driver to refuse to run.


 That leaf is KVM-only. bhyve doesn't have any additional hypervisor 
leaves beyond 0x400


 (the spec for this is https://lwn.net/Articles/301888/)

later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"