Re: bhyve clock issue

[Neel Natu]

What was the timecounter being used when you were seeing the time skew?

Best
Neel

> On Apr 26, 2016, at 6:46 AM, Ilario Febi <if...@schema31.it> wrote:
> 
> Yes, 
> 
> I found errors in /var/log/messages and fix adding: 
> 
> kern.timecounter.hardware=TSC-low
> into /etc/sysctl.conf file
> 
> Thanks for support.
> 
> 
> 
> Il 14/04/16 18:13, Neel Natu ha scritto:
>> Hi,
>> 
>> Did you see anything interesting in the ntpd log file around the time the 
>> large skew happened?
>> 
>> Best
>> Neel
>> 
>>> On Apr 13, 2016, at 3:44 AM, Ilario Febi <if...@schema31.it> wrote:
>>> 
>>> Hi All
>>> 
>>> I have a strange issue about the clock of some FreeBSD Bhyve guests,
>>> maybe someone of you can help me to diagnose it and work on a
>>> solution.
>>> 
>>> In detail, sometimes (without an apparent regularity) one of four
>>> bhyve guest's clock goes exactly 7 minutes and 10 seconds in the
>>> future.
>>> 
>>> This clock stays unaligned for about 20 minutes, then, without any
>>> human intervention, the clock realign to proper condition.
>>> 
>>> My configuration is:
>>> 
>>> envrm02 (bhyve HOST, 10.2-RELEASE-p14)
>>> -> og-qa (bhyve guest - 10.2-RELEASE-p14)
>>> -> og-ci (bhyve guest - 10.2-RELEASE-p14)
>>> -> og-test (bhyve guest - 10.2-RELEASE-p14)
>>> -> og-prod-rm (bhyve guest - 10.2-RELEASE-p14)
>>> All the og-* machines just have a running copy of tomcat.
>>> 
>>> I built a simple test to catch the error with python/fabric... the test is:
>>> 
>>> fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}'
>>> 
>>> which simply run the command "date" on all hosts.
>>> 
>>> So, the result is:
>>> @Mac-mini ~/p/f/> ./check_clock.sh
>>> Thu Apr  7 17:38:00 CEST 2016 [envrm02] out
>>> Thu Apr  7 17:38:00 CEST 2016 [og-qa] out
>>> Thu Apr  7 17:38:00 CEST 2016 [og-ci] out
>>> Thu Apr  7 17:38:00 CEST 2016 [og-test] out
>>> Thu Apr  7 17:38:00 CEST 2016 [og-prod-rm] out
>>> 
>>> @Mac-mini ~/p/f/> ./check_clock.sh
>>> Thu Apr  7 18:38:00 CEST 2016 [envrm02] out
>>> Thu Apr  7 18:45:10 CEST 2016 [og-qa] out
>>> Thu Apr  7 18:38:00 CEST 2016 [og-ci] out
>>> Thu Apr  7 18:38:00 CEST 2016 [og-test] out
>>> Thu Apr  7 18:38:00 CEST 2016 [og-prod-rm] out
>>> 
>>> @Mac-mini ~/p/f/> ./check_clock.sh
>>> Thu Apr  7 18:41:05 CEST 2016 [envrm02] out
>>> Thu Apr  7 18:48:15 CEST 2016 [og-qa] out
>>> Thu Apr  7 18:41:05 CEST 2016 [og-ci] out
>>> Thu Apr  7 18:41:05 CEST 2016 [og-test] out
>>> Thu Apr  7 18:41:05 CEST 2016 [og-prod-rm] out
>>> 
>>> @Mac-mini ~/p/f/> ./check_clock.sh
>>> Thu Apr  7 18:46:51 CEST 2016 [envrm02] out
>>> Thu Apr  7 18:46:52 CEST 2016 [og-qa] out
>>> Thu Apr  7 18:46:52 CEST 2016 [og-ci] out
>>> Thu Apr  7 18:46:53 CEST 2016 [og-test] out
>>> Thu Apr  7 18:46:53 CEST 2016 [og-prod-rm] out
>>> 
>>> The configuration is the same for all hosts and is this:
>>> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd
>>> [envrm02] out: ntpd_enable="YES"
>>> [og-qa] out: ntpd_enable="YES"
>>> [og-qa] out: ntpd_sync_on_start="YES"
>>> [og-ci] out: ntpd_enable="YES"
>>> [og-ci] out: ntpd_sync_on_start="YES"
>>> [og-test] out: ntpd_enable="YES"
>>> [og-test] out: ntpd_sync_on_start="YES"
>>> [og-prod-rm] out: ntpd_enable="YES"
>>> [og-prod-rm] out: ntpd_sync_on_start="YES"
>>> 
>>> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v 
>>> 'out: #'
>>> [envrm02] Executing task ''
>>> [envrm02] run: cat /etc/ntp.conf
>>> [envrm02] out:
>>> [envrm02] out: server 0.freebsd.pool.ntp.org iburst
>>> [envrm02] out: server 1.freebsd.pool.ntp.org iburst
>>> [envrm02] out: server 2.freebsd.pool.ntp.org iburst
>>> [envrm02] out:
>>> [envrm02] out: restrict default limited kod nomodify notrap nopeer noquery
>>> [envrm02] out: restrict -6 default limited kod nomodify notrap nopeer 
>>> noquery
>>> [envrm02] out: restrict 127.0.0.1
>>> [envrm02] out: restrict -6 ::1
>>> [envrm02] out: restrict 127.127.1.0
>>> 
>>> [og-qa] Executing task ''
>>> [og-qa] run: cat /etc

Re: Understanding Bhyve shutdown

[Neel Natu]

Hi Roman,

On Wed, Apr 13, 2016 at 3:55 AM, Roman Bogorodskiy
 wrote:
> Hi,
>
> I was trying to get better understanding of how to properly shutdown VMs
> in bhyve, but unfortunately the documentation does not provide much
> details on that.
>
> Specifically, handbook [I] suggests to reboot a machine and then run
> bhyvectl --destroy on it.
>
> The bhyvectl(8) manpage mentions the '--force-reset' and
> '--force-poweroff' switches, but does not give details on those.
>
> I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
>
> I tried all the options I know and wrote down the results. I also have
> some questions, hopefully you'll be able to answer some of them.
>
> 1. bhyvectl --vm=$name --destroy
>
>  * looks like hard poweroff in the guest
>  * the corresponding bhyve(8) process goes away
>  * /dev/vmm/ entry goes away
>
> In my experience, it's a dangerous way to shutdown a VM because
> sometimes it appears it damages the image and VM fails to boot with
> something like this:
>
> ---
> Starting devd.
> mode = 0100600, inum = 170269, fs = /
> panic: ffs_valloc: dup alloc
> cpuid = 0
> KDB: stack backtrace:
> #0 0x80984e30 at kdb_backtrace+0x60
> #1 0x809489e6 at vpanic+0x126
> #2 0x809488b3 at panic+0x43
> #3 0x80b74a6e at ffs_valloc+0x84e
> #4 0x80bb60ad at ufs_makeinode+0x7d
> #5 0x80bb24fd at ufs_create+0x2d
> #6 0x80e71841 at VOP_CREATE_APV+0xa1
> #7 0x809cd9e6 at uipc_bindat+0x346
> #8 0x809c5488 at kern_bindat+0x108
> #9 0x809c52a7 at sys_bind+0x77
> #10 0x80d4b3f7 at amd64_syscall+0x357
> #11 0x80d30adb at Xfast_syscall+0xfb
> Uptime: 3s
>
> Dump failed. Partition too small.
> ---
>

Yup, this is biggest hammer you could use to shutdown a virtual
machine. As you noticed, this is usually a bad thing because it does
not give the guest OS an opportunity to shutdown cleanly.

> 2. kill -SIGTERM $bhyve_pid
>
> If guest supports ACPI shutdown:
>
>  * guest shuts down cleanly
>  * the corresponding bhyve(8) process terminates
>  * /dev/vmm entry is still here, need bhyvectl --destroy for complete
>cleanup
>
> If guest does not support ACPI shutdown (such as doing sysctl
> hw.acpi.power_button_state=NONE):
>
>  * Nothing happens
>
>  Q1: Is there a way to know if a guest reacted to power button but
>  waiting for the bhyve process to terminate?

Not really, except to wait for some amount of time to give the guest a
chance to shutdown.

>  Q2: Why it's not done via bhyvectl (it seems that it's easier for users
>  + don't have to overload a useful SIGTERM signal)
>

It seems natural to overload SIGTERM to do this. For e.g. when the
host is shutting down it will send a SIGTERM to all running processes
and translating this into an ACPI poweroff event for the guest allows
it to shutdown cleanly.

> 3. bhyvectl --vm=$name --force-poweroff
>
>  * looks like hard poweroff in the guest
>  * the corresponding bhyve(8) process goes away
>  * /dev/vmm entry is still here, need bhyvectl --destroy for complete
>cleanup
>
> Q: what's the practical difference with just doing --destroy right away?
>

'force-poweroff' is equivalent to a hard power off on real hardware.

The only practical difference between '--force-poweroff' and
'--destroy' is that destroy will also remove the device node in
/dev/vmm and release any memory allocated for the guest back to the
host.

> 4. bhyvectl --vm=$name --force-reset
>
> Looks very similar to item #3 with just different exit code (reboot
> appears to be using 0, while shutdown and halt use 1 and 2).
>
> Q: what's the practical use of it?
>

The exit code can be used by scripts on top of 'bhyve' to decide
whether to restart execution of the guest (reset) or stop completely
(poweroff).

> Would greatly appreciate if somebody could provide more details on that.
> I guess we'll need to update Handbook with this information as well
> because it needs to mention SIGTERM for ACPI shutdown at least.
>

Hope that helps.

best
Neel

> Roman Bogorodskiy
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve clock issue

[Neel Natu]

Hi,

Did you see anything interesting in the ntpd log file around the time the large 
skew happened?

Best
Neel

> On Apr 13, 2016, at 3:44 AM, Ilario Febi  wrote:
> 
> Hi All
> 
> I have a strange issue about the clock of some FreeBSD Bhyve guests,
> maybe someone of you can help me to diagnose it and work on a
> solution.
> 
> In detail, sometimes (without an apparent regularity) one of four
> bhyve guest's clock goes exactly 7 minutes and 10 seconds in the
> future.
> 
> This clock stays unaligned for about 20 minutes, then, without any
> human intervention, the clock realign to proper condition.
> 
> My configuration is:
> 
> envrm02 (bhyve HOST, 10.2-RELEASE-p14)
> -> og-qa (bhyve guest - 10.2-RELEASE-p14)
> -> og-ci (bhyve guest - 10.2-RELEASE-p14)
> -> og-test (bhyve guest - 10.2-RELEASE-p14)
> -> og-prod-rm (bhyve guest - 10.2-RELEASE-p14)
> All the og-* machines just have a running copy of tomcat.
> 
> I built a simple test to catch the error with python/fabric... the test is:
> 
> fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}'
> 
> which simply run the command "date" on all hosts.
> 
> So, the result is:
> @Mac-mini ~/p/f/> ./check_clock.sh
> Thu Apr  7 17:38:00 CEST 2016 [envrm02] out
> Thu Apr  7 17:38:00 CEST 2016 [og-qa] out
> Thu Apr  7 17:38:00 CEST 2016 [og-ci] out
> Thu Apr  7 17:38:00 CEST 2016 [og-test] out
> Thu Apr  7 17:38:00 CEST 2016 [og-prod-rm] out
> 
> @Mac-mini ~/p/f/> ./check_clock.sh
> Thu Apr  7 18:38:00 CEST 2016 [envrm02] out
> Thu Apr  7 18:45:10 CEST 2016 [og-qa] out
> Thu Apr  7 18:38:00 CEST 2016 [og-ci] out
> Thu Apr  7 18:38:00 CEST 2016 [og-test] out
> Thu Apr  7 18:38:00 CEST 2016 [og-prod-rm] out
> 
> @Mac-mini ~/p/f/> ./check_clock.sh
> Thu Apr  7 18:41:05 CEST 2016 [envrm02] out
> Thu Apr  7 18:48:15 CEST 2016 [og-qa] out
> Thu Apr  7 18:41:05 CEST 2016 [og-ci] out
> Thu Apr  7 18:41:05 CEST 2016 [og-test] out
> Thu Apr  7 18:41:05 CEST 2016 [og-prod-rm] out
> 
> @Mac-mini ~/p/f/> ./check_clock.sh
> Thu Apr  7 18:46:51 CEST 2016 [envrm02] out
> Thu Apr  7 18:46:52 CEST 2016 [og-qa] out
> Thu Apr  7 18:46:52 CEST 2016 [og-ci] out
> Thu Apr  7 18:46:53 CEST 2016 [og-test] out
> Thu Apr  7 18:46:53 CEST 2016 [og-prod-rm] out
> 
> The configuration is the same for all hosts and is this:
> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd
> [envrm02] out: ntpd_enable="YES"
> [og-qa] out: ntpd_enable="YES"
> [og-qa] out: ntpd_sync_on_start="YES"
> [og-ci] out: ntpd_enable="YES"
> [og-ci] out: ntpd_sync_on_start="YES"
> [og-test] out: ntpd_enable="YES"
> [og-test] out: ntpd_sync_on_start="YES"
> [og-prod-rm] out: ntpd_enable="YES"
> [og-prod-rm] out: ntpd_sync_on_start="YES"
> 
> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v 'out: 
> #'
> [envrm02] Executing task ''
> [envrm02] run: cat /etc/ntp.conf
> [envrm02] out:
> [envrm02] out: server 0.freebsd.pool.ntp.org iburst
> [envrm02] out: server 1.freebsd.pool.ntp.org iburst
> [envrm02] out: server 2.freebsd.pool.ntp.org iburst
> [envrm02] out:
> [envrm02] out: restrict default limited kod nomodify notrap nopeer noquery
> [envrm02] out: restrict -6 default limited kod nomodify notrap nopeer noquery
> [envrm02] out: restrict 127.0.0.1
> [envrm02] out: restrict -6 ::1
> [envrm02] out: restrict 127.127.1.0
> 
> [og-qa] Executing task ''
> [og-qa] run: cat /etc/ntp.conf
> [og-qa] out:
> [og-qa] out: server 0.freebsd.pool.ntp.org iburst
> [og-qa] out: server 1.freebsd.pool.ntp.org iburst
> [og-qa] out: server 2.freebsd.pool.ntp.org iburst
> [og-qa] out:
> [og-qa] out: restrict default limited kod nomodify notrap nopeer noquery
> [og-qa] out: restrict -6 default limited kod nomodify notrap nopeer noquery
> [og-qa] out: restrict 127.0.0.1
> [og-qa] out: restrict -6 ::1
> [og-qa] out: restrict 127.127.1.0
> 
> [og-ci] Executing task ''
> [og-ci] run: cat /etc/ntp.conf
> [og-ci] out:
> [og-ci] out: server 0.freebsd.pool.ntp.org iburst
> [og-ci] out: server 1.freebsd.pool.ntp.org iburst
> [og-ci] out: server 2.freebsd.pool.ntp.org iburst
> [og-ci] out:
> [og-ci] out: restrict default limited kod nomodify notrap nopeer noquery
> [og-ci] out: restrict -6 default limited kod nomodify notrap nopeer noquery
> [og-ci] out: restrict 127.0.0.1
> [og-ci] out: restrict -6 ::1
> [og-ci] out: restrict 127.127.1.0
> 
> [og-test] Executing task ''
> [og-test] run: cat /etc/ntp.conf
> [og-test] out:
> [og-test] out: server 0.freebsd.pool.ntp.org iburst
> [og-test] out: server 1.freebsd.pool.ntp.org iburst
> [og-test] out: server 2.freebsd.pool.ntp.org iburst
> [og-test] out:
> [og-test] out: restrict default limited kod nomodify notrap nopeer noquery
> [og-test] out: restrict -6 default limited kod nomodify notrap nopeer noquery
> [og-test] out: restrict 127.0.0.1
> [og-test] out: restrict -6 ::1
> [og-test] out: restrict 127.127.1.0
> 
> Some observation:
> * The clock of envrm02 (the HOST) always seems to be correct.
> * The issue happens on a (apparently) 

Re: -current host, 10.1 client loops

[Neel Natu]

On Thu, Mar 17, 2016 at 12:38 AM, Poul-Henning Kamp  wrote:
> 
> In message 
> , 
> =?UTF-8?Q?Olivier_Cochard=2D
> Labb=C3=A9?= writes:
>
>
>>> How do I tell that script to use ahci-hd ?
>>>
>>>
>>You just need to replace virtio-blk by ahci-hd in the script, like:
>
> Ok, it also hung with ahci-hd and two disks...
>
> Just to make sure I have asked about this:  Are there any special
> params needed to run a 10.1-R guest ?
>

https://www.freebsd.org/releases/10.1R/errata.html#open-issues

Can you try to reproduce after setting 'vfs.unmapped_buf_allowed=0' at
the guest loader prompt?

The issue doesn't mention bhyve specifically but given that it's a
simple workaround it should be worth a try.

best
Neel

> --
> Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
> p...@freebsd.org | TCP/IP since RFC 956
> FreeBSD committer   | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: -current host, 10.1 client loops

[Neel Natu]

Hi,

On Tue, Mar 15, 2016 at 7:36 AM, Poul-Henning Kamp  wrote:
> I'm seing bhyve go into some kind of endless loop while trying to
> compile the gcc port on 10.1 as guest.
>
> In one case CTRL-T on the console kept working, but showed an rm(1)
> process raking up CPU time.
>
> Are there known bogon in current/bhyve or using 10.1-R/i386 as a guest
> I have not spotted ?
>
> How does one debug stuff like this ?
>

I usually do the following things to get a sense for what might be going on:

- "top -H" on the host to figure out if guest vcpu threads are spinning
- "top -H" inside the guest if possible
- "bhyvectl --vm vmname --cpu vcpuid --get-stats" on the host

If there is nothing obvious from the above then I will recompile the
host kernel with KTR enabled (vmm.ko has detailed tracing at KTR_GEN).
This is usually very helpful to understand what might be going on.

Also, if it is possible to reproduce with a single vcpu then it will
help when analyzing the output of ktrdump.

best
Neel

> Poul-Henning
>
> Bhyve started with:
>
> sh /usr/share/examples/bhyve/vmrun.sh \
> -m 1G \
> -t ${VMN} \
> -d ${P}.root.dd \
> -d ${P}.swap.dd \
> -d ${P}.tami_install.dd \
> -d ${P}.tami_git.dd \
> vm${VMU} || true
>
> Host:
> 11.0-CURRENT #4 r296808: Sun Mar 13 22:39:59 UTC 2016
>
> CPU: AMD Athlon(tm) II X3 455 Processor (3311.18-MHz K8-class CPU)
>   Origin="AuthenticAMD"  Id=0x100f53  Family=0x10  Model=0x5  
> Stepping=3
>   
> Features=0x178bfbff
>   Features2=0x802009
>   AMD 
> Features=0xee500800
>   AMD 
> Features2=0x837ff
>   SVM: NP,NRIP,NAsids=64
>   TSC: P-state invariant
> real memory  = 17179869184 (16384 MB)
> avail memory = 16573935616 (15806 MB)
> Event timer "LAPIC" quality 400
> ACPI APIC Table: <090712 APIC1033>
> FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
> FreeBSD/SMP: 1 package(s) x 3 core(s)
>  cpu0 (BSP): APIC ID:  0
>  cpu1 (AP): APIC ID:  1
>  cpu2 (AP): APIC ID:  2
> ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 
> 64/32 (20150818/tbfadt-649)
>
>
> Guest:
> 10.1-RELEASE i386
>
>
>
>
> --
> Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
> p...@freebsd.org | TCP/IP since RFC 956
> FreeBSD committer   | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: -current host, 10.1 client loops

[Neel Natu]

Hi,

On Tue, Mar 15, 2016 at 2:30 PM, Poul-Henning Kamp <p...@phk.freebsd.dk> wrote:
> 
> In message 
> <cafgre9hqyq9tsm8pnzcazq2awkenstaelcn0vsyhg4wzk4f...@mail.gmail.com>
> , Neel Natu writes:
>
>>Also, if it is possible to reproduce with a single vcpu then it will
>>help when analyzing the output of ktrdump.
>
> Well, on the gutfeeling that it might help I gave a '-c 1' and that
> seems to prevent the problem from happening so I could get my builds
> done.
>
> So some kind of synchronization issue maybe ?
>

Yes, that might be possible.

A couple of suggestions to narrow it down further:
- try to reproduce with 2 or 4 vcpus (3 cpus is probably not a well
tested configuration)
- try to reproduce with the ahci-hd device emulation instead of virtio-blk.

If this problem is reproducible then I am happy to work with you to
get to the bottom of this.

best
Neel

> --
> Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
> p...@freebsd.org | TCP/IP since RFC 956
> FreeBSD committer   | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ath0 (wlan0) not working in virtulised guest with bhyve

[Neel Natu]

On Sun, Feb 28, 2016 at 7:17 AM, richard bader
 wrote:
> Hey Neel,
>>
>> I suspect that the ath0 controller does not support MSI/MSI-x
>> interrupts. The output of 'pciconf -lvbc' on the host will confirm if
>> this is the case.
>>
> thanks for the hint. The controller should support MSI interrupts.
>
> root@test10:/usr/src/sys # pciconf -lvbc
>
> ath0@pci0:5:0:0:class=0x028000 card=0x30a4168c chip=0x002e168c
> rev=0x01 hdr=0x00
> vendor = 'Atheros Communications Inc.'
> device = 'AR9287 Wireless Network Adapter (PCI-Express)'
> class  = network
> bar   [10] = type Memory, range 64, base 0xf710, size 65536, enabled
> cap 01[40] = powerspec 3  supports D0 D1 D3  current D0
> cap 05[50] = MSI supports 1 message
> cap 10[60] = PCI-Express 2 legacy endpoint max data 128(128) link x1(x1)
>  speed 2.5(2.5) ASPM L0s/L1(L0s/L1)
> ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
> ecap 0002[140] = VC 1 max VC0
> ecap 0003[160] = Serial 1 001517241412
> ecap 0004[170] = Power Budgeting 1
>
>
> But as I read in https://wiki.freebsd.org/dev/ath_hal%284%29/AR9300, MSI
> interrupts are not supported before AR9300 HAL.
>
> So i'll buy an other card and try again.
>
> But it would be a good idea to add the check for supported MSI/MSI-x
> interrupts to the https://wiki.freebsd.org/bhyve/pci_passthru .
>

Done, thanks for the suggestion.

best
Neel

> thanks
>
> richard
>
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: ath0 (wlan0) not working in virtulised guest with bhyve

[Neel Natu]

Hi Richard,

On Sat, Feb 27, 2016 at 2:01 AM, richard bader
 wrote:
> Hello,
>
> using ath0 (and wlan) on the host is working fine. But trying the same on
> the guest does not work.
>
> on the host freebsd 10.2 ist running
> on the guest freebsd 10.3 (10.3-BETA2 FreeBSD 10.3-BETA2 #0) is running
>
> the guest was started with:
> /usr/sbin/bhyve -c 2 -m 2048M  -A -H -P -g 0  -s 0:0,hostbridge -s
> 1:0,lpc  -s 2:0,virtio-net,tap0  -s 3:0,virtio-net,tap1  -s
> 4:0,virtio-blk,wlan.bader.loc.img  -s 5,passthru,5/0/0  -l com1,stdio   wlan
>
> As simple experiment I try to scan for wlans using:
>root@wlan:~ #ifconfig wlan0 create wlandev ath0 && ifconfig wlan0 up scan
> but no wlan appears.
>
>
> As adrian chadd (freebsd-wirel...@freebsd.org) noticed, the irq16 is not
> working:
>
> root@wlan:~ # vmstat -i
> interrupt  total   rate
> irq4: uart0  224  0
> cpu0:timer 10956 25
> irq265: virtio_pci0  295  0
> irq271: virtio_pci2 1267  2
> cpu1:timer  2982  7
> Total  15724 36
>
>
> root@wlan:~ # dmesg | grep irq
> ioapic0  irqs 0-23 on motherboard
> atrtc0:  port 0x70-0x71 irq 8 on acpi0
> attimer0:  port 0x40-0x43 irq 0 on acpi0
> virtio_pci0:  port 0x2000-0x201f mem
> 0xc000-0xc0001fff irq 16 at device 2.0 on pci0
> virtio_pci1:  port 0x2020-0x203f mem
> 0xc0002000-0xc0003fff irq 17 at device 3.0 on pci0
> virtio_pci2:  port 0x2040-0x207f mem
> 0xc0004000-0xc0005fff irq 18 at device 4.0 on pci0
> ath0:  mem 0xc001-0xc001 irq 16 at device 7.0 on pci0
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
> atkbd0:  irq 1 on atkbdc0
>
>
>
> the output of 'ifconfig wlan0' is the same as on the working host:
>
>
> root@wlan:~ # dmesg | grep ath0
> ath0:  mem 0xc001-0xc001 irq 16 at device 5.0 on pci0
> ath0: [HT] enabling HT modes
> ath0: [HT] enabling short-GI in 20MHz mode
> ath0: [HT] 1 stream STBC receive enabled
> ath0: [HT] 1 stream STBC transmit enabled
> ath0: [HT] 2 RX streams; 2 TX streams
> ath0: AR9287 mac 384.2 RF5133 phy 15.15
> ath0: 2GHz radio: 0x; 5GHz radio: 0x00c0
>
>
>
> any hints to get it working?
>

I suspect that the ath0 controller does not support MSI/MSI-x
interrupts. The output of 'pciconf -lvbc' on the host will confirm if
this is the case.

best
Neel

> thanks
>
> richard
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: bhyve with Linux guest, how to safely handle updates?

[Neel Natu]

Hi Dean,

On Thu, Jan 28, 2016 at 10:55 AM, dweimer  wrote:
> On 2016-01-26 8:13 pm, Sergey Manucharian wrote:
>>
>> Excerpts from dweimer's message from Tue 26-Jan-16 19:07:
>>>
>>>
>>> Is there anything that normally needs to be done after a Linux kernel
>>> update to refresh the grub2-bhyve setup?
>>
>>
>> The kernel update should not have any effect since grub-bhyve uses the
>> virtual disk mapping file, which should point to your linux drive.
>>
>> I'm using the following command:
>>
>> $ sudo grub-bhyve -m /path/to/device.map -r hd0,msdos1 -M 1024M debian
>>
>> where "device.map" contains the following:
>>
>> (hd0) /dev/zvol/zroot/linuxdisk1
>> (cd0) /stuff/vm/bhyve/debian/debian-testing-amd64-2015-11-30.iso
>>
>> "hd0" can be a real disk device, e.g. /dev/sda, or an image file (in
>> my case it's a ZFS volume).
>>
>> How do you use that VM in VBox? If it's a .vdi file, bhyve will not be
>> able to recognize it. You should use a raw HDD image file. To make it
>> compatible with VBox you can create a .vmdk file pointing to that raw
>> image.
>>
>> --
>> Sergey
>
>
> I am back to testing again, copied my ZFS Boot Environment over to a VMware
> virtual machine, renamed it and changed IPs, removed the virtual box stuff,
> and enabled bhyve.
>
> I did some searching and found out that I was using
> https://github.com/churchers/vm-bhyve to manage the bhyve virtual machines
> starting and stopping. Sticking with zvol for disk backing, I know its less
> portable.
>
> I have been able to install a couple of debian virtual machines and play
> around with them. So far I have been unable to duplicate the issue I had
> before. My current issue which maybe related to running inside a VMware
> virtual machine. Is the Linux hwclock and system clock sync issues. If I
> power off the vm and reboot it it believes that the disk was modified in the
> future and appears to hang. Its actually doing a fsck I just don't see
> status if you wait long enough it finally does come up.
>
> Has anyone else ran into this issue? I have actually ran the hwclock
> -systohc --utc prior to powering down and still had the issue. Tried
> changing the hwclock to system time by excluding the --utc from the command
> no change. Incidentally whether I use the --utc or not the hwclock --show
> always displays the local time. I couldn't seem to find any documentation on
> bhyve whether or not I should tell the guests that the hwclock is in utc or
> local time.
>

The "-u" option of bhyve(8) will configure the RTC to present UTC time
to the guest (default is localtime).

best
Neel

> --
> Thanks,
>Dean E. Weimer
>http://www.dweimer.net/
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Bhyve UEFI VM on non-UEFI host

[Neel Natu]

> On Dec 21, 2015, at 10:33 AM, Evan Rowley  wrote:
> 
> Hello
> 
> I am interested in running bhyve VMs utilizing UEFI, as described in
> Peter's Windows guide: http://pr1ntf.xyz/windowsunderbhyve.html
> 
> The host machine will be an dual AMD Opteron 6166 HE processor, AMD SR5670
> / SP5100 chipset, Supermicro H8DGU-F motherboard. There is no UEFI support
> on this motherboard and the BIOS is standard AMI like the other Supermicro
> boards from this era.
> 
> Despite not having native UEFI support on the motherboard, will I still be
> able to use virtualized UEFI via bhyve, for booting Windows VMs?
> 

Yes, that's correct.

Best
Neel

> 
> 
> -- 
> - EJR
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: BAR and pci passthrough

[Neel Natu]

On Wed, Dec 2, 2015 at 3:13 AM, G. Amanakis via freebsd-virtualization
 wrote:
> I am facing the following problem:
> on a X9SCM with an E3-1220Lv2 I am trying to passthrough the onboard usb 
> controllers. I disable the usb module from the kernel config and using pptdev 
> the controllers are assigned to ppt devices. However running bhyve on FreeBSD 
> 10.2 with:
>
> sudo bhyve -AI -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 
> -s 3:0,virtio-blk,./linux.img -s 4:0,passthru,0/29/0 -l com1,stdio -c 2 -m 
> 2048M linuxguest
>
> returns :
>
> passthru device 0/29/0 BAR 0: base 0xdf823000 or size 0x400 not page 
> aligned
>
> The problem probably that the length of the bar is smaller and not aligned 
> with the pagesize. Could the length of the BAR be modified in order to 
> perform the pci passthrough?

Yes, that's correct - the size of the BAR is not a multiple of the
page size which leads to the error. If this BAR is mapped into the
guest's address space then it will "leak" an additional 3K into the
guest (since the minimum nested mapping is 4KB in size).

It is hard to fix this in the general case if you want to truly
passthrough the BAR to the guest.

best
Neel

> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to 
> "freebsd-virtualization-unsubscr...@freebsd.org"
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: passthru requires guest memory to be wired

[Neel Natu]

Hi Craig,

On Tue, Aug 25, 2015 at 11:53 AM, Craig Rodrigues rodr...@freebsd.org wrote:
 Hi,

 I updated one of my FreeBSD boxes to latest CURRENT, and when I started
 a bhyve VM which uses PCI passthru, I got this error:

 passthru requires guest memory to be wired

 According to this commit:
 https://reviews.freebsd.org/rS284539

 '-S' needs to be passed to bhyveload *and* bhyve if PCI passthru is used.

 It looks like this change did not make it to 10.2R.  Will this change go
 into stable/10?


No, there are no plans to MFC the change.

 Should this info be added to: https://wiki.freebsd.org/bhyve/pci_passthru


Yup, added it just now. Thanks for pointing it out.

best
Neel

 --
 Craig
 ___
 freebsd-virtualization@freebsd.org mailing list
 https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: virtio disk device file /dev/hast/shared does not exist. Creating it ...

[Neel Natu]

Hi Nikos,

On Fri, Jul 31, 2015 at 6:53 AM, Nikos Vassiliadis nv...@gmx.com wrote:
 Hi,

 While trying to use a HAST device as my storage I got this message.

 Booting...
 virtio disk device file /dev/hast/shared does not exist.
 Creating it ...
 GDB: no debug ports present
 KDB: debugger backends: ddb
 KDB: current backend: ddb
 Copyright (c) 1992-2015 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994


 Does anybody know what creating it mean in this regard?


I assume you are using vmrun.sh to start the virtual machine.

If the backing file for the virtio disk device does not exist then the
script will create it before launching the virtual machine.

See 'make_and_check_diskdev()' function in vmrun.sh.

best
Neel

 Thank in advance,
 Nikos

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: centos 7.1 with multiple virtual processors

[Neel Natu]

Hi Andriy,

On Tue, Jul 14, 2015 at 6:55 AM, Andriy Gapon a...@freebsd.org wrote:
 On 23/06/2015 11:05, Andriy Gapon wrote:
 On 23/06/2015 10:26, Neel Natu wrote:
 [snip]
 Does this ever happen with a single vcpu guest?

 Never seen the problem with a single CPU so far.
 Also, never had that problem with FreeBSD guests.

 The other mystery is the NMIs the host is receiving. I (re)verified to
 make sure that bhyve/vmm.ko do not assert NMIs so it has to be
 something else on the host that's doing it ...

 But the correlation with the multi-CPU non-FreeBSD guests seems to be 
 significant.

 Today I've got another NMI with a Linux guest, but a few things were 
 different:
 - the host was fresh after a reboot
 - the guest was single CPU
 - the NMI happened during boot up of the very first VM

 kernel: NMI ISA 2c, EISA ff
 kernel: NMI ... going to debugger

 Not what causes those NMIs but when they happen they always happen during a
 bhyve guest boot...


Could you update the host with the following patch?
https://people.freebsd.org/~neel/patches/ktr_stray_nmi.patch

It enables KTR_GEN logging by default and turns it off when a VM-exit
due to NMI is detected. If there is a pattern to the NMIs then it can
be correlated with the guest activity around that time.

best
Neel

 --
 Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve vs signals

[Neel Natu]

Hi Andriy,

On Tue, Jul 14, 2015 at 1:36 AM, Andriy Gapon a...@freebsd.org wrote:

 I couldn't find any information in bhyve(8) on how bhyve responds to various
 signals.  It seems that at least SIGTERM is intercepted and translated to an
 ACPI power event.  How about other signals?  What's the best signal to use to
 kill a hung VM that can not be stopped gracefully?


'bhyvectl --force-poweroff --vm=vmname' is a good way to forcibly kill
a wedged VM.

best
Neel

 --
 Andriy Gapon
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: centos 7.1 with multiple virtual processors

[Neel Natu]

Hi Andriy,

On Mon, Jun 22, 2015 at 11:45 PM, Andriy Gapon a...@freebsd.org wrote:
 On 23/06/2015 05:37, Neel Natu wrote:
 Hi Andriy,

 FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus
 fine on my host with 8 physical cores.

 I have some questions about your setup inline.

 On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon a...@freebsd.org wrote:

 If I run a CentOS 7.1 VM with more than one CPU more often than not it would
 hang on startup and bhyve would start spinning.

 The following are the last messages seen in the VM:

 Switching to clocksource hpet
 [ cut here ]
 WARNING: at kernel/time/clockevents.c:239 
 clockevents_program_event+0xdb/0xf0()
 Modules linked in:
 CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1
 Hardware name:   BHYVE, BIOS 1.00 03/14/2014
   cab5bdb6 88003fc03e08 81604eaa
  88003fc03e40 8106e34b 800f423f 800f423f
  81915440   88003fc03e50
 Call Trace:
  IRQ  [81604eaa] dump_stack+0x19/0x1b
  [8106e34b] warn_slowpath_common+0x6b/0xb0
  [8106e49a] warn_slowpath_null+0x1a/0x20
  [810ce6eb] clockevents_program_event+0xdb/0xf0
  [810cf211] tick_handle_periodic_broadcast+0x41/0x50
  [81016525] timer_interrupt+0x15/0x20
  [8110b5ee] handle_irq_event_percpu+0x3e/0x1e0
  [8110b7cd] handle_irq_event+0x3d/0x60
  [8110e467] handle_edge_irq+0x77/0x130
  [81015cff] handle_irq+0xbf/0x150
  [81077df7] ? irq_enter+0x17/0xa0
  [816172af] do_IRQ+0x4f/0xf0
  [8160c4ad] common_interrupt+0x6d/0x6d
  EOI  [8126e359] ? selinux_inode_alloc_security+0x59/0xa0
  [811de58f] ? __d_instantiate+0xbf/0x100
  [811de56f] ? __d_instantiate+0x9f/0x100
  [811de60d] d_instantiate+0x3d/0x70
  [8124d748] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130
  [8124da82] __create_file+0x1c2/0x2c0
  [81a6c6bf] ? set_graph_function+0x1f/0x1f
  [8124dbcb] debugfs_create_dir+0x1b/0x20
  [8112c1ce] tracing_init_dentry_tr+0x7e/0x90
  [8112c250] tracing_init_dentry+0x10/0x20
  [81a6c6d2] ftrace_init_debugfs+0x13/0x1fd
  [81a6c6bf] ? set_graph_function+0x1f/0x1f
  [810020e8] do_one_initcall+0xb8/0x230
  [81a45203] kernel_init_freeable+0x18b/0x22a
  [81a449db] ? initcall_blacklist+0xb0/0xb0
  [815f33f0] ? rest_init+0x80/0x80
  [815f33fe] kernel_init+0xe/0xf0
  [81614d3c] ret_from_fork+0x7c/0xb0
  [815f33f0] ? rest_init+0x80/0x80
 ---[ end trace d5caa1cab8e7e98d ]---


 A few questions to narrow this down:
 - Is the host very busy when the VM is started (or what is the host
 doing when this happened)?

 The host typically is not heavily loaded.  There is X server running and some
 applications.  I'd imagine that those could cause some additional latency but
 not CPU starvation.


Yup, I agree.

Does this ever happen with a single vcpu guest?

The other mystery is the NMIs the host is receiving. I (re)verified to
make sure that bhyve/vmm.ko do not assert NMIs so it has to be
something else on the host that's doing it ...

best
Neel

 - How many vcpus are you giving to the VM?
 - How many cores on the host?

 I tried only 2 / 2.


 At the same time sometimes there is one or more of spurious NMIs on the 
 _host_
 system:
 NMI ISA c, EISA ff
 NMI ... going to debugger


 Hmm, that's interesting. Are you using hwpmc to do instruction sampling?

 hwpmc driver is in the kernel, but it was not used.

 bhyve seems to spin here:
 vmm.ko`svm_vmrun+0x894
 vmm.ko`vm_run+0xbb7
 vmm.ko`vmmdev_ioctl+0x5a4
 kernel`devfs_ioctl_f+0x13b
 kernel`kern_ioctl+0x1e1
 kernel`sys_ioctl+0x16a
 kernel`amd64_syscall+0x3ca
 kernel`0x8088997b

 (kgdb) list *svm_vmrun+0x894
 0x813c9194 is in svm_vmrun
 (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895).
 1890
 1891static __inline void
 1892enable_gintr(void)
 1893{
 1894
 1895__asm __volatile(stgi);
 1896}
 1897
 1898/*
 1899 * Start vcpu with specified RIP.


 Yeah, that's not surprising because host interrupts are blocked when
 the cpu is executing in guest context. The 'enable_gintr()' enables
 interrupts so it gets blamed by the interrupt-based sampling.

 In this case it just means that the cpu was in guest context when a
 host-interrupt fired.

 I see.  FWIW, that was captured with DTrace.

 --
 Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: dtrace fbt vs vmm

[Neel Natu]

Hi Andriy,

On Mon, Jun 22, 2015 at 9:17 AM, Andriy Gapon a...@freebsd.org wrote:
 On 22/06/2015 19:08, Peter Grehan wrote:
 Hi Andriy,

 I haven't investigated this issue further, but if there are any unsafe 
 functions
 in vmm it would be nice to blacklist them in the fbt code.
 Perhaps blacklisting everything from vmm module could be a stopgap solution.

  Intel or AMD ? We've been able to run dtrace on Intel; less (to no) 
 coverage on
 AMD so there's quite possibly bugs there.

 Peter,

 it's AMD again.
 Thanks!


This was an issue with 'svm_launch()' FBT tracing and is fixed in r284712.

Details are in the commit message:
https://svnweb.freebsd.org/base?view=revisionrevision=284712

best
Neel

 --
 Andriy Gapon
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: centos 7.1 with multiple virtual processors

[Neel Natu]

Hi Andriy,

FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus
fine on my host with 8 physical cores.

I have some questions about your setup inline.

On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon a...@freebsd.org wrote:

 If I run a CentOS 7.1 VM with more than one CPU more often than not it would
 hang on startup and bhyve would start spinning.

 The following are the last messages seen in the VM:

 Switching to clocksource hpet
 [ cut here ]
 WARNING: at kernel/time/clockevents.c:239 
 clockevents_program_event+0xdb/0xf0()
 Modules linked in:
 CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1
 Hardware name:   BHYVE, BIOS 1.00 03/14/2014
   cab5bdb6 88003fc03e08 81604eaa
  88003fc03e40 8106e34b 800f423f 800f423f
  81915440   88003fc03e50
 Call Trace:
  IRQ  [81604eaa] dump_stack+0x19/0x1b
  [8106e34b] warn_slowpath_common+0x6b/0xb0
  [8106e49a] warn_slowpath_null+0x1a/0x20
  [810ce6eb] clockevents_program_event+0xdb/0xf0
  [810cf211] tick_handle_periodic_broadcast+0x41/0x50
  [81016525] timer_interrupt+0x15/0x20
  [8110b5ee] handle_irq_event_percpu+0x3e/0x1e0
  [8110b7cd] handle_irq_event+0x3d/0x60
  [8110e467] handle_edge_irq+0x77/0x130
  [81015cff] handle_irq+0xbf/0x150
  [81077df7] ? irq_enter+0x17/0xa0
  [816172af] do_IRQ+0x4f/0xf0
  [8160c4ad] common_interrupt+0x6d/0x6d
  EOI  [8126e359] ? selinux_inode_alloc_security+0x59/0xa0
  [811de58f] ? __d_instantiate+0xbf/0x100
  [811de56f] ? __d_instantiate+0x9f/0x100
  [811de60d] d_instantiate+0x3d/0x70
  [8124d748] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130
  [8124da82] __create_file+0x1c2/0x2c0
  [81a6c6bf] ? set_graph_function+0x1f/0x1f
  [8124dbcb] debugfs_create_dir+0x1b/0x20
  [8112c1ce] tracing_init_dentry_tr+0x7e/0x90
  [8112c250] tracing_init_dentry+0x10/0x20
  [81a6c6d2] ftrace_init_debugfs+0x13/0x1fd
  [81a6c6bf] ? set_graph_function+0x1f/0x1f
  [810020e8] do_one_initcall+0xb8/0x230
  [81a45203] kernel_init_freeable+0x18b/0x22a
  [81a449db] ? initcall_blacklist+0xb0/0xb0
  [815f33f0] ? rest_init+0x80/0x80
  [815f33fe] kernel_init+0xe/0xf0
  [81614d3c] ret_from_fork+0x7c/0xb0
  [815f33f0] ? rest_init+0x80/0x80
 ---[ end trace d5caa1cab8e7e98d ]---


A few questions to narrow this down:
- Is the host very busy when the VM is started (or what is the host
doing when this happened)?
- How many vcpus are you giving to the VM?
- How many cores on the host?


 At the same time sometimes there is one or more of spurious NMIs on the _host_
 system:
 NMI ISA c, EISA ff
 NMI ... going to debugger


Hmm, that's interesting. Are you using hwpmc to do instruction sampling?

 bhyve seems to spin here:
 vmm.ko`svm_vmrun+0x894
 vmm.ko`vm_run+0xbb7
 vmm.ko`vmmdev_ioctl+0x5a4
 kernel`devfs_ioctl_f+0x13b
 kernel`kern_ioctl+0x1e1
 kernel`sys_ioctl+0x16a
 kernel`amd64_syscall+0x3ca
 kernel`0x8088997b

 (kgdb) list *svm_vmrun+0x894
 0x813c9194 is in svm_vmrun
 (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895).
 1890
 1891static __inline void
 1892enable_gintr(void)
 1893{
 1894
 1895__asm __volatile(stgi);
 1896}
 1897
 1898/*
 1899 * Start vcpu with specified RIP.


Yeah, that's not surprising because host interrupts are blocked when
the cpu is executing in guest context. The 'enable_gintr()' enables
interrupts so it gets blamed by the interrupt-based sampling.

In this case it just means that the cpu was in guest context when a
host-interrupt fired.

best
Neel

 --
 Andriy Gapon
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: How to bind a thread to a CPU?

[Neel Natu]

Hi Stefan,

 On Jun 13, 2015, at 12:12 PM, Stefan Andritoiu stefan.andrit...@gmail.com 
 wrote:
 
 Hi Neel,
 
 On Sat, Jun 13, 2015 at 9:00 PM, Neel Natu neeln...@gmail.com wrote:
 Hi Stefan,
 
 On Sat, Jun 13, 2015 at 3:33 AM, Stefan Andritoiu
 stefan.andrit...@gmail.com wrote:
 Hi,
 
 How can I pin a thread to run only on a specific CPU?
 Is it enough just to set the struct cpuset *td_cpuset of the thread?
 
 sched_bind() is the proper way to do this.
 
 But in sched_bind() I notice:
  KASSERT(td == curthread, (sched_bind: can only bind curthread));
 
 How can I bind a thread that is not curthread?

I don't know offhand but your original suggestion of 'td_cpuset' does seem like 
a reasonable place to start.

Best
Neel

 
 
 best
 Neel
 
 Thank you,
 Stefan
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: How to bind a thread to a CPU?

[Neel Natu]

Hi Stefan,

On Sat, Jun 13, 2015 at 3:33 AM, Stefan Andritoiu
stefan.andrit...@gmail.com wrote:
 Hi,

 How can I pin a thread to run only on a specific CPU?
 Is it enough just to set the struct cpuset *td_cpuset of the thread?


sched_bind() is the proper way to do this.

best
Neel

 Thank you,
 Stefan
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: corrupting zfs pools?

[Neel Natu]

Hi Andriy,

On Wed, Jun 3, 2015 at 11:05 PM, Andriy Gapon a...@freebsd.org wrote:
 On 04/06/2015 07:37, Neel Natu wrote:
 Ok, there are some differences in our systems. The interesting ones
 are number of ASIDs (64 versus 65536), flush-by-asid capability,
 vmcb-clean capability and the number of cores.

 I was able to mimic all of these on my Opteron but still wasn't able
 to reproduce the issue. I am going to get a Sempron tomorrow which
 belongs to the same processor family as the Athlon II so hoping that
 it is easier to repro.

 BTW does this happen consistently on your system?

 Yes, it happened every time I tried that scenario.

 Thank you very much!  If there is anything else that I can do on my side to
 help the investigation just let me know.


I tried to repro on a Sempron 3850 APU but still no luck.

Since you can repro it readily I'll follow up with you so we can
narrow this down on your end.

best
Neel

 --
 Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: corrupting zfs pools?

[Neel Natu]

Hi Andriy,

On Wed, Jun 3, 2015 at 3:01 AM, Andriy Gapon a...@freebsd.org wrote:
 On 03/06/2015 00:40, Neel Natu wrote:
 Perhaps, but I wasn't able to repro this. I tried your recipe to repro
 on two systems running -current at r283917.

 - Intel Sandybridge server: Xeon E52650  with 8 cores/16 threads and 64GB 
 memory
   - hdd1/hdd2 on UFS
   - hdd1/hdd2 on ZFS

 - AMD Opteron server: Opteron 6230 with 8 cores and 16GB memory
   - hdd1/hdd2 on UFS

 Can you provide some more details about your setup? I can then try to
 repro on a system similar to your setup.

 - uname -a

 FreeBSD trant 11.0-CURRENT FreeBSD 11.0-CURRENT #134 r283188+ab4f83f(devel): 
 Fri
 May 22 15:55:27 EEST 2015 avg@trant:/usr/obj/usr/src/sys/TRANT  amd64

 This is not a pure FreeBSD, there are some local changes, but none to vmm or 
 VM.

 - sysctl hw.model

 hw.model: AMD Athlon(tm) II X2 250 Processor

 - sysctl hw.ncpu

 hw.ncpu: 2

 - sysctl hw.physmem

 hw.physmem: 8029335552

 - sysctl hw.vmm

 hw.vmm.npt.pmap_flags: 507
 hw.vmm.svm.num_asids: 64
 hw.vmm.svm.disable_npf_assist: 0
 hw.vmm.svm.features: 15
 hw.vmm.svm.vmcb_clean: 959
 hw.vmm.vmx.vpid_alloc_failed: 0
 hw.vmm.vmx.posted_interrupt_vector: -1
 hw.vmm.vmx.cap.posted_interrupts: 0
 hw.vmm.vmx.cap.virtual_interrupt_delivery: 0
 hw.vmm.vmx.cap.invpcid: 0
 hw.vmm.vmx.cap.monitor_trap: 0
 hw.vmm.vmx.cap.unrestricted_guest: 0
 hw.vmm.vmx.cap.pause_exit: 0
 hw.vmm.vmx.cap.halt_exit: 0
 hw.vmm.vmx.initialized: 0
 hw.vmm.vmx.cr4_zeros_mask: 0
 hw.vmm.vmx.cr4_ones_mask: 0
 hw.vmm.vmx.cr0_zeros_mask: 0
 hw.vmm.vmx.cr0_ones_mask: 0
 hw.vmm.ept.pmap_flags: 0
 hw.vmm.vrtc.flag_broken_time: 1
 hw.vmm.ppt.devices: 0
 hw.vmm.iommu.initialized: 0
 hw.vmm.bhyve_xcpuids: 4
 hw.vmm.topology.cpuid_leaf_b: 1
 hw.vmm.topology.cores_per_package: 1
 hw.vmm.topology.threads_per_core: 1
 hw.vmm.create: beavis
 hw.vmm.destroy: beavis
 hw.vmm.force_iommu: 0
 hw.vmm.trace_guest_exceptions: 0
 hw.vmm.ipinum: 251
 hw.vmm.halt_detection: 1

 - host filesystem underlying hdd1 and hdd2

 It's ZFS.


Ok, there are some differences in our systems. The interesting ones
are number of ASIDs (64 versus 65536), flush-by-asid capability,
vmcb-clean capability and the number of cores.

I was able to mimic all of these on my Opteron but still wasn't able
to reproduce the issue. I am going to get a Sempron tomorrow which
belongs to the same processor family as the Athlon II so hoping that
it is easier to repro.

BTW does this happen consistently on your system?

best
Neel

 Thank you!

 --
 Andriy Gapon
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: corrupting zfs pools?

[Neel Natu]

Hi Andriy,

On Tue, Jun 2, 2015 at 9:45 AM, Andriy Gapon a...@freebsd.org wrote:
 On 02/06/2015 14:14, Andriy Gapon wrote:

 I am doing a simple experiment.

 I get FreeBSD image from here:
 ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz

 Then I run in bhyve with two additional disks created with truncate -s 4g:
 $ bhyveload -m 1G -d
 ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test
 $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s
 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img
 -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s
 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l
 com2,/dev/nmdm0A -c 2 -m 1g test

 Note sectorsize=512/4096 options.  Not sure if it's them that cause the 
 trouble.

 Then, in the VM:
 $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2
 $ zfs create -p l2arc-test/ROOT/initial
 $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f -

 Afterwards, zpool status -v reports no problem.
 But then I run zpool scrub and get the following in the end:
 $ zpool status -v
   pool: l2arc-test
  state: ONLINE
 status: One or more devices has experienced an error resulting in data
 corruption.  Applications may be affected.
 action: Restore the file in question if possible.  Otherwise restore the
 entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
   scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun  2 13:58:17 2015
 config:

 NAMESTATE READ WRITE CKSUM
 l2arc-test  ONLINE   0 0 9
   mirror-0  ONLINE   0 018
 vtbd1   ONLINE   0 025
 vtbd2   ONLINE   0 023

 errors: Permanent errors have been detected in the following files:

 /l2arc-test/ROOT/initial/usr/bin/svnlitesync
 /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz
 /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz

 /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a


 The same issue is reproducible with ahci-hd.

 My host system is a recent amd64 CURRENT as well.  The hardware platform is 
 AMD.


 I used the following monstrous command line to reproduce the test in qemu:
 $ qemu-system-x86_64 -smp 2 -m 1024 -drive
 file=/tmp/livecd2/R2.img,format=raw,if=none,id=bootd -device
 virtio-blk-pci,drive=bootd -drive
 file=/tmp/l2arc-test/hdd1,if=none,id=hdd1,format=raw -device
 virtio-blk-pci,drive=hdd1,logical_block_size=4096 -drive
 file=/tmp/l2arc-test/hdd2,id=hdd2,if=none,format=raw -device
 virtio-blk-pci,drive=hdd2,logical_block_size=4096 -drive
 file=/tmp/l2arc-test/ssd,id=ssd,if=none,format=raw -device
 virtio-blk-pci,drive=ssd,logical_block_size=4096 ...

 And several other variations of logical_block_size and physical_block_size.
 The tests a re very slow, but there are no checksum errors.

 So, I suspect guest memory corruption caused by bhyve.  Perhaps the problem is
 indeed specific to AMD-V.


Perhaps, but I wasn't able to repro this. I tried your recipe to repro
on two systems running -current at r283917.

- Intel Sandybridge server: Xeon E52650  with 8 cores/16 threads and 64GB memory
  - hdd1/hdd2 on UFS
  - hdd1/hdd2 on ZFS

- AMD Opteron server: Opteron 6230 with 8 cores and 16GB memory
  - hdd1/hdd2 on UFS

Can you provide some more details about your setup? I can then try to
repro on a system similar to your setup.

- uname -a
- sysctl hw.model
- sysctl hw.ncpu
- sysctl hw.physmem
- sysctl hw.vmm
- host filesystem underlying hdd1 and hdd2

best
Neel

 --
 Andriy Gapon
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: can we get some interaction between halt/reboot and bhyve?

[Neel Natu]

Hi Paul

On Tue, May 26, 2015 at 11:18 PM, Paul Vixie p...@redbarn.org wrote:


 Craig Rodrigues wrote:
 ...

 I haven't tried them, but Michael Dexter has written more comprehensive
 rc.d scripts from bhyve:

 http://vmrc.bsd.lv/

 these look good but i'm waiting for man pages, while meanwhile i bake my
 own toolset.

 rc.d scripts work well if you can live with the normal shutdown process
 invoking rc.d scripts.
 For my uses, I can live with shutdown and rc.d.

 so can i except in emergencies.

 If you want to halt/reboot/poweroff, then you still have the original
 problem that Paul pointed out.

 reboot.c currently detects paging. is there a way to have it detect disk
 i/o in general?

 note, 30 seconds isn't always enough. on a DCC server with 4GB mmap()'d,
 i've run out of time. so, a sysctl for both the minimum (currently 5
 seconds) and maximum (currently 30 seconds) reboot i/o drain period
 would be great. i would code these and submit them if these changes were
 welcomed.


Yes, please do.

best
Neel

 --
 Paul Vixie
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Is the BHyVe guest as suitable for high-performance disk IO as the host?

[Neel Natu]

Hi Tinker,

On Sun, May 10, 2015 at 6:16 AM, Tinker ti...@openmailbox.org wrote:
 Hi Neel,

 Thank you very much for your response -

 That's great!

 I guess this should deliver the full capacity for basically any IO system
 around, be it a fast SSD or ramdisk. (Since the guest may not need to flush
 data immediately to the host, I guess read performance is the more sensitive
 point.)

 Which disk emulation mode is best for a BSD guest, virtio-blk or
 ahci-hd?


You'll need to experiment because there are so many moving pieces: the
guest driver implementation, VM exit overhead, capabilities of a
emulated disk controller. From the narrow perspective of VM-exit
overhead 'virtio-blk' is better because the CPU decodes all the
information needed to handle the I/O exit.

 In general, should any other consideration be made for performance (using
 the direct or nocache BHyVe configuration options, particular
 consideration for proper sector alignment when using a disk image stored on
 the host's SSD-based ZFS, mounting the host ZFS filesystem with noatime)?


Using nocache will prevent double-buffering of the virtual disk file
in the host. Using direct will guarantee consistency of the virtual
disk file on the underlying storage device. If you are purely focused
on performance then don't use either option but be aware that you'll
be using buffer cache on the host and also that the guest filesystem
could be corrupted on an unplanned host shutdown.

With respect to alignment I am not sure if you can request a specific
alignment when creating a flat file for the virtual disk. However, if
you use a ZVOL as a backing device then it can be created on top of a
partition which has the correct alignment.

best
Neel

 Kind regards,
 Tinker


 On 2015-05-10 04:03, Neel Natu wrote:

 Hi Tinker,

 On Sat, May 9, 2015 at 9:49 AM, Tinker ti...@openmailbox.org wrote:

 Hi!

 For an environment with very heavy parallell IO, should the performance
 be
 just as good in a BHyVe guest as in the FreeBSD host environment?

 What I thought of is that I guess within the host environment, the
 storage
 subsystem should have all kinds of optimizations like an internal work
 queue
 that pushes lots of work alinearly/asynchronously to the disk controller
 and
 this way allows it, in turn, to give all its performance.

 Does the virtualized disk interface carry over all that goodness to the
 guest?


 bhyve creates 8 worker threads for each virtual disk controller (both
 ahci and virtio-blk).

 All guest I/O is handled asynchronously by these worker threads which
 provide parallelism.

 (https://wiki.freebsd.org/bhyve seems to say yes, presuming you configure
 BHyVe to run the virtual disk in AHCI mode?)


 The wiki is out of date.

 Since r280037 the virtio-blk emulation also gets the benefits of using
 the block_if worker threads.

 best
 Neel

 Thanks!
 Tinker

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org


___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Panic on 11.0-CURRENT when trying kldload vmm

[Neel Natu]

Hi Markiyan,

On Thu, Apr 16, 2015 at 1:30 PM, Markiyan Kushnir
markiyan.kush...@gmail.com wrote:
 2015-04-16 23:20 GMT+03:00 Neel Natu neeln...@gmail.com:
 Hi Markiyan,

 On Thu, Apr 16, 2015 at 1:01 PM, Markiyan Kushnir
 markiyan.kush...@gmail.com wrote:
 22:13:~$ uname -a
 FreeBSD mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #3
 r281432: Sun Apr 12 01:35:34 EEST 2015
 root@:/work/obj/work/src.svn/sys/MAREK  amd64


 Went kernel panic twice today when trying to load vmm.  Never seen
 this before (the previous kernel was built out of r278973, Feb 19
 2015, was doing well).  The panic is not reliable in the sense I could
 not reproduce it at will just loading/unloading vmm.

 I regularly work in my bhyve guests, sometimes swap between vmm and
 vboxdrv, all my scenarios haven't changed from the previous Feb
 kernel.  Except that I re-built my today panicking kernel extending
 the MINIMAL config (including what I need) as opposed to the previous
 one reducing GENERIC (excluding what I don't need).

 Both today panic's back traces look identical. Attaching core.txt.1
 (the latter one) and the output of sysctl kern.conftxt. Let me know
 if someone wants to debug it.


 I don't think that 'core.txt.1' made it through.

 Regardless, can you try to reproduce after applying r281559. I think
 it should fix the panic.

 https://svnweb.freebsd.org/changeset/base/281559


 ah ok. Saved on google drive:
 https://drive.google.com/file/d/0B9Q-zpUXxqCnam0zNjFRcF9XMHc/view?usp=sharing


Yup, its the one that is fixed in r281559.

best
Neel

 and yes, will shift to r281559, and see if it's ok.

 Thanks,
 Markiyan.

 best
 Neel

 Thanks,
 Markiyan.

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Problem hiding USB host controller

[Neel Natu]

Hi Luis,

On Fri, Apr 10, 2015 at 4:50 PM, Luis P. Mendes luisl...@gmail.com wrote:
 Hi Neel,


 I updated my STABLE to r281384M and applied the patch.  It seems that
 the patch solved the issue:

 # pciconf -vl | grep ppt
 ppt0@pci0:0:20:0:   class=0x0c0330 card=0x8c311849 chip=0x8c318086
 rev=0x05 hdr=0x00
 ppt1@pci0:0:26:0:   class=0x0c0320 card=0x8c2d1849 chip=0x8c2d8086
 rev=0x05 hdr=0x00
 ppt2@pci0:1:0:0:class=0x010601 card=0x91721849 chip=0x91721b4b
 rev=0x11 hdr=0x00

 As I'm not a developer, please tell me whether this patch will be
 added to STABLE or if I need to patch again when I update the system
 in the future.


I have committed the patch to HEAD:
https://svnweb.freebsd.org/base?view=revisionrevision=281499

I'll MFC it to 10-stable in a couple of weeks.

best
Neel

 Thank you very much,


 Luis

 2015-04-10 6:30 GMT+01:00 Neel Natu neeln...@gmail.com:
 Hi Luis,

 This following patch should do the trick:
 https://people.freebsd.org/~neel/patches/usb_pci_probe_default.patch

 The patch is against -current although it applies cleanly to 10-stable as 
 well.

 best
 Neel

 On Thu, Apr 9, 2015 at 3:32 PM, Luis P. Mendes luisl...@gmail.com wrote:
 Hi,

 2015-04-09 22:25 GMT+01:00 Neel Natu neeln...@gmail.com:
 Hi Luis,

 On Thu, Apr 9, 2015 at 1:43 PM, Luis P. Mendes luisl...@gmail.com wrote:
 Hi,

 I'm trying to passthrough a USB host controller (2.0 or 3.0) but there
 seems to be some kind of problem.

 # uname -a
 FreeBSD leao 10.1-STABLE FreeBSD 10.1-STABLE #0 r279201: Mon Feb 23
 18:55:46 UTC 2015
 r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

 VT-d is enabled in bios.
 CPU is a Intel Xeon E3-1246v3 Haswell


 # cat /boot/loader.conf
 # load radeon firmware not presented
 pptdevs=0/26/0 1/0/0 0/20/0
 blackhole_load=YES

 # pciconf -lv
 # only relevant
 ehci0@pci0:0:26:0:  class=0x0c0320 card=0x8c2d1849 chip=0x8c2d8086
 rev=0x05 hdr=0x00
 vendor = 'Intel Corporation'
 device = 'Lynx Point USB Enhanced Host Controller'
 class  = serial bus
 subclass   = USB

 xhci0@pci0:0:20:0:  class=0x0c0330 card=0x8c311849 chip=0x8c318086
 rev=0x05 hdr=0x00
 vendor = 'Intel Corporation'
 device = 'Lynx Point USB xHCI Host Controller'
 class  = serial bus
 subclass   = USB

 ppt0@pci0:1:0:0:class=0x010601 card=0x91721849 chip=0x91721b4b
 rev=0x11 hdr=0x00


 Only the 1:0:0 disk controler is hidden.  The two USB controllers, one
 ehci and one xhci are not.

 Why don't the controllers get hidden?
 Is the blackhole_load=YES still relevant?  I tried with and without
 that line to no avail.
 Is my STABLE version too old?


 The blackhole driver is not needed anymore since that functionality
 was moved into vmm.ko.

 Can you replace the line blackhole_load=YES to vmm_load=YES in 
 loader.conf?

 The vmm_load=YES line was already present.
 The full contents of the file are:

  % cat /boot/loader.conf
 net.fibs=4

 radeonkmsfw_CAICOS_pfp_load=YES
 radeonkmsfw_CAICOS_me_load=YES
 radeonkmsfw_BTC_rlc_load=YES
 radeonkmsfw_CAICOS_mc_load=YES
 radeonkms_load=YES

 vmm_load=YES
 nmdm_load=YES
 if_bridge_load=YES
 if_tap_load=YES

 pptdevs=1/0/0 0/20/0 0/26/0



 Thanks,


 Luis
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Problem hiding USB host controller

[Neel Natu]

Hi Luis,

On Thu, Apr 9, 2015 at 1:43 PM, Luis P. Mendes luisl...@gmail.com wrote:
 Hi,

 I'm trying to passthrough a USB host controller (2.0 or 3.0) but there
 seems to be some kind of problem.

 # uname -a
 FreeBSD leao 10.1-STABLE FreeBSD 10.1-STABLE #0 r279201: Mon Feb 23
 18:55:46 UTC 2015
 r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

 VT-d is enabled in bios.
 CPU is a Intel Xeon E3-1246v3 Haswell


 # cat /boot/loader.conf
 # load radeon firmware not presented
 pptdevs=0/26/0 1/0/0 0/20/0
 blackhole_load=YES

 # pciconf -lv
 # only relevant
 ehci0@pci0:0:26:0:  class=0x0c0320 card=0x8c2d1849 chip=0x8c2d8086
 rev=0x05 hdr=0x00
 vendor = 'Intel Corporation'
 device = 'Lynx Point USB Enhanced Host Controller'
 class  = serial bus
 subclass   = USB

 xhci0@pci0:0:20:0:  class=0x0c0330 card=0x8c311849 chip=0x8c318086
 rev=0x05 hdr=0x00
 vendor = 'Intel Corporation'
 device = 'Lynx Point USB xHCI Host Controller'
 class  = serial bus
 subclass   = USB

 ppt0@pci0:1:0:0:class=0x010601 card=0x91721849 chip=0x91721b4b
 rev=0x11 hdr=0x00


 Only the 1:0:0 disk controler is hidden.  The two USB controllers, one
 ehci and one xhci are not.

 Why don't the controllers get hidden?
 Is the blackhole_load=YES still relevant?  I tried with and without
 that line to no avail.
 Is my STABLE version too old?


The blackhole driver is not needed anymore since that functionality
was moved into vmm.ko.

Can you replace the line blackhole_load=YES to vmm_load=YES in loader.conf?

best
Neel


 Luis

 PS.  Keep up the excellent work bringing
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: Abort trap

[Neel Natu]

Hi,

On Sat, Apr 4, 2015 at 8:20 PM, Евгений IRON i...@mail.ua wrote:
  Hello all,

 After I upgraded FreeBSD STABLE up to r281095, I can't boot up VM with CentOS 
 6.

 # uname -a
 FreeBSD cobra 10.1-STABLE FreeBSD 10.1-STABLE #8 r281095: Sun Apr 5 05:20:57 
 EEST 2015 root@cobra:/usr/obj/usr/src/sys/COBRA amd64
 # cat bitrix.grub
 linux /vmlinuz-2.6.32-504.12.2.el6.x86_64 ro 
 root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD 
 KEYTABLE=us rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 
 console=ttyS0 crashkernel=auto rd_LVM_LV=VolGroup/lv_root rd_NO_DM
 initrd /initramfs-2.6.32-504.12.2.el6.x86_64.img
 boot

 # cat bitrix_start.sh
 #!/usr/local/bin/bash

 ifconfig tap0 mtu 9000
 ifconfig bridge0 deletem em0
 ifconfig bridge0 addm em0 addm tap0
 bhyvectl --destroy --vm=bitrix  /dev/null
 grub-bhyve -m device.map -r hd0,msdos1 -M 4096M bitrix  bitrix.grub
 bhyve -c 4 -AI -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 
 3:0,virtio-blk,/dev/zvol/zroot/bitrix -l com1,stdio -m 4096M bitrix


 Console output:
 ...
 CPU: Physical Processor ID: 0
 alternatives: switching to unfair spinlock
 ACPI: Core revision 20090903
 ACPI Exception: AE_NO_ACPI_TABLES, While loading namespace from ACPI tables 
 (20090903/tbxface-605)
 ACPI: Unable to load the System Description Tables
 ftrace: converting mcount calls to 0f 1f 44 00 00
 ftrace: allocating 21919 entries in 86 pages
 APIC routing finalized to flat.
 ..TIMER: vector=0x30 apic1=0 pin1=0 apic2=-1 pin2=-1
 ..MP-BIOS bug: 8254 timer not connected to IO-APIC
 ...trying to set up timer (IRQ0) through the 8259A ...
 . (found apic 0 pin 0) ...
 ... failed.
 ...trying to set up timer as Virtual Wire IRQ...
 . failed.
 ...trying to set up timer as ExtINT IRQ...
 Unhandled outb 0x0071
 ./bitrix_start.sh: line 7: 5202 Abort trap (core dumped) bhyve -c 4 -AI -H -P 
 -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 
 3:0,virtio-blk,/dev/zvol/zroot/bitrix -l com1,stdio -m 4096M bitrix
 Tell me please, how can I fix this problem?
 I can rebuild kernel with debug enabled and make backtrace of core dump if it 
 will help somehow to find a reason. I guess  281075  commit is the reason.

 Thanks a lot!

I am surprised that you even got that far. I was getting ACPI
compilation failures in bhyve after r281075 upgraded the ACPICA in
stable/10.

This is fixed in r281134. Can you try running your guest after syncing
past this change?

best
Neel

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Several bhyve quirks

[Neel Natu]

Hi Julian,

On Wed, Mar 25, 2015 at 2:24 AM, Julian Hsiao mad...@nyanisore.net wrote:
 Hi,

 I'm running bhyve on 10.1, mostly with OpenBSD (5.7) guests, and I ran into
 a few strange issues:

 1. The guest RTC is several hours off every time I start bhyve.  The host
 RTC is set to UTC, and /etc/localtime on both the host and guests are set to
 US/Pacific (currently PDT).  I thought maybe bhyve is setting the RTC to the
 local time, and indeed changing TZ environment variable affects the guest's
 RTC.  However, with TZ=UTC the guest is still off by an hour, and to get the
 correct offset I set TZ='UTC+1'; perhaps something's not handling DST
 correctly?

 Also, one time the offset was mysteriously tens of hours off (i.e. the guest
 RTS is a day or two ahead), and the condition persisted across multiple host
 and guest reboots.  Unfortunately, the problem went away a few hours later
 and I was unable to reproduce it since.


The problem is that in 10.1 (and earlier) bhyve defaulted to a 12-hour
RTC format but some guests like OpenBSD and Linux assume that it is
configured in the 24-hour format.

The 12-hour format indicates PM time by setting the most significant
bit in the 'hour' byte. Since the guest is not prepared to mask this
bit it thinks that the time is 68 hours ahead of the actual time (but
only for PM times - everything goes back to normal during AM times).

This is fixed in HEAD where the RTC device model defaults to 24-hour time.

 https://github.com/freebsd/freebsd/commit/b341fa888c7a3b71ef8fb36ed40f08b7ceb8c486
 suggests that I'm on the right track, but it doesn't explain the off-by-one
 nor the (one time) multi-day offset.


The one-hour offset is a bug due to my interpretation of the 12-hour format.

I am going to fix this in HEAD shortly but here is a patch for 10.1 and earlier:
https://people.freebsd.org/~neel/patches/bhyve_openbsd_rtc.patch

 As an aside, the commit message implies that this only affects OpenBSD
 guest, when in fact this probably affects all guests (at least also Linux).
 Perhaps he meant you cannot configure OpenBSD to assume that the RTC is set
 to local time instead of UTC.

 2. What's the preferred solution for minimizing guest clock drift in bhyve?
 Based on some Google searches, I run ntpd in the guests and set
 kern.timecounter.hardware=acpitimer0 instead of the default acpihpet0.
 acpitimer0 drifts by ~600 ppm while acpihpet0 drifts by ~1500 ppm; why?


I don't know but I am running experiments that I hope will provide some insight.

best
Neel

 3. Even moderate guest disk I/O completely kills guest network performance.
 For example, whenever security(8) (security(7) in FreeBSD) runs, guest
 network throughput drops from 150+ Mbps to ~20 Mbps, and jitter from ping
 jumps from 0.01 ms to 100+ ms.  If I try to build something in the guest,
 then network becomes almost unusable.

 The network performance degradation only affects the guest that's generating
 the I/O; high I/O on guest B doesn't affect guest A, nor would high I/O on
 the host.

 I'm using both virtio-blk and virio-net drivers, and the guests' disk images
 are backed by zvol+geli.  Removing geli has no effect.

 There are some commits in CURRENT that suggests improved virtio performance,
 but I'm not comfortable running CURRENT.  Is there a workaround I could use
 for 10.1?

 4. virtio-blk always reports the virtual disk as having 512-byte sectors,
 and so I get I/O errors on OpenBSD guests when the disk image is backed by
 zvol+geli with 4K sector size.  Curiously, this only seems to affect
 zvol+geli; with just zvol it seems to work.  Also, it works either way on
 Linux guests.

 ATM I changed the zvol / geli sector size to 512 bytes, which probably made
 #2 worse.  I think this bug / feature is addressed by:
 https://github.com/freebsd/freebsd/commit/02e846756ee99b849987a9bb6f57566fc70360c7,
 but again is there a workaround to force a specific sector size for 10.1?

 5. This may be better directed at OpenBSD but I'll ask here anyway: if I
 enable virtio-rnd then OpenBSD would not boot with couldn't map interrupt
 error.  The kernel in bsd.rd will boot, but not the installed kernel (or the
 one built from STABLE; I forgot).  Again, Linux seems unaffected, but I
 couldn't tell if it's actually working.

 Julian Hsiao


 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve failure: vm_run error -1, errno 25

[Neel Natu]

Hi Craig,

On Sun, Mar 1, 2015 at 12:54 PM, Craig Rodrigues rodr...@freebsd.org wrote:
 Hi,

 I recently did the following:

 (1)  Updated to svn r279489, rebuilt world/kernel and rebooted.

 (2)  Ran this: sh /usr/share/examples/bhyve/vmrun.sh -m 4G -d /vm/
 scan-ci.FreeBSD.org/disk.img vmtest1

 (3)  Got this error:  vm_run error -1, errno 25
See attached log.

 Any idea what is going on?  This VM used to work, and I did not
 get this error before.


I think that this is due to world and kernel not being in sync:

[neel@wreck ~]$ uname -K
1100062
[neel@wreck ~]$ uname -U
1100039

Can you update world and see if the issue persists?

best
Neel

 This is on wreck.ysv.freebsd.org, so Neel and Peter should have access to
 debug if necessary.

 Thanks.

 --
 Craig

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: OpenBSD guest in bhyve on AMD CPU

[Neel Natu]

Hi Jason,

On Tue, Feb 10, 2015 at 9:40 PM, Neel Natu neeln...@gmail.com wrote:
 Hi Jason,

 On Tue, Feb 10, 2015 at 7:38 PM, Jason Tubnor ja...@tubnor.net wrote:
 Hi,

 As posted in IRC, I am checking here if anyone has come across this
 issue.  During boot on OpenBSD I get:

 panic: trap type 4, code=0, pc=811b7134


 I suspect that the guest is getting a #GP exception when accessing an
 MSR that is not emulated by bhyve.

 Can you start the guest with the -w option passed to bhyve?


r279227 has a fix for this and should allow booting without the -w.

https://svnweb.freebsd.org/base?view=revisionrevision=279227

best
Neel

 best
 Neel

 The operating system has halted.
 Please press any key to reboot.

 I don't get this with the same ISO on an Intel CPU.  Below is the
 console from the OpenBSD boot, the dmesg from FreeBSD and the bhyve
 command.  Any suggestions or tips greatly appreciated.

 Thanks,

 Jason.

 --
 console

 Copyright (c) 1982, 1986, 1989, 1991, 1993
 The Regents of the University of California.  All rights reserved.
 Copyright (c) 1995-2015 OpenBSD. All rights reserved.  http://www.OpenBSD.org

 OpenBSD 5.7-beta (RAMDISK_CD) #733: Tue Jan 20 15:39:52 MST 2015
 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
 real mem = 1056964608 (1008MB)
 avail mem = 1027194880 (979MB)
 mainbus0 at root
 bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xf101f (9 entries)
 bios0: vendor BHYVE version 1.00 date 03/14/2014
 acpi0 at bios0: rev 2
 acpi0: sleep states S5
 acpi0: tables DSDT APIC FACP HPET MCFG
 acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
 cpu0 at mainbus0: apid 0 (boot processor)
 cpu0: AMD Phenom(tm) II X4 955 Processor, 3214.13 MHz
 cpu0: 
 FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16,POPCNT,HV,NXE,MMXX,FFXSR,PAGE1GB,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,SKINIT,PERF,ITSC
 cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
 64b/line 16-way L2 cache, 6MB 64b/line 48-way L3 cache
 cpu0: ITLB 32 4KB entries fully associative, 16 4MB entries fully associative
 cpu0: DTLB 48 4KB entries fully associative, 48 4MB entries fully associative
 fatal protection fault in supervisor mode
 trap type 4 code 0 rip 811b7134 cs 8 rflags 10286 cr2  0 cpl e
 rsp 818058e0
 panic: trap type 4, code=0, pc=811b7134

 The operating system has halted.
 Please press any key to reboot.

 -
 dmesg

 Copyright (c) 1992-2015 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 10.1-STABLE #0 r278028: Sun Feb  1 18:05:27 UTC 2015
 r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
 CPU: AMD Phenom(tm) II X4 955 Processor (3214.18-MHz K8-class CPU)
   Origin=AuthenticAMD  Id=0x100f43  Family=0x10  Model=0x4  Stepping=3
   
 Features=0x178bfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT
   Features2=0x802009SSE3,MON,CX16,POPCNT
   AMD 
 Features=0xee500800SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!
   AMD 
 Features2=0x37ffLAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT
   SVM: NP,NRIP,NAsids=64
   TSC: P-state invariant
 real memory  = 17179869184 (16384 MB)
 avail memory = 16063627264 (15319 MB)
 Event timer LAPIC quality 400
 ACPI APIC Table: GBTGBTUACPI
 FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 FreeBSD/SMP: 1 package(s) x 4 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
 ioapic0: Changing APIC ID to 8
 ioapic0 Version 2.1 irqs 0-23 on motherboard
 random: Software, Yarrow initialized
 kbd1 at kbdmux0
 acpi0: GBT GBTUACPI on motherboard
 acpi0: Power Button (fixed)
 acpi0: reservation of 0, a (3) failed
 acpi0: reservation of 10, cfbf (3) failed
 cpu0: ACPI CPU on acpi0
 cpu1: ACPI CPU on acpi0
 cpu2: ACPI CPU on acpi0
 cpu3: ACPI CPU on acpi0
 attimer0: AT timer port 0x40-0x43 irq 0 on acpi0
 Timecounter i8254 frequency 1193182 Hz quality 0
 Event timer i8254 frequency 1193182 Hz quality 100
 atrtc0: AT realtime clock port 0x70-0x73 irq 8 on acpi0
 Event timer RTC frequency 32768 Hz quality 0
 Timecounter ACPI-fast frequency 3579545 Hz quality 900
 acpi_timer0: 32-bit timer at 3.579545MHz port 0x4008-0x400b on acpi0
 acpi_button0: Power Button on acpi0
 pcib0: ACPI Host-PCI bridge port 0xcf8-0xcff on acpi0
 pci0: ACPI PCI bus on pcib0
 pcib1: ACPI PCI-PCI bridge at device 1.0 on pci0
 pci1: ACPI PCI bus on pcib1
 vgapci0: VGA-compatible display port 0xee00-0xeeff mem
 0xd000-0xdfff,0xfdfe-0xfdfe,0xfde0-0xfdef irq
 18 at device 5.0 on pci1
 vgapci0: Boot video

Re: [current] bhyve under VMware borked?

[Neel Natu]

Hi Marcel,

On Fri, Nov 7, 2014 at 7:21 PM, Marcel Moolenaar mar...@xcllnt.net wrote:

 On Nov 7, 2014, at 4:29 PM, Michael Dexter edi...@callfortesting.org 
 wrote:

 On 11/7/14 12:50 PM, Marcel Moolenaar wrote:
 This is VMware Fusion 6.0.5 (2209127) running on Mac OS X 10.10.

 Just in case: Have you selected Enable hypervisor applications in this 
 virtual machine in the Processors: Advanced options settings?

 Michael

 Yes. Reverting the change Peter talked about fixed it for me.


There is now a proper fix for this in r279228.
https://svnweb.freebsd.org/base?view=revisionrevision=279228

best
Neel

 --
 Marcel (mobile)
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve and PCI Passthrough a USB Controller

[Neel Natu]

Hi Philipp,

On Mon, Feb 16, 2015 at 3:09 AM, Philipp Wuensche cryx-free...@h3q.com wrote:
 Peter Grehan wrote:

  Would you be willing to try a patch that relaxes that check ?

 Yes! The system isn't in production yet so I can try out stuff.


Can you try this patch?
https://people.freebsd.org/~neel/patches/bhyve_ignore_unsupported_pba.patch

best
Neel

 Any suggestions for known to be working USB Controllers with bhyve and
 PCI passthrough?

  I've not tried with any discrete cards, but any XHCI controller that
 has the table and PBA regions either in separate BARs or not in the same
 4K region (recommended behaviour in the PCIe spec) should be fine.

 I ordered another controller with an uPD720200 chipset, maybe this one
 will work out of the box.

 Greetings,
 Philipp
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Bhyve crash: Effective AML package length is zero.

[Neel Natu]

Hi William,

On Sun, Dec 28, 2014 at 3:07 PM, William B will...@firstyear.id.au wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Good morning,

 I am having an issue with bhyve on my 10-STABLE amd64 system.

 When running the following:

 /usr/sbin/bhyveload -m 1 -d 
 /var/lib/libvirt/images/FreeBSD-10.1-RELEASE-amd64-disc1.iso bhyve
 /usr/sbin/bhyve -c 1 -m 1 -A -I -H -P -s 0:0,hostbridge -s 
 2:0,virtio-net,tap0,mac=52:54:00:dc:cd:c5 -s 
 4:0,ahci-cd,/var/lib/libvirt/images/FreeBSD-10.1-RELEASE-amd64-disc1.iso -s 
 3:0,ahci-hd,/var/lib/libvirt/images/bhyve_freebsd.img -Y bhyve


You'll need to give the VM a bit more than 1MB of memory :-)

I think something like -m 512 or -m 1G will work better.

best
Neel

 I recieve the following:

 /tmp/bhyve.1f9VBHc 84:   Name (PPRT, Package ()
 Remark   2095 - Effective AML package length is zero ^

 vm exit[0]
 reason  VMX
 rip 0x802db5c0
 inst_length 3
 status  0
 exit_reason 48
 qualification   0x0184
 inst_type   0
 inst_error  0


 When inserted into gdb I see the following output from bt:

 #0  0x00080110a6ca in thr_kill () from /lib/libc.so.7
 #1  0x0008011df149 in abort () from /lib/libc.so.7
 #2  0x004099f1 in vm_loop (ctx=0x801c160c0, vcpu=0, 
 rip=18446744071565063616) at /usr/src/usr.sbin/bhyve/bhyverun.c:608
 #3  0x0040806b in fbsdrun_start_thread (param=0x838600) at 
 /usr/src/usr.sbin/bhyve/bhyverun.c:245
 #4  0x000800e814f5 in pthread_create () from /lib/libthr.so.3
 #5  0x in ?? ()


 I am happy to debug this further if needed. Any other advice would be 
 appreciated.

 I am offlist, so please include me in replies.

 Note the -Y is due to mptable errors when I run without it. mptable requires 
 mapped mem.

 hw.machine: amd64
 hw.model: Intel(R) Core(TM) i3-4010U CPU @ 1.70GHz
 hw.ncpu: 4
 hw.machine_arch: amd64


 - --
 Sincerely,

 William Brown

 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2

 iQIcBAEBAgAGBQJUoI0hAAoJEO/EFteBqAmaBvIP/0rds9YwAfWJoRKX89o7tsf/
 Gka/oIy5NdazWqjt0nNQCv3bB9xvotSCcolDGyLjmDCh0N7y2hWatWyostCTL85L
 ZBtxkbiPOSuwm0BbVmBXfjvIZnYciwk9jTcPVYKMW78IwcWfz0hWu6fzLe7psmRR
 tfhW8mam1ZyZIcPgroXH+TSRllmYdTV9IJK2pGZuUPTjoWHNfNMvejn2R1+3fgZj
 4yaj7aLnhY+Y+R8JldKOuMfNvIZAV/TF7SKxl1chbE2VuYee/RJr90de/1k+iP4q
 4kpkdoNxhP2Xk65RbaIFJHs7JES1XE8oYjxCD2XxR1sZfyva9tZhK/2nVinVmyKc
 za1day5CzqOGHPiAW/u18B4Qg55+nwthCWsIE5tb0+/USm+ea1/mVlXN2gK6IPqJ
 d/tVuBpm+N9liTyFdEDGarR2gYayqOc3Xo9KbIX6CXUuXbVgmovLgs0hRGMhdwDP
 AgT/XcgXt91kwkGWrUCSdd74VQt0RP+qpoqxz4fqiLjBg6MuZqkDLlUx8MjjSPRT
 +EG0gbJKXq14+Vir7utBFaOanVYvYAXgsc5V0Pnihbukcf6a3knt0nH0YfWwDRjT
 Qiydm1/Fnw1IZLZ1dNR1lYeXoPVfQCTE99tTKNsMhW+mbr+6h5QrzGHALd4qSkba
 qisg8VXJnKZUaK+qUHvm
 =ETzm
 -END PGP SIGNATURE-
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FreeBSD bhyve VM Exiting

[Neel Natu]

Hi Shawn,

On Fri, Dec 26, 2014 at 6:26 AM, Shawn Webb latt...@gmail.com wrote:
 Hey All,

 I'm running a FreeBSD/amd64 11-current (r276204) guest VM in bhyve. It
 seems that running Poudriere in it causes the VM to exit.

 This is what bhyve outputs when the VM dies:

 reason  VMX
 rip 0x80d68564
 inst_length 5
 status  0
 exit_reason 2
 qualification   0x
 inst_type   0
 inst_error  0

 I have no idea what reason VMX means or what exit_reason 2 means. Is
 there a way to decode that?


Exit reason '2' is a triple fault.

 What kinds of info/data do you guys need to debug this?


You'll need a FreeBSD host with svn revision = 276098 and a custom
kernel config to enable KTR.
https://svnweb.freebsd.org/base?view=revisionrevision=276098

Then you can set the following tunable hw.vmm.trace_guest_exceptions
to 1 before loading vmm.ko. This will cause bhyve to trace all guest
exceptions via KTR. You can verify that the tunable took effect using
'sysctl hw.vmm.trace_guest_exceptions'.

When the bhyve process exits you can then dump the KTR entries using
something like this: 'sudo ktrdump -cto /tmp/ktrdump.out'. You'll
probably want to do this in a script so the ktrdump(8) happens
immediately after bhyve(8) exits.

I use the following options in my kernel config to enable KTR:
options KTR
options KTR_MASK=(KTR_GEN)
options KTR_ENTRIES=(4*1024*1024)

Hope this helps.

best
Neel

 Thanks,

 Shawn
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve failure

[Neel Natu]

Hi Julian,

On Sat, Nov 29, 2014 at 6:31 PM, Neel Natu neeln...@gmail.com wrote:
 Hi Julian,

 On Fri, Nov 28, 2014 at 7:36 AM, Julian Elischer jul...@freebsd.org wrote:
 On 11/28/14, 3:12 AM, Neel Natu wrote:

 Hi Julian,

 On Thu, Nov 27, 2014 at 7:02 AM, Julian Elischer jul...@freebsd.org
 wrote:

 I added WITNESS to a kernel I've been running and during boot I got this
 result:

 vm exit[0]
  reason  VMX
  rip 0x8078c1a7
  inst_length 7
  status  0
  exit_reason 2

 That's a triple fault (from sys/amd64/vmm/intel/vmcs.h)

 #define EXIT_REASON_TRIPLE_FAULT2


  qualification   0x
  inst_type   0
  inst_error  0
 Abort trap (core dumped)

 anyone have a decoder ring?

 A few questions to narrow this down:

 Did you add WITNESS to the guest kernel or the host kernel when this
 happened?

 yes I added it to the guest kernel.. I suspect it ay be related to this,
 especially since there was a kernel module loaded that was not recompiled.
 I'm trying to ascertain now whether a withness kernel requires special
 handling for its modules.



 Is this readily reproducible?

 don't know yet.


 Its interesting that the guest went all the way to a triple fault
 instead of stopping at the double fault handler.

 I'll be happy to track this down if there is a reliably repro case
 (guest binaries would be sufficient).


I just committed a change that should let you trace every exception
seen by the guest:
https://svnweb.freebsd.org/changeset/base/276098

It will let you trace all the exceptions leading up to the triple
fault via ktr(4).

best
Neel

 best
 Neel


 Can you share the host and guest revision numbers?


 guest is custom sources based on stable-10 on the same day as 10.0 was
 released.basically whatever svn revision# was on 10.0 but on the stable
 branch.
 host is
 FreeBSD 10.1-BETA1 #0 r271671: Tue Sep 16 09:31:23 PDT 2014





 best
 Neel

 julian
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org


___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: vmrun.sh error

[Neel Natu]

Hi Susan,

On Tue, Dec 9, 2014 at 5:05 PM, Susan Stanziano
susan.stanzi...@xinuos.com wrote:

 Hello,

 Craig suggested that I send this question to this mail alias:

 A couple of comments after creating VM's and using the vmrun.sh in 
 /usr/share/examples/bhyve/vmrun.sh:

 1) I know you can't run two VM's at the same time with the same vmname, but 
 the error message from bhyveload is not informative to a new user:

 bhyveload: could not open vmname

 2) The solution with the multiple use of a vmname seems to be that after 
 vmrun.sh issues the bhyvectl --destroy command, if there was a sleep 1 
 added, giving time for the destroy, the error message from bhyveload wouldn't 
 have to happen.

 What do you think?


I think the issue you are hinting at is that if /dev/vmm/testvm exists
there isn't a reliable way to know whether or not 'testvm' is actually
running. There is no good way to fix this until we get to a single
process model.

However, you know for sure that if '/dev/vmm/testvm' does not exist
then it is also not running.

So, perhaps the following patch fixes your problem:
https://people.freebsd.org/~neel/patches/vmrun_destroy.patch

best
Neel

 Thanks,

 Susan Stanziano


 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Neel Natu]

Hi Nikos,

On Sun, Nov 30, 2014 at 7:58 AM, Nikos Vassiliadis nv...@gmx.com wrote:
 Hi,

 On 11/30/14 02:43, Neel Natu wrote:

 Can you provide the output of 'pciconf -lvbc pci0:3:0:0' on the FreeBSD
 host?

 This is assuming pci0:3:0:0 is the wlan device being passed through
 based on an earlier email.


 Yes, it is. Here is the output:

 none0@pci0:3:0:0:   class=0x028000 card=0xe042105b chip=0x472714e4
 rev=0x01 hdr=0x00
 vendor = 'Broadcom Corporation'
 device = 'BCM4313 802.11b/g/n Wireless LAN Controller'
 class  = network
 bar   [10] = type Memory, range 64, base 0xb350, size 16384,
 enabled
 cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
 cap 09[58] = vendor (length 120)
 cap 05[48] = MSI supports 1 message, 64 bit

Ok, this device does have MSI which is what I wanted to confirm.

Its very likely that the device doesn't work when passed to the guest
because your system doesn't have VT-d.

best
Neel

 cap 10[d0] = PCI-Express 1 endpoint max data 128(128) link x1(x1)
  speed 2.5(2.5) ASPM L1(L1)
 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
 ecap 0002[13c] = VC 1 max VC0
 ecap 0003[160] = Serial 1 dc210c84
 ecap 0004[16c] = Power Budgeting 1


 Thanks, Nikos
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Neel Natu]

Hi Nikos,

On Sun, Nov 30, 2014 at 7:37 AM, Nikos Vassiliadis nv...@gmx.com wrote:
 Hi,

 On 11/30/14 00:48, Craig Rodrigues wrote:

 I'm not sure, but according to this datasheet:


 http://ark.intel.com/products/65700/Intel-Core-i3-3110M-Processor-3M-Cache-2_40-GHz

 this CPU does not support VT-d, which is needed for PCI passthru.


 Indeed. Good catch, Neel should this be the case?

Definitely!

vmm.ko should probably not even attach the ppt driver to pptdevs if
an IOMMU is absent.

best
Neel
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: Unsupported MSI-X configuration: 2/0/0

[Neel Natu]

Hi Nikos,

On Sat, Nov 29, 2014 at 10:45 AM, Nikos Vassiliadis nv...@gmx.com wrote:


 On 11/29/14 18:38, Anish Gupta wrote:

  Another thing I should mention is that acpidump -t | grep DMAR
 reports nothing. It is mentioned in the documentation but I didn't see
 until now.

 DMAR/VT-d has been supported for sometime now by Intel chipset. You want
 to check your BIOS set-up to enable VT-d.


 There are very few setting in the BIOS setup and nothing related to VT-d. My
 CPU and CPU settings are these:

 CPU: Intel(R) Core(TM) i3-3110M CPU @ 2.40GHz (2394.61-MHz K8-class CPU)
   Origin = GenuineIntel  Id = 0x306a9  Family = 0x6  Model = 0x3a
 Stepping = 9

 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE

 Features2=0x3dbae3bfSSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,XSAVE,OSXSAVE,AVX,F16C
   AMD Features=0x28100800SYSCALL,NX,RDTSCP,LM
   AMD Features2=0x1LAHF
   Structured Extended Features=0x281FSGSBASE,SMEP,ERMS
   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
   TSC: P-state invariant, performance statistics


 I have attached the output of acpidump -t in case you'd like to take a
 look.

  Actually my goal is to use the wifi adapter from a VM. The bge device
 is just testing and I thought I should report it/ask about it.
 What is the error you see in VM for enabling wifi adapter? Is the
 corresponding driver getting attached to WiFi adapter?


 When I load the kernel module, I get this:

 [  542.919615] cfg80211: Calling CRDA to update world regulatory domain
 [  542.932441] wl :00:07.0: can't derive routing for PCI INT A
 [  542.932449] wl :00:07.0: PCI INT A: no GSI
 [  543.002253] wlan0: Broadcom BCM4727 802.11 Hybrid Wireless Controller
 6.30.223.248 (r487574)


Can you provide the output of 'pciconf -lvbc pci0:3:0:0' on the FreeBSD host?

This is assuming pci0:3:0:0 is the wlan device being passed through
based on an earlier email.

best
Neel


 After that, there is a wlan0 device but I am not sure whether it works.
 There are no new kernel warnings or error messages but no matter what I do
 (scan, create an AP etc) there is zero reaction. But that might a
 problematic driver that the VM is using. That's on Linux 3.16.something.

 I have tried to load another driver and there is a complain regarding PCI
 which is common to both drivers:

 [ 1515.692558] bcma-pci-bridge :00:07.0: can't derive routing for PCI
 INT A
 [ 1515.692568] bcma-pci-bridge :00:07.0: PCI INT A: no GSI

 Didnt test further with this driver.

 On FreeBSD, I tried to do the same with an ndis windows driver. It didn't
 get very far because some windows drivers and ndis and amd64 don't like each
 other very much:)

  Then I start bhyve ... -s 10,passthru,2/0/0 ... and it spits:
 Unsupported MSI-X configuration: 2/0/0
 and quits.

 ppt0@pci0:2:0:0:class=0x02 card=0x06471025 chip=0x16b514e4
 rev=0x10 hdr=0x00


 I will try to reproduce the problem if I get hold of this device.


 Great, thank you.

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Unsupported cmd:c4 (ada0:ahcich0:0:0:0): READ_MUL. ACB: c4 00 00 00 00 40 00 00 00 00 01 00

[Neel Natu]

Hi Bjorn,

On Fri, Nov 28, 2014 at 2:42 AM, Ask Bjørn Hansen a...@develooper.com wrote:
 Hi everyone,

 I’ve spent some time off this week upgrading my 
 build-my-personal-NanoBSD-image tool from a VMware virtual machine to a 
 physical box running a 32-bit build-box under bhyve.

 So far so good — all that has been working really well.

 My nanobsd image is only used on Soekris and PC Engines boxes and my 
 production images are really small (some 100MB storage and ~32MB memory), so 
 I have a crufty kernel config that excluded a bunch of stuff I (thought I) 
 don’t need.

 Now I’d like to run my built image under bhyve for testing. It’s easier for 
 me if the disk comes up as an ada device, so I’m using ahci-hd instead of 
 virtio-blk. With virtio-blk the system boots but the device has the “wrong 
 name.

 With ahci-hd it can’t read my partition for some reason.

 I’ve tried with both file and zvol backing store.

 This is the error I’m getting:

 ada0: BHYVE SATA DISK 001 ATA-8 SATA 2.x device
 ada0: Serial Number 123456
 ada0: 300.000MB/s transfers (SATA 2.x, PIO4, PIO 8192bytes)
 ada0: Command Queueing enabled
 ada0: 260MB (532480 512 byte sectors: 16H 63S/T 528C)
 ada0: Previously was known as ad4
 Unsupported cmd:c4
   (ada0:ahcich0:0:0:0): READ_MUL. ACB: c4 00 c1 1f 08 40 00 
 00 00 00 04 00
 (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
 (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT )
 (ada0:ahcich0:0:0:0): RES: 41 04 00 00 00 40 00 00 00 00 00
 (ada0:ahcich0:0:0:0): Retrying command
 Unsupported cmd:c4


 Any suggestion for what I’m doing wrong? Did I compile my kernel with some 
 idiotic set of not-working options?


Are you, by any chance, setting the tunable 'hw.ata.ata_dma' to 0 to
disable DMA?

I ask this because in the dmesg above you have:
ada0: 300.000MB/s transfers (SATA 2.x, PIO4, PIO 8192bytes)

In my setup, I get this instead:
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)

best
Neel


 Ask

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: 'repz' instruction not supported.

[Neel Natu]

Hi Ashutosh,

On Wed, Nov 19, 2014 at 10:30 PM, Ashutosh Kumar mrashut...@hotmail.com wrote:
 Hi
 We have found that emulation of 'repz' instruction is not supported in bhyve. 
 This is causing VM_EXIT for Guest OS. Do we have plans to add support for 
 instructions like 'repz'.


'repz' is actually an instruction prefix. The two instructions with
which 'repz' is allowed are 'cmps' and 'scas'. Both these instructions
typically don't need to be emulated since they operate on strings in
regular guest memory.

Is your guest emitting these instructions to access MMIO? Can you
share an objdump with the offending instructions?

best
Neel

 RegardsAshutosh
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: 'repz' instruction not supported.

[Neel Natu]

Hi Ashutosh,

On Fri, Nov 21, 2014 at 2:07 AM, Ashutosh Kumar mrashut...@hotmail.com wrote:
 Yes you are right 'repz' prefixes is added to the ‘cmps’ and 'scas'
 instructions.

 The actual opcode is “F3 A5” which is ‘rep movs’ (disassembly is showing
 ‘repz’).



 Yes the guest is emitting these instructions to access MMIO (local apic)


Bummer.

I was hoping that it was a prefix stuck by the compiler for
optimization (that could be ignored):
http://repzret.org/p/repzret/

In any case, the primitives you will need to emulate MOVS exist in
vmm_instruction_emul.c. I don't have any plans to work on this but
would be happy to review/submit the patch that does the emulation.

best
Neel



 Regards
 Ashutosh


 Date: Fri, 21 Nov 2014 00:08:15 -0800
 Subject: Re: 'repz' instruction not supported.
 From: neeln...@gmail.com
 To: mrashut...@hotmail.com
 CC: freebsd-virtualization@freebsd.org


 Hi Ashutosh,

 On Wed, Nov 19, 2014 at 10:30 PM, Ashutosh Kumar mrashut...@hotmail.com
 wrote:
  Hi
  We have found that emulation of 'repz' instruction is not supported in
  bhyve. This is causing VM_EXIT for Guest OS. Do we have plans to add 
  support
  for instructions like 'repz'.
 

 'repz' is actually an instruction prefix. The two instructions with
 which 'repz' is allowed are 'cmps' and 'scas'. Both these instructions
 typically don't need to be emulated since they operate on strings in
 regular guest memory.

 Is your guest emitting these instructions to access MMIO? Can you
 share an objdump with the offending instructions?

 best
 Neel

  RegardsAshutosh
  ___
  freebsd-virtualization@freebsd.org mailing list
  http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
  To unsubscribe, send any mail to
  freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance?

[Neel Natu]

Hi Tinker,

On Wed, Nov 12, 2014 at 2:08 PM, Tinker ti...@openmailbox.org wrote:
 On 2014-11-12 23:55, Allan Jude wrote:

 On 2014-11-12 16:39, ti...@openmailbox.org wrote:

 Hi!

 In order justify giving energy to BHyVe, I need to know if it's
 future-proof in that the 16 vCPU limit can be increased?

 Please let the world know if BHyVe's 16 vCPU limit can be lifted in some
 way, by configuration, patch, etc. (and if you want to, why this limit
 is in place by default today).

 Thanks!!
 Tinker

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org


 You can increase the limit by editing sys/amd64/include/vmm.h

 #define VM_MAXCPU   16

 From what I've been told, things scale badly above 24 CPUs. They plan to
 solve this issue, but have not yet. If you system has enough cores to
 support using more than 16 per VM, you can modify the file and recompile
 the kernel and use as many CPUs as you want, but not much testing has
 happened with bigger numbers.


 Dear Allan,

 Thank you very much for responding.

 Aha, great!


 Only for completeness, do you have any particular idea about
  * When the scaling above 24 vCPU:s will be optimized, like approx how many
 years away is it (like 1 or more than 1)?, and

bhyve allocates memory for the maximum number of vcpus when the
virtual machine is created. This is simple to implement and also fits
in nicely with bhyve's loader model where the guest loader
(bhyveload/grub-bhyve) is separate from bhyve.

The downside is that if VM_MAXCPU is a large number then there is a
large memory penalty for virtual machines that only use 1 or 2 vcpus.
Hence the desire to cap it at a smallish number.

There are a few ways to deal with this:
- patch the code to change VM_MAXCPU (this is what you need to do today)
- allow the maximum vcpus to be changed via a tunable (this can be
done in short order)
- the limit can go away when bhyve moves to a single process model
because we'll know the number of vcpus at VM creation time.

  * What the technological reason for the scaling is, is it that somehow the
 BHyVe instances on the different cores need to inter-communicate, for
 instance that all disk and network IO is done via one single core currently?


Actually, the performance depends more on the workload and the guest
OS so you should definitely try how it goes for your application.
We'll be happy to help fix any issues that come up.

best
Neel


 In all cases, your response is great news, as your baseline answer that it's
 doable and only a question of optimization and tweaking of present
 functionality -

 Thanks again!

 Tinker

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: moving from virtualbox to bhyve

[Neel Natu]

Hi John,

On Mon, Oct 20, 2014 at 9:59 PM, John freebsd-li...@potato.growveg.org wrote:
 On Mon, Oct 20, 2014 at 05:55:13PM -0700, Neel Natu wrote:

 It seems that you are booting the guest from the disk that was
 originally used with vbox.

 Yes, this is what I want to do, because there's data on there.


If you are comfortable sharing your disk image then I can try to
reproduce locally and hopefully get a better grip on what's happening.

best
Neel

 Do you have the same issue booting from an ubuntu ISO and doing a
 fresh install to disk in bhyve?

 No, that seems to work.

 cheers,
 --
 John
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Re[2]: ACPI Exception on shutdown

[Neel Natu]

[Adding virtualizat...@freebsd.org]

On Sat, Sep 27, 2014 at 12:12 AM, Denis Menshikov da...@inbox.ru wrote:
 Good day,

 Here is more information about host and guest system:

 host:

 FreeBSD vmserver 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16
 22:34:59 UTC 2014 r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
 amd64


Ok, that explains it. ACPI poweroff support in bhyve is not present in
10.0 RELEASE. It will be present in the upcoming 10.1 RELEASE though.

If you don't want to wait until 10.1 is released you can upgrade your
host to 10-stable or -current.

 guest:

 creating...

 sudo bhyveload -m 256 -d FreeBSD-10.0-RELEASE-amd64-disc1.iso testvm
 sudo bhyve -c 2 -s 0,amd_hostbridge -s 1,lpc -s 1:1,ahci-hd,test_hdd -s
 2,ahci-cd,FreeBSD-10.0-RELEASE-amd64-disc1.iso -l com1,/dev/nmdm1A -a -A -H
 -P -m 256 -I testvm
 sudo bhyvectl --vm=testvm --destroy
 sudo bhyveload -m 256 -d test_hdd testvm
 sudo bhyve -c 2 -s 0,amd_hostbridge -s 1,lpc -s 1:1,ahci-hd,test_hdd -s -l
 com1,/dev/nmdm1A -a -A -H -P -m 256 -I testvm

 uname -a

 FreeBSD test 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16
 22:34:59 UTC 2014 r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC
 amd64

 shutdown like:

 shutdown -p now

 and then:

 root@test:~ # shutdown -p now
 Shutdown NOW!
 shutdown: [pid 615]
 root@test:~ #
 *** FINAL System shutdown message from root@test ***

 System going down IMMEDIATELY



 System shutdown time has arrived
 Sep 27 10:57:39 test shutdown: power-down by root:
 Stopping cron.
 Waiting for PIDS: 559.
 Stopping sshd.
 Waiting for PIDS: 524.
 Stopping devd.
 Waiting for PIDS: 301.
 Writing entropy file:.
 .
 Terminated
 Sep 27 10:57:39 test syslogd: exiting on signal 15
 Waiting (max 60 seconds) for system process `vnlru' to stop...done
 Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
 Waiting (max 60 seconds) for system process `syncer' to stop...
 Syncing disks, vnodes remaining...4 3 1 0 0 done
 All buffers synced.
 Uptime: 3m11s
 ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S5_]
 (20130823/hwxface-637)
 acpi0: AcpiEnterSleepStatePrep failed - AE_NOT_FOUND

 The operating system has halted.
 Please press any key to reboot.

 Rebooting...
 cpu_reset: Stopping other CPUs

 Best regards, Denis
 P.S. Why the message cpu_reset: Stopping other CPUs is displayed on my
 terminal, it correctly?


Yes, that is normal.

best
Neel

 Fri, 26 Sep 2014 17:36:18 -0700 от Neel Natu neeln...@gmail.com:

 Hi Denis,

 On Fri, Sep 26, 2014 at 12:25 PM, Denis Menshikov da...@inbox.ru wrote:
 Good day,

 faced with this problem: when I shutdown the virtual machine (FreeBSD
 amd64 release-guest) power-off does not occur, a message is displayed:

 ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S5_]
 (20130823/hwxface-637)
 acpi0: AcpiEnterSleepStatePrep failed - AE_NOT_FOUND

 The virtual machine is running with the following command:
 sudo bhyve -c 2 -s 0,amd_hostbridge -s 1,lpc -s 1:1,ahci-hd,test_hdd -l
 com1,stdio -a -A -H -P -m 256 -I testvm

 What am I doing wrong?


 Can you give a few more details about your setup so it can be reproduced?

 - Output of 'uname -a' on the host (or the release/svn revision number)
 - Output of 'uname -a' on the guest (or the release/svn revision number)
 - How are you shutting down the guest? Are you issuing a poweroff
 command from the guest or are you sending a SIGTERM to the bhyve
 process?

 best
 Neel

 --
 С уважением Denis Menshikov
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org



 --
 С уважением Denis Menshikov
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: ACPI Exception on shutdown

[Neel Natu]

Hi Denis,

On Fri, Sep 26, 2014 at 12:25 PM, Denis Menshikov da...@inbox.ru wrote:
  Good day,

 faced with this problem: when I shutdown the virtual machine (FreeBSD amd64 
 release-guest) power-off does not occur, a message is displayed:

 ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S5_] 
 (20130823/hwxface-637)
 acpi0: AcpiEnterSleepStatePrep failed - AE_NOT_FOUND

 The virtual machine is running with the following command:
 sudo bhyve -c 2 -s 0,amd_hostbridge -s 1,lpc -s 1:1,ahci-hd,test_hdd -l 
 com1,stdio -a -A -H -P -m 256 -I testvm

 What am I doing wrong?


Can you give a few more details about your setup so it can be reproduced?

- Output of 'uname -a' on the host (or the release/svn revision number)
- Output of 'uname -a' on the guest (or the release/svn revision number)
- How are you shutting down the guest? Are you issuing a poweroff
command from the guest or are you sending a SIGTERM to the bhyve
process?

best
Neel

 --
 С уважением Denis Menshikov
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Directly reserve an interrupt IDT entry for Hyper-V

[Neel Natu]

Hi John,

On Wed, Aug 20, 2014 at 8:31 AM, John Baldwin j...@freebsd.org wrote:
 On Wednesday, August 20, 2014 9:31:54 am Roger Pau Monné wrote:
 On 20/08/14 11:19, Wei Hu wrote:
  Hello,
 
  Sending to Xen, drivers and virtualization mailing lists since this might
 be of interest to the folks on these aliases.
 
  I am working for Microsoft to improve the performance of FreeBSD running
 on Hyper-V. Right now I am adding a feature in the vmbus driver which could
 handle the host-guest channel communications on all vCPUs simultaneously. In
 order to achieve this, the hypervisor will send same interrupt concurrently on
 all the vCPUs. The traditional way on FreeBSD to set up interrupt handling for
 devicse, such as calling bus_alloc_resource() to reserve an IRQ line, and then
 calling bus_setup_intr() to create a vector, doesn't seem to work in this
 case. It seems if the interrupt is routed via legacy IRQ, it can only be
 active on one vCPU at a time. In order to allow the same interrupt to be
 handled on all vCPUs concurrently, all I need is an IDT entry, not an IRQ
 line.
 
  I checked current FreeBSD code. It looks to me Xen directly uses the
 vector number IDT_EVTCHN (0x93) to achieve the same purpose. I am proposing
 both Xen and Hyper-V share this same vector. Following is a little bit detail
 of my proposal for the changes in the current kernel.
 
 
  1.   In machdep.c:
 
   #ifdef XENHVM
 
  setidt(IDT_EVTCHN, IDTVEC(xen_intr_upcall), SDT_SYSIGT, SEL_UPL,
 0);
 
  #else
 
  setidt(IDT_EVTCHN, IDTVEC(hv_vmbus_intr), SDT_SYSIGT, SEL_UPL,
 0);
 
  #endif
 
  2.   Apic_vector.S
 
  Add IDTVEC(hv_vmbus_intr) to call Hyper-V vmbus interrupt service routine.
 
  Any  thoughts, objections and feedbacks are all welcome.

 Hello,

 I don't think using the same IDT vector is the right approach, I would
 just pick a different IDT vector and use that for Hyper-V. Using the
 same IDT vector (like your suggestion above) would prevent shipping a
 kernel with with both Hyper-V and Xen support (like it's done now in
 GENERIC).

 Roger.

 Hmm, can't you make this a runtime check to only call setidt() if you detect
 you are under the appropriate hypervisor?

 Also, bhyve currently has a hackish way of requesting a free IDT slot.
 Perhaps it would be best if I added little API to reserve an IDT slot assuming
 that callers could accept a dynamic IDT vector rather than a static one.


Yup, it'll be good to get rid of vmm_ipi.c.

best
Neel

 --
 John Baldwin
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Problem with vmexit on mtrap

[Neel Natu]

Hi Martin,

On Sat, Aug 16, 2014 at 12:33 PM, Martin Steegmanns
mar...@unix-users.de wrote:
 On Tue, Aug 12, 2014 at 06:39:18PM -0700, Neel Natu wrote:
 The VM-exit instruction length field is valid only for a subset of VM
 exits. See section 27.2.4 Information for VM exits due to instruction
 execution in the Intel SDM.

 In particular, the instruction length is not guaranteed to be valid if
 the VM-exit is due to a hardware exception. Therefore it cannot be
 used to skip over the UD2 instruction.

 On my machine the VM-exit instruction length field was set to '2' for
 the first UD2 and '5' for the second UD2.

 OK, thx for the clarification.

 For this specific test, you can either hardcode the instruction length
 to '2' if the VM exit is due to a UD2 or use an instruction like OUT
 to a specific I/O port to trigger the monitor-trap-flag on and off. A
 VM-exit due to OUT will have the correct value in the VM-exit
 instruction length field.

 But this instruction length issue only affects my way to toggle
 the MTF bit. The MTF itself does not rely internally on the
 instruction length field, or does it?
 As far as I understand, bhyve does not need a valid instruction length
 for MTF, because the handler returns VMEXIT_RESTART. No need for bhyve
 to adjust the rip on vmentry.

 If I set the MTF bit via bhyvectl, the guest system still
 seems to enter a loop.
 My mtrap handler writes the RIP to a file, but all I see are high
 addresses e.g:

 0x806bf0b0 Xapic_isr1

 According to kdb, these are addresses point to Xapic_isr1 and
 interrupt handlers.

 I wonder if a vmexit caused by the MTF could overlay with another
 vmexit. With the MTF bit set, I expect the guest system to
 behave exactly as without the MTF bit. Of course slower due to
 single stepping :).


On my Xeon E5-2650 running at 2.0GHz a single vcpu VM is still not at
the login prompt after 7+ hours with MTRAP enabled.

However, it is making forward progress and is chugging through the
/etc/rc startup scripts very slowly.

best
Neel

 Regards,
 Martin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Problem with vmexit on mtrap

[Neel Natu]

Hi Martin,

On Tue, Aug 12, 2014 at 2:24 AM, Martin Steegmanns mar...@unix-users.de wrote:
 Hello list!
 I modified bhyve to toggle the MTF bit on an UD2 instruction.
 In a guest system, a program does:
 __asm__ __volatile__(
 UD2
 NOP
 xor %rax,%rax
 NOP
 UD2
 );

 On the first UD2, MTF bit is correctly set, but the second
 UD2 is never reached (waited some hours).
 If I manually reset the MTF bit via bhyvectl --setcap,
 guest executione reaches the second UD2 instruction.

 A diff of my modifications is attached to this mail.

 Am I missing something on vmenter that makes the guest loop forever?


The VM-exit instruction length field is valid only for a subset of VM
exits. See section 27.2.4 Information for VM exits due to instruction
execution in the Intel SDM.

In particular, the instruction length is not guaranteed to be valid if
the VM-exit is due to a hardware exception. Therefore it cannot be
used to skip over the UD2 instruction.

On my machine the VM-exit instruction length field was set to '2' for
the first UD2 and '5' for the second UD2.

For this specific test, you can either hardcode the instruction length
to '2' if the VM exit is due to a UD2 or use an instruction like OUT
to a specific I/O port to trigger the monitor-trap-flag on and off. A
VM-exit due to OUT will have the correct value in the VM-exit
instruction length field.

best
Neel

 Regards,
 Martin

 snippet of ktr dump
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1074 vm vm1[0]: 
 unhandled mtf vmexit at 0x804dde70
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1063 vm vm1[0]: Resume 
 execution at 0x804dde41
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:2652 vm vm1[0]: 
 returning from vmx_run: exitcode 6
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1074 vm vm1[0]: 
 unhandled mtf vmexit at 0x804dde41
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1063 vm vm1[0]: Resume 
 execution at 0x804dde39
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:2652 vm vm1[0]: 
 returning from vmx_run: exitcode 6
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1074 vm vm1[0]: 
 unhandled mtf vmexit at 0x804dde39
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1063 vm vm1[0]: Resume 
 execution at 0x804dde30
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:2652 vm vm1[0]: 
 returning from vmx_run: exitcode 6
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1074 vm vm1[0]: 
 unhandled mtf vmexit at 0x804dde30
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1063 vm vm1[0]: Resume 
 execution at 0x804dde27
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:2652 vm vm1[0]: 
 returning from vmx_run: exitcode 6
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1074 vm vm1[0]: 
 unhandled mtf vmexit at 0x804dde27
 /usr/src/sys/modules/vmm/../../amd64/vmm/intel/vmx.c:1063 vm vm1[0]: Resume 
 execution at 0x804dde1f
 /snippet

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: jenkins bhyve vms crashing and burning after several days of use

[Neel Natu]

Hi,

On Thu, Jun 26, 2014 at 3:43 PM, Neel Natu neeln...@gmail.com wrote:
 Hi Sean,

 On Thu, Jun 26, 2014 at 3:23 PM, Sean Bruno sbr...@ignoranthack.me wrote:
 On Thu, 2014-06-26 at 15:00 -0700, Neel Natu wrote:
 Hi Sean,

 On Thu, Jun 26, 2014 at 2:46 PM, Sean Bruno sbr...@ignoranthack.me wrote:
  On Thu, 2014-06-26 at 14:42 -0700, Sean Bruno wrote:
  so, we're seeing the bhyve vms running in the freebsd cluster for
  jenkins crashing and burning after a couple of days of use.
 
  vm exit[9]
  reason  VMX
  rip 0x29286336
  inst_length 3
  status  0
  exit_reason 49
  qualification   0x
  inst_type   0
  inst_error  0
 
 
  It looks like we have an active core file on havoc.ysv if you have a
  moment to look at it:
 
  http://people.freebsd.org/~sbruno/bhyve.core
 
  FreeBSD havoc.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #2
  r267362: Wed Jun 11 14:56:34 UTC 2014
  sbr...@havoc.freebsd.org:/usr/obj/usr/src/sys/HAVOC  amd64
 
 
  Also, from chaos.ysv
 
  http://people.freebsd.org/~sbruno/bhyve.core.chaos
 
  FreeBSD chaos.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1
  r267362: Wed Jun 11 15:50:24 UTC 2014
  sbr...@chaos.ysv.freebsd.org:/usr/obj/usr/src/sys/CHAOS  amd64
 

 Can you tell us the processor and memory configuration on havoc and chaos?

 Also, could you execute the following commands on havoc:

 # bhyvectl --vm=vmname --cpu=9 --get-vmcs-guest-physical-address
 -- this will output the offending guest physical address that
 triggered the EPT misconfiguration

 # bhyvectl --vm=vmname --get-gpa-pmap=gpa_from_above
 -- this will output the page table entries in the EPT that map to the
 offending GPA

 Hopefully that provides us with something to work with.

 best
 Neel

 

 chaos:
 CPU: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (2200.05-MHz K8-class CPU)
   Origin=GenuineIntel  Id=0x206d6  Family=0x6  Model=0x2d  Stepping=6
 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE
 Features2=0x1fbee3ffSSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX
   AMD Features=0x2c100800SYSCALL,NX,Page1GB,RDTSCP,LM
   AMD Features2=0x1LAHF
   TSC: P-state invariant, performance statistics
 avail memory = 66298322944 (63227 MB)

 havoc:
 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
 CPU: Intel(R) Xeon(R) CPU   E5620  @ 2.40GHz (2400.14-MHz
 K8-class CPU)
   Origin=GenuineIntel  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE
 Features2=0x29ee3ffSSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI
   AMD Features=0x2c100800SYSCALL,NX,Page1GB,RDTSCP,LM
   AMD Features2=0x1LAHF
   TSC: P-state invariant, performance statistics
 avail memory = 16571621376 (15803 MB)


 Thanks, we'll see if there are relevant errata for these processors.


Actually these processors have entirely different microarchitectures
(Nehalem and Sandybridge) so its unlikely that this is due to
processor errata.


 There appear to be three vms running on havoc:
 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm1 --cpu=9
 --get-vmcs-guest-physical-address
 gpa[9]  0x
 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm2 --cpu=9
 --get-vmcs-guest-physical-address
 gpa[9]  0x
 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm3 --cpu=9
 --get-vmcs-guest-physical-address
 gpa[9]  0x

 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm1 --cpu=9
 --get-gpa-pmap=0x
 gpa 0: 0x32c936e007 0x32c9353007 0x32c9352007 0

 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm2 --cpu=9
 --get-gpa-pmap=0x
 gpa 0: 0x3286cb0007 0x33ad105007 0x319b1fd007 0

 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm3 --cpu=9
 --get-gpa-pmap=0x
 gpa 0: 0x32c9348007 0x32c9339007 0


 But there's no information available on chaos at the moment as there are
 no active vms running.


 Sorry, I should explained a bit more.

 After a bhyve(8) exits because of the EPT misconfiguration error there
 are breadcrumbs left over in the VMCS as well as the nested page
 tables. We can use them to diagnose what happened.

 The bhyvectl commands above should be executed after the VM exits but
 before it is restarted again. Once it restarts, the breadcrumbs get
 written over and are of no use.

 The --vm=vmname passed to the bhyvectl command should be of the
 virtual machine that crashed.
 The --cpu=vcpuid passed to the bhyvectl command should be the
 vcpuid that detected the EPT misconfiguration. The reason I used '9'
 as an example above was because you saw this on the console:

 vm exit

Re: jenkins bhyve vms crashing and burning after several days of use

[Neel Natu]

Hi Sean,

On Thu, Jun 26, 2014 at 2:46 PM, Sean Bruno sbr...@ignoranthack.me wrote:
 On Thu, 2014-06-26 at 14:42 -0700, Sean Bruno wrote:
 so, we're seeing the bhyve vms running in the freebsd cluster for
 jenkins crashing and burning after a couple of days of use.

 vm exit[9]
 reason  VMX
 rip 0x29286336
 inst_length 3
 status  0
 exit_reason 49
 qualification   0x
 inst_type   0
 inst_error  0


 It looks like we have an active core file on havoc.ysv if you have a
 moment to look at it:

 http://people.freebsd.org/~sbruno/bhyve.core

 FreeBSD havoc.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #2
 r267362: Wed Jun 11 14:56:34 UTC 2014
 sbr...@havoc.freebsd.org:/usr/obj/usr/src/sys/HAVOC  amd64


 Also, from chaos.ysv

 http://people.freebsd.org/~sbruno/bhyve.core.chaos

 FreeBSD chaos.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1
 r267362: Wed Jun 11 15:50:24 UTC 2014
 sbr...@chaos.ysv.freebsd.org:/usr/obj/usr/src/sys/CHAOS  amd64


Can you tell us the processor and memory configuration on havoc and chaos?

Also, could you execute the following commands on havoc:

# bhyvectl --vm=vmname --cpu=9 --get-vmcs-guest-physical-address
-- this will output the offending guest physical address that
triggered the EPT misconfiguration

# bhyvectl --vm=vmname --get-gpa-pmap=gpa_from_above
-- this will output the page table entries in the EPT that map to the
offending GPA

Hopefully that provides us with something to work with.

best
Neel


 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: jenkins bhyve vms crashing and burning after several days of use

[Neel Natu]

Hi Sean,

On Thu, Jun 26, 2014 at 3:23 PM, Sean Bruno sbr...@ignoranthack.me wrote:
 On Thu, 2014-06-26 at 15:00 -0700, Neel Natu wrote:
 Hi Sean,

 On Thu, Jun 26, 2014 at 2:46 PM, Sean Bruno sbr...@ignoranthack.me wrote:
  On Thu, 2014-06-26 at 14:42 -0700, Sean Bruno wrote:
  so, we're seeing the bhyve vms running in the freebsd cluster for
  jenkins crashing and burning after a couple of days of use.
 
  vm exit[9]
  reason  VMX
  rip 0x29286336
  inst_length 3
  status  0
  exit_reason 49
  qualification   0x
  inst_type   0
  inst_error  0
 
 
  It looks like we have an active core file on havoc.ysv if you have a
  moment to look at it:
 
  http://people.freebsd.org/~sbruno/bhyve.core
 
  FreeBSD havoc.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #2
  r267362: Wed Jun 11 14:56:34 UTC 2014
  sbr...@havoc.freebsd.org:/usr/obj/usr/src/sys/HAVOC  amd64
 
 
  Also, from chaos.ysv
 
  http://people.freebsd.org/~sbruno/bhyve.core.chaos
 
  FreeBSD chaos.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1
  r267362: Wed Jun 11 15:50:24 UTC 2014
  sbr...@chaos.ysv.freebsd.org:/usr/obj/usr/src/sys/CHAOS  amd64
 

 Can you tell us the processor and memory configuration on havoc and chaos?

 Also, could you execute the following commands on havoc:

 # bhyvectl --vm=vmname --cpu=9 --get-vmcs-guest-physical-address
 -- this will output the offending guest physical address that
 triggered the EPT misconfiguration

 # bhyvectl --vm=vmname --get-gpa-pmap=gpa_from_above
 -- this will output the page table entries in the EPT that map to the
 offending GPA

 Hopefully that provides us with something to work with.

 best
 Neel

 

 chaos:
 CPU: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (2200.05-MHz K8-class CPU)
   Origin=GenuineIntel  Id=0x206d6  Family=0x6  Model=0x2d  Stepping=6
 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE
 Features2=0x1fbee3ffSSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX
   AMD Features=0x2c100800SYSCALL,NX,Page1GB,RDTSCP,LM
   AMD Features2=0x1LAHF
   TSC: P-state invariant, performance statistics
 avail memory = 66298322944 (63227 MB)

 havoc:
 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
 CPU: Intel(R) Xeon(R) CPU   E5620  @ 2.40GHz (2400.14-MHz
 K8-class CPU)
   Origin=GenuineIntel  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE
 Features2=0x29ee3ffSSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI
   AMD Features=0x2c100800SYSCALL,NX,Page1GB,RDTSCP,LM
   AMD Features2=0x1LAHF
   TSC: P-state invariant, performance statistics
 avail memory = 16571621376 (15803 MB)


Thanks, we'll see if there are relevant errata for these processors.


 There appear to be three vms running on havoc:
 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm1 --cpu=9
 --get-vmcs-guest-physical-address
 gpa[9]  0x
 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm2 --cpu=9
 --get-vmcs-guest-physical-address
 gpa[9]  0x
 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm3 --cpu=9
 --get-vmcs-guest-physical-address
 gpa[9]  0x

 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm1 --cpu=9
 --get-gpa-pmap=0x
 gpa 0: 0x32c936e007 0x32c9353007 0x32c9352007 0

 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm2 --cpu=9
 --get-gpa-pmap=0x
 gpa 0: 0x3286cb0007 0x33ad105007 0x319b1fd007 0

 r...@havoc.ysv:/home/sbruno # bhyvectl --vm=vm3 --cpu=9
 --get-gpa-pmap=0x
 gpa 0: 0x32c9348007 0x32c9339007 0


 But there's no information available on chaos at the moment as there are
 no active vms running.


Sorry, I should explained a bit more.

After a bhyve(8) exits because of the EPT misconfiguration error there
are breadcrumbs left over in the VMCS as well as the nested page
tables. We can use them to diagnose what happened.

The bhyvectl commands above should be executed after the VM exits but
before it is restarted again. Once it restarts, the breadcrumbs get
written over and are of no use.

The --vm=vmname passed to the bhyvectl command should be of the
virtual machine that crashed.
The --cpu=vcpuid passed to the bhyvectl command should be the
vcpuid that detected the EPT misconfiguration. The reason I used '9'
as an example above was because you saw this on the console:

vm exit[9]
reason  VMX
rip 0x29286336

Hope that helps.

best
Neel

 sean

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd

Re: interrupt storm on ahci

[Neel Natu]

Hi Steve,

On Sat, Jun 7, 2014 at 2:28 PM, Peter Grehan gre...@freebsd.org wrote:
 Hi Steve,


 I'm running a FreeBSD guest in bhyve on a FreeBSD host. Both are running
 FreeBSD CURRENT, r266947. I've gotten this message about 12 times since
 boot:

 interrupt storm detected on irq268:; throttling interrupt source

 vmstat -i shows:

 irq268: ahci1  236514222839

 ahci1 is the second disk connected to the system:

 ahci1: Intel ICH8 AHCI SATA controller mem 0xc0002400-0xc00027ff irq 18
 at device 4.0 on pci0

 The VM itself runs poudriere and was building a bunch of packages. At the
 moment, the VM seems to be in a rather odd state. The poudriere jails are
 running, but not doing anything. Ideas?


  Is this an 8.* host ? I don't believe AHCI has MSI support on that version,
 and AHCI legacy interrupts haven't had a huge amount of testing under load.

  If it is 8.*, I'd recommend using virtio-blk for the block device until we
 can work out what's going wrong.


The KTR trace would be useful to figure out what's happening.

To do that you can compile the host kernel and vmm.ko with the
following options:
options KTR
options KTR_MASK=(KTR_GEN)
options KTR_ENTRIES=(4*1024*1024)

And when you see the interrupt storm message in the guest you can execute:
sudo ktrdump -cto /tmp/ktrdump.out

best
Neel

 later,

 Peter.


 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: SR-IOV Patch Series 2/7: bhyve integration

[Neel Natu]

Hi Ryan,

On Mon, May 26, 2014 at 6:51 PM, Ryan Stone ryst...@gmail.com wrote:
 The bhyve work to interoperate is quite simple.  PCI Passthrough
 through the ppt driver works just fine with SR-IOV VFs.  The three
 changes are:

 - allow ppt devices to detach.  This would happen if the administrator
 destroyed VFs with iovctl -D
 - allow the SR-IOV infrastructure to force the ppt driver to attach to
 certain devices by using the special BUS_PROBE_NOWILDCARD return value
 from device_probe().  This will have no effect on regular devices, but
 the SR-IOV infrastructure can create VF devices in a way where
 BUS_PROBE_NOWILDCARD will match the device.  This allows passthrough
 of a given VF to be configured through iovctl (see ioctl.conf(5) for
 an example of how to do this).
 - Currently if vmm.ko is loaded and no ppt devices are present, the
 IOMMU is not enabled.  If VFs are subsequently created as passthrough
 devices it will not be possible to pass them through to VMs.  This
 patch adds a new tunable that will force the IOMMU to be enabled.

 Note: Currently ppt suffers from a lack of locking.  This is now a
 problem because ppt devices can be destroyed, so there would be races
 here.  I tried to address this by adding a mutex but that failed in
 ways that I could not figure out how to resolve (basically, some parts
 of ppt are called into from within a critical section, so a spinlock
 is needed, but other parts of ppt call into the VM subsystem to do
 M_WAITOK allocations, so an sx lock is needed.  It wasn't at all
 obvious to me how to resolve that conflict, so I'm hoping that a more
 experienced bhyve person has an idea).


Your changes look good.

Also, yes, locking is now an issue. Let me try a couple of things and
see how to fix this properly.

best
Neel


 http://people.freebsd.org/~rstone/patches/iov/0004-Allow-passthrough-devices-to-be-hinted.patch
 [PATCH 04/21] Allow passthrough devices to be hinted.

 Allow the ppt driver to attach to devices that were hinted to be
 passthrough devices by the PCI code creating them with a driver
 name of ppt.

 Add a tunable that allows the IOMMU to be forced to be used.  With
 SR-IOV passthrough devices the VFs may be created after vmm.ko is
 loaded.  The current code will not initialize the IOMMU in that
 case, meaning that the passthrough devices can't actually be used.
 ---
  sys/amd64/vmm/io/ppt.c | 72 
 +++---
  sys/amd64/vmm/vmm.c|  7 -
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: [PATCH] Flexible vcpu pinning configuration

[Neel Natu]

Hi Roman,

On Thu, May 1, 2014 at 7:30 AM, Roman Bogorodskiy no...@freebsd.org wrote:
   Neel Natu wrote:

 Hi Roman,

 On Sun, Apr 27, 2014 at 3:45 AM, Roman Bogorodskiy no...@freebsd.org wrote:
  I've created an initial version of the patch which allows more flexible
  vcpu pinning configuration.
 
  Current schema is:
 
  bhyve -p N
 
  pins vcpu i to hostcpu N + i.
 
  The propsed extension is:
 
  bhyve -p N:M  -p 0:1 -p 3:5
 
  which pins vcpu N to host pcpu M. Option needs to be specified
  individually for each vcpu.
 
  So it works like that for me:
 
  sudo /usr/sbin/bhyve -p 0:0 -p 1:3 -c 2 ...
 
  # sudo cpuset -g -t 100262
  tid 100262 mask: 0
  # sudo cpuset -g -t 100264
  tid 100264 mask: 3
 
  PS I used cpumat_t* array to store these values instead of int, because
  if the idea is OK, I'll extend it to support ranges like e.g. cpuset(1)
  supports, e.g.: 1:2-5.
 
  The questions are:
 
   - Is it OK to chance '-p' arg syntax or it's better to introduce a new
 one?
 

 I think we can reuse the -p option unless anybody objects vociferously.

   - Is the syntax OK (currently: 'vcpu:pcpu', later
 'vcpu:pcpuN-pcpuM,pcpuX)?

 Yup, I think that works fine.

 The patch looks good in general but I have a few comments:

 - Scope of 'vcpupmap[]' should be restricted to 'static'.

 - usage() and man page need to be updated.

 - pincpu_parse():
 The option parsing can be made much easier by using:

   if (sscanf(str, %d:%d, vcpu, pcpu) == 2) {
   /* success */
   } else {
   return (-1);
   }

 If the same vcpu is specified multiple times then we should
 malloc(sizeof(cpuset_t)) only the first time:

   if (vcpumap[vcpu] != NULL)
   mask = vcpumap[vcpu];
   else
   mask = malloc(sizeof(cpuset_t));

 We need to range-check 'vcpu' before using it as an index into the
 'vcpumap[]' array.

 best
 Neel

 Attached an updated patch.

A slightly modified version was submitted:
http://svnweb.freebsd.org/base?view=revisionrevision=265376


 I'm still inclined to use something like parselist() from
 usr.bin/cpuset/cpuset.c, but I don't want to copy/paste and I don't know
 where it'd make sense to move it so it was usable outside of cpuset?


Hmm, not sure really … but you can get started by copying it into
bhyve and if there is a better place to put it we can do that too.

 PS While reading bhyverun.c, I think I spotted a typo: in fbsdrun_deletecpu()
 error message says fprintf(stderr, addcpu:  Should it be deletecpu: 
 instead?

Thanks. This is fixed now:
http://svnweb.freebsd.org/base?view=revisionrevision=265366

best
Neel


 Roman Bogorodskiy
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve and serial port emulation problem

[Neel Natu]

Hi Olivier,

On Wed, Apr 30, 2014 at 3:28 PM, Olivier Cochard-Labbé
oliv...@cochard.me wrote:
 On Wed, Apr 30, 2014 at 7:33 PM, Neel Natu neeln...@gmail.com wrote:

 Hi Olivier,


 Can you share the command line options you pass the bhyve(8)?


 Hi Neel,

 sure, here it is:

 bhyveload -m 256M -d /tmp/BSDRP/BSDRP_1 -c /dev/nmdm1A BSDRP_1
 bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc -s
 2:0,virtio-net,tap0,mac=58:9c:fc:01:00:01 -s
 1:0,virtio-blk,/tmp/BSDRP/BSDRP_1 -l com1,/dev/nmdm1A BSDRP_1


The UPDATING entry in r264175 describes how to fix this:
http://svnweb.freebsd.org/base?view=revisionrevision=264175

Briefly, you will now need to change your /etc/ttys to look like this:
ttyu0   /usr/libexec/getty 3wire vt100   on secure

If your virtual machine has an older world then your /etc/gettytab
will not have the entry for 3wire and you may want to use
3wire.9600 instead.

best
Neel

 Regards,

 Olivier
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve and serial port emulation problem

[Neel Natu]

Hi Olivier,

On Tue, Apr 29, 2014 at 2:12 PM, Olivier Cochard-Labbé
oliv...@cochard.me wrote:
 Hi,

 I'm using the same nanobsd image on Virtualbox and bhyve (-CURRENT r265101).

 The nanobsd image I'm using is configured for serial port as default
 console:

 [root@nanobsd]~# cat /boot.config
 -S38400 -h
 [root@nanobsd]~# grep ttyu0 /etc/ttys
 ttyu0   /usr/libexec/getty std.38400  vt100   on secure

 The Virtualbox VM is configured for creating a serial pipe:
 VBoxManage modifyvm nanobsd --uart1 0x3F8 4 --uartmode1 server
 /tmp/nanobsd.serial
 and I connect to this guest using socat unix-connect:/tmp/nanobsd.serial
 stdio,raw,echo=0,icanon=0
 = And it's works without problem.

 The bhyve is configured with a com1 redirecting to a nmdm device: -l
 com1,/dev/nmdm1
 and I connect to this guest using cu -l /dev/nmdm1B

 = But this bhyve console didn't works correctly: I can't use tool like
 less/more (even ssh-keygen can't ask for a passphrase).
 If I run truss with these tools, It seems they hang for opening /dev/tty.

 Some differences between virtualbox and bhyve (don't know if it can help):

 [root@vbox_guest]~# kenv | grep console
 comconsole_pcidev=
 comconsole_port=1016
 comconsole_speed=9600
 console=comconsole

 [root@bhyve_guest]~# kenv | grep console
 console=userboot


Can you share the command line options you pass the bhyve(8)?

best
Neel

 Regards,

 Olivier
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Google Summer of Code 2014 question

[Neel Natu]

Hi John,

On Wed, Mar 19, 2014 at 2:36 PM, John Matty jcm...@pitt.edu wrote:
 Hello everybody,

 My name is John, and I was interested in doing Google Summer of Code for
 FreeBSD this Summer.  I am currently interested in writing a virtual machine
 execution language.  A rough draft of my proposal can be found on the
 website for GSoC 2014 (I am not exactly sure how to get to it from the
 outside though, sorry), however the brief description of my project is to
 specify and implement a domain specific language to start a virtual machine,
 execute given commands, and store the results of the given command. Then,
 using a text comparison tool (maybe something similar to `diff'), compare
 the obtained output to the expected output.

 I was currently thinking that this would mainly be used for doing repeated
 testing of software on FreeBSD running in a virtual machine, however I was
 interested to see if anyone had any ideas for other uses or applications for
 this language so that I could focus on making it as useful as possible for
 what people would actually be using it for, or if anyone had any other
 comments or suggestions?

 I was also wondering if anyone would be interested in mentoring me for my
 project this summer?

 Finally, as I am new to this mailing list, if this is not the correct place
 to ask this question or anything else like that please let me know so I can
 move my question to the correct place.

 Thank you!


I could certainly make use of this in bhyve development.

For e.g., whenever we make changes to bhyve there is a large test
matrix that needs to be addressed:
- various guest OSes: FreeBSD (various releases), Linux (various
flavors), OpenBSD etc.
- 32-bit or 64-bit virtual machine
- number of vcpus and guest memory sizes
- multiple device emulations (virtio, ahci, passthru)
- timer modes (PIT, LAPIC, HPET, ACPI)
- interrupt delivery (8259, legacy, MSI, MSI-X)
- shutdown types (reboot, halt, acpi power off)

A tool like the one you are proposing would be very useful to catch regressions.

best
Neel

 --John

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve core dump...

[Neel Natu]

Hi Andrea,

On Thu, Feb 27, 2014 at 6:58 AM, Andrea Brancatelli
abrancate...@schema31.it wrote:
 Hello guys.

 I was installing mysql56 in a freebsd10 inside freebsd 10 and while doing
 the install the machine bombed out. In the console this is what I got:

 Assertion failed: (aior != NULL), function ahci_handle_dma, file
 /usr/src/usr.sbin/bhyve/pci_ahci.c, line 445.
 Abort trap (core dumped)

 The VM was calculating the SHA256 of the distfile when it bombed out.
 Nothing else was going on inside the VM at that time.

 I even have the .core if it matters.


The core file would greatly help if you can make it available.

best
Neel

 Any idea?

 --




 *Andrea BrancatelliSchema 31 S.r.l. - Socio UnicoResponsabile ITROMA -
 FIRENZE - PALERMO ITALYTel: +39. 06.98.358.472*

 *Cell: +39 331.2488468Fax: +39. 055.71.880.466Società del Gruppo SC31
 ITALIA*
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Bhyve segfaults when accessing the network

[Neel Natu]

Hi Paul,

On Sun, Feb 23, 2014 at 7:57 AM, Paul Schenkeveld free...@psconsult.nl wrote:
 Hi,

 I tried to get bhyve to run on my home server but got consistent
 crashes as soon as I touch the network.

 Hardware: Supermicro X9SPV-F3610ME-O (embedded i5) with 16GB RAM

 Host kernels (+userland of the same version):
 10.0 RC5
 10.0 RELEASE
 10 STABLE

 The kernel has been stripped (unneeded drivers and stuff like DEBUG,
 AUDIT, CAPABILITIES, MAC removed)

 Guest operating systems:
 9.2 RELEASE
 10.0 RELEASE

 Procedure:
 # ifconfig tap0 create up
 # ifconfig bridge0 create up
 # ifconfig bridge0 addm tap0 addm em0
 # kldload vmm
 # mkdir /vm/test
 # cd /vm/test
 # ln -s /images/FreeBSD-10.0-RELEASE-amd64-disc1.iso release.iso
 # sh vmrun.sh -c 1 -m 512 -t tap0

 do a default install, no games/ports, IPv4 DHCP, no IPv6
 add 'console /usr/libexec/getty std.9600 unknown on secure' to
  /etc/ttys
 add 'ifconfig_vtnet0=DHCP' to /etc/rc.conf
 reboot guest

 At the point where DHCP sends the first query I get a segmentation fault
 and the guest stops.

 I can provoke the same segfault by not putting the DHCP line into
 rc.conf and trying to ifconfig vtnet0 or by adding tap0 to bridge0 in
 the host after the guest has stopped.

 Although it worked perfectly fine with bhyve a couple of months ago
 (with an older build of 10.0-CURRENT that I don't have here anymore), I
 suspected the hardware and tried the same on my Macbook Pro with Vmware
 Fusion with a 10.0 RELEASE guest.

 The stripped 10 STABLE guest segfaults in the same way, the other two
 both work.

 I bought the home server for building software for older releases
 (mostly FreeBSD 8 and 9), my Macbook with Fusion is too slow and often
 in use for other purposes.

 Is this behaviour seen by others?

 I can provide more details or core dumps if that helps locating/solving
 the problem.


I am not clear about what exactly is segfaulting: is it the bhyve
process or the guest kernel? Do you have a screenshot and/or a
backtrace of the segfaulting entity?

best
Neel

 With kind regards,

 Paul Schenkeveld
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Detect of BHyve VM was powered off or rebooted?

[Neel Natu]

Hi Craig,

 On Feb 11, 2014, at 2:29 PM, Craig Rodrigues rodr...@freebsd.org wrote:
 
 On Tue, Feb 11, 2014 at 11:11 AM, John Baldwin j...@freebsd.org wrote:
 
 On Sunday, February 09, 2014 7:03:41 pm Neel Natu wrote:
 Hi Craig,
 
 On Sun, Feb 9, 2014 at 1:11 PM, Craig Rodrigues rodr...@freebsd.org
 wrote:
 Hi,
 
 I posted some rc.d scripts that I am using to boot a BHyve VM
 and send the output to a serial console using the /dev/nmdm
 driver:
 
 http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-
 January/002040.html
 
 It works quite well.  There is some things I would like to improve,
 and would like some advice on the best way to do it.
 
 (1)  If the VM was destroyed with bhyvectl --destroy --vm ${VM_NAME},
  then I do not want to automatically restart the VM in the script.
  User should manually:  service bhyvevm start
 
 (2)  If the VM was powered down, via shutdown -p, or halt -p,
   then in my script I do not want to restart the VM in the script.
   User should manually:  service bhyvevm start
 
 (3)   If the VM was rebooted via reboot or shutdown -r,
   then I *do* want the script to restart the VM.
 
 I think if I change my start_vm.sh script to do something like:
 
 
 
 (
 while [ -e /dev/vmm/${VM} ]; do
/usr/sbin/bhyve -c 16 -m 8G -A -H -P -g 0 -s 0:0,hostbridge -s
 1:0,lpc
 -s 2:0,virtio-net,${TAP} -s 3:0,virtio-blk,${IMG} -l com1,${CONS_A}
 ${VM}
 done
 
 )  
 
 
 then this might cover cases (1) and (3), but what will cover
 case (2)?
 
 The exit code of the bhyve process will be 0 if it exited because the
 guest rebooted and will be non-zero if the guest did an acpi poweroff.
 You can use that to distinguish between cases (2) and (3).
 
 Having said that there are error conditions for which bhyve exits with
 a non-zero exit code. So, we'll need to explicitly define an exit code
 to distinguish between an acpi poweroff and these error conditions.
 
 OTOH, in all the cases when bhyve exits with a non-zero exit code, you
 will want to exit the loop which would treat it the same as shutdown -p.  I
 think you can just do this:
 
while [ -e /dev/vmm/${VM} ]; do
if ! bhyve ...; then
break
fi
done
 
 
 
 One question, if bhyve exits, do I have to call bhyveload again before
 calling bhyve?

Yes, that is correct. You will also need to destroy the VM after bhyve exits.

Best
Neel

 
 The /usr/share/examples/bhyve/vmrun.sh has a loop which does:
 
while [ 1 ] ; do
 
  ...
  bhyvectl --destroy
  bhyveload
  bhyve
 
   ...
done
 
 --
 Craig
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Detect of BHyve VM was powered off or rebooted?

[Neel Natu]

Hi Craig,

On Sun, Feb 9, 2014 at 1:11 PM, Craig Rodrigues rodr...@freebsd.org wrote:
 Hi,

 I posted some rc.d scripts that I am using to boot a BHyve VM
 and send the output to a serial console using the /dev/nmdm
 driver:

 http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-January/002040.html

 It works quite well.  There is some things I would like to improve,
 and would like some advice on the best way to do it.

 (1)  If the VM was destroyed with bhyvectl --destroy --vm ${VM_NAME},
   then I do not want to automatically restart the VM in the script.
   User should manually:  service bhyvevm start

 (2)  If the VM was powered down, via shutdown -p, or halt -p,
then in my script I do not want to restart the VM in the script.
User should manually:  service bhyvevm start

 (3)   If the VM was rebooted via reboot or shutdown -r,
then I *do* want the script to restart the VM.

 I think if I change my start_vm.sh script to do something like:



 (
  while [ -e /dev/vmm/${VM} ]; do
 /usr/sbin/bhyve -c 16 -m 8G -A -H -P -g 0 -s 0:0,hostbridge -s 1:0,lpc
 -s 2:0,virtio-net,${TAP} -s 3:0,virtio-blk,${IMG} -l com1,${CONS_A} ${VM}
  done

 )  


 then this might cover cases (1) and (3), but what will cover
 case (2)?

The exit code of the bhyve process will be 0 if it exited because the
guest rebooted and will be non-zero if the guest did an acpi poweroff.
You can use that to distinguish between cases (2) and (3).

Having said that there are error conditions for which bhyve exits with
a non-zero exit code. So, we'll need to explicitly define an exit code
to distinguish between an acpi poweroff and these error conditions.

best
Neel


 Thanks for any advice.

 --
 Craig
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Detect of BHyve VM was powered off or rebooted?

[Neel Natu]

Hi Craig,

On Sun, Feb 9, 2014 at 2:18 PM, Craig Rodrigues rodr...@freebsd.org wrote:
 On Sun, Feb 9, 2014 at 1:32 PM, Aryeh Friedman 
 aryeh.fried...@gmail.comwrote:




 On Sun, Feb 9, 2014 at 4:11 PM, Craig Rodrigues rodr...@freebsd.orgwrote:

 Hi,

 I posted some rc.d scripts that I am using to boot a BHyve VM
 and send the output to a serial console using the /dev/nmdm
 driver:


 http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-January/002040.html

 It works quite well.  There is some things I would like to improve,
 and would like some advice on the best way to do it.

 (1)  If the VM was destroyed with bhyvectl --destroy --vm ${VM_NAME},
   then I do not want to automatically restart the VM in the script.
   User should manually:  service bhyvevm start

 (2)  If the VM was powered down, via shutdown -p, or halt -p,
then in my script I do not want to restart the VM in the script.
User should manually:  service bhyvevm start

 (3)   If the VM was rebooted via reboot or shutdown -r,
then I *do* want the script to restart the VM.

 I think if I change my start_vm.sh script to do something like:



 (
  while [ -e /dev/vmm/${VM} ]; do
 /usr/sbin/bhyve -c 16 -m 8G -A -H -P -g 0 -s 0:0,hostbridge -s 1:0,lpc
 -s 2:0,virtio-net,${TAP} -s 3:0,virtio-blk,${IMG} -l com1,${CONS_A} ${VM}
  done

 )  


 then this might cover cases (1) and (3), but what will cover
 case (2)?

 Thanks for any advice.


 The options you gave (which are really the only ones available) do not
 distinguish between the reason for the termination of the instance of bhyve
 pointing to /dev/vmm/XXX  (it just does a normal termination).


 Did you play with any of these flags to bhyvectl?

[--get-vmcs-exit-ctls]
[--get-vmcs-exit-reason]
[--get-vmcs-exit-qualification]
[--get-vmcs-exit-interruption-info]

 If I do:
 bhyvectl --get-vmcs-exit-reason --vm vm1

 I get:
 vmcs_exit_reason[0] 0x001e

 If I look at:

 http://svnweb.freebsd.org/base/head/sys/amd64/vmm/intel/vmcs.h?view=markup#l310

 we have:
 #define EXIT_REASON_INOUT 30

 Linux has this stuff also:
 http://lxr.free-electrons.com/source/arch/x86/include/asm/vmx.h?v=3.1#L250

 so I guess these values are defined in some Intel manual.


Yes, but this an entirely different thing. The VMCS exit reason is
used to inform the hypervisor what condition caused the guest to stop
executing - e.g. a nested page fault.

Please don't use it to infer whether the guest wanted to reboot or poweroff :-)



 In my playing with this for almost a month the only solution seems to be
 put a trigger on instance shutdown via whatever method to leave a cookie
 file behind on the host (note I have not nor plan to do this in any of my
 own work).


 Yes, I can see why this might need to be done.  I might need to do that
 myself.  However, if bhyvectl can be used to provide adequate status
 information as to why the VM exited, I might be able to avoid this.

 It would be nicer though if /usr/sbin/bhyve returned
 the EXIT_REASON in its status code.


Agreed - see my reply to your original post.

best
Neel




   An other question is it wise to auto-reboot regardless of reason of the
 hypervisor termination?


 I think the answer is it depends.  If someone
 does reboot or shutdown -r, I think it is reasonable to restart the
 VM.  If someone does halt -p or shutdown -p, I think it is reasonable
 to not restart the VM.  For any other error condition, I think it is
 reasonable to not restart the VM.

 I think that there is no  single answer for all users and all applications.
 In my case, I am running BHyve VM's as headless which
 I access via the /dev/nmdm driver.  If the VM reboots normally, I want it
 to restart.

  --
 Craig
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Detect of BHyve VM was powered off or rebooted?

[Neel Natu]

Hi Aryeh,

On Sun, Feb 9, 2014 at 2:34 PM, Aryeh Friedman aryeh.fried...@gmail.com wrote:
 On Sun, Feb 9, 2014 at 5:18 PM, Craig Rodrigues rodr...@freebsd.org wrote:




 On Sun, Feb 9, 2014 at 1:32 PM, Aryeh Friedman 
 aryeh.fried...@gmail.comwrote:




 On Sun, Feb 9, 2014 at 4:11 PM, Craig Rodrigues rodr...@freebsd.orgwrote:

 Hi,

 I posted some rc.d scripts that I am using to boot a BHyve VM
 and send the output to a serial console using the /dev/nmdm
 driver:


 http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-January/002040.html

 It works quite well.  There is some things I would like to improve,
 and would like some advice on the best way to do it.

 (1)  If the VM was destroyed with bhyvectl --destroy --vm ${VM_NAME},
   then I do not want to automatically restart the VM in the script.
   User should manually:  service bhyvevm start

 (2)  If the VM was powered down, via shutdown -p, or halt -p,
then in my script I do not want to restart the VM in the script.
User should manually:  service bhyvevm start

 (3)   If the VM was rebooted via reboot or shutdown -r,
then I *do* want the script to restart the VM.

 I think if I change my start_vm.sh script to do something like:



 (
  while [ -e /dev/vmm/${VM} ]; do
 /usr/sbin/bhyve -c 16 -m 8G -A -H -P -g 0 -s 0:0,hostbridge -s
 1:0,lpc
 -s 2:0,virtio-net,${TAP} -s 3:0,virtio-blk,${IMG} -l com1,${CONS_A}
 ${VM}
  done

 )  


 then this might cover cases (1) and (3), but what will cover
 case (2)?

 Thanks for any advice.


 The options you gave (which are really the only ones available) do not
 distinguish between the reason for the termination of the instance of bhyve
 pointing to /dev/vmm/XXX  (it just does a normal termination).


 Did you play with any of these flags to bhyvectl?

[--get-vmcs-exit-ctls]
[--get-vmcs-exit-reason]
[--get-vmcs-exit-qualification]
[--get-vmcs-exit-interruption-info]

 If I do:
 bhyvectl --get-vmcs-exit-reason --vm vm1

 I get:
 vmcs_exit_reason[0] 0x001e

 If I look at:


 http://svnweb.freebsd.org/base/head/sys/amd64/vmm/intel/vmcs.h?view=markup#l310

 we have:
 #define EXIT_REASON_INOUT 30


 You will likely need to detect the following ones also (all for different
 reasons that may or may not have anything to do with an issue with the VM
 instead of some host [OS or hardware] issue):

 #define EXIT_REASON_EXT_INTR 1
 #define EXIT_REASON_INIT 3
 #define EXIT_REASON_HLT 12
 #define EXIT_REASON_CPUID 10
 #define EXIT_REASON_VMLAUNCH 20


These are VMCS exit reasons that have nothing to do whatsoever with
the guest wanting to reboot or power itself off.

best
Neel


 Linux has this stuff also:
 http://lxr.free-electrons.com/source/arch/x86/include/asm/vmx.h?v=3.1#L250

 so I guess these values are defined in some Intel manual.



 In my playing with this for almost a month the only solution seems to be
 put a trigger on instance shutdown via whatever method to leave a cookie
 file behind on the host (note I have not nor plan to do this in any of my
 own work).


 Yes, I can see why this might need to be done.  I might need to do that
 myself.  However, if bhyvectl can be used to provide adequate status
 information as to why the VM exited, I might be able to avoid this.

 It would be nicer though if /usr/sbin/bhyve returned
 the EXIT_REASON in its status code.




   An other question is it wise to auto-reboot regardless of reason of the
 hypervisor termination?


 I think the answer is it depends.  If someone
 does reboot or shutdown -r, I think it is reasonable to restart the
 VM.  If someone does halt -p or shutdown -p, I think it is reasonable
 to not restart the VM.  For any other error condition, I think it is
 reasonable to not restart the VM.

 I think that there is no  single answer for all users and all applications.
 In my case, I am running BHyve VM's as headless which
 I access via the /dev/nmdm driver.  If the VM reboots normally, I want it
 to restart.


 It seems almost any boot failure would be host side for example if there is
 no SSH access (which if the instance is never logged into then it is likely
 a host issue) then it is highly unlikely the VM is in bootable form (this
 is after a manual reboot of it)... an other reason (just learned the
 hardway) is the user uses a mismatch between backing file formats (my case
 using raw images with ahci-hd)
 --
 Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Networking issues

[Neel Natu]

Hi Sebastian,

On Fri, Feb 7, 2014 at 12:03 PM,  ick...@gmail.com wrote:
 Hello virtualization-lovers,

 I am very dedicated to FreeBSD since 7.2 and welcome the new bhyve
 hypervisor. :)

 So I set everything up, launched the guest and set up the network. The
 problem now is, I can reach the host IP, but not the default gateway. Did I
 forget to set something up?
 The system is a hosted root box.

 My current setup (i changed the IP's except the last octet):
 ---snip---
 host:
 # ifconfig
 re0: flags=8943UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST metric 0
 mtu 1500

 options=82099RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE
 ether 30:85:a9:ed:01:22
 inet 192.168.0.137 netmask 0xffe0 broadcast 192.168.0.159
 inet6 fe80::3285:a9ff:feed:122%re0 prefixlen 64 scopeid 0x1
 nd6 options=29PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL
 media: Ethernet autoselect (1000baseT full-duplex)
 status: active
 bridge0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu
 1500
 ether 02:0d:2a:df:6e:00
 nd6 options=1PERFORMNUD
 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
 member: tap0 flags=143LEARNING,DISCOVER,AUTOEDGE,AUTOPTP
 ifmaxaddr 0 port 4 priority 128 path cost 200
 member: re0 flags=143LEARNING,DISCOVER,AUTOEDGE,AUTOPTP
 ifmaxaddr 0 port 1 priority 128 path cost 2
 tap0: flags=8943UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST metric 0
 mtu 1500
 options=8LINKSTATE
 ether 00:bd:fe:79:0e:00
 nd6 options=29PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL
 media: Ethernet autoselect
 status: active
 Opened by PID 16910

 # netstat -rn
 Routing tables

 Internet:
 DestinationGatewayFlagsRefs  Use  Netif Expire
 default5.9.157.129UGS 0   293321re0
 192.168.0.128/27   link#1 U   0   30re0
 192.168.0.137  link#1 UHS 00lo0
 127.0.0.1  link#2 UH  0 1606lo0

 ---snip---

 ---snip---
 guest:
 # ifconfig
 vtnet0: flags=8943UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST metric 0
 mtu 1500
 options=80028VLAN_MTU,JUMBO_MTU,LINKSTATE
 ether 00:a0:98:18:c4:69
 inet 192.168.0.154 netmask 0xffe0 broadcast 192.168.0.159
 inet6 fe80::2a0:98ff:fe18:c469%vtnet0 prefixlen 64 scopeid 0x1
 nd6 options=29PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL
 media: Ethernet 10Gbase-T full-duplex
 status: active
 lo0: flags=8049UP,LOOPBACK,RUNNING,MULTICAST metric 0 mtu 16384
 options=63RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6
 inet6 ::1 prefixlen 128
 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
 inet 127.0.0.1 netmask 0xff00
 nd6 options=21PERFORMNUD,AUTO_LINKLOCAL

 # netstat -rn
 Routing tables

 Internet:
 DestinationGatewayFlagsRefs  Use  Netif Expire
 default5.9.157.129UGS 0  418 vtnet0
 192.168.0.128/27   link#1 U   0   24 vtnet0
 192.168.0.154  link#1 UHS 00lo0
 127.0.0.1  link#2 UH  0   24lo0

 ---snip---

 ping host - guest works

 # ping 192.168.0.154
 PING 192.168.0.154 (192.168.0.154): 56 data bytes
 64 bytes from 192.168.0.154: icmp_seq=0 ttl=64 time=0.083 ms
 64 bytes from 192.168.0.154: icmp_seq=1 ttl=64 time=0.094 ms


 ping guest - host works

 # ping 192.168.0.137
 PING 192.168.0.137 (192.168.0.137): 56 data bytes
 64 bytes from 192.168.0.137: icmp_seq=0 ttl=64 time=0.398 ms
 64 bytes from 192.168.0.137: icmp_seq=1 ttl=64 time=0.069 ms


 ping 173.194.70.102 (google.com) from guest - doesn't work...
 # ping 173.194.70.102
 PING 173.194.70.102 (173.194.70.102): 56 data bytes
 --- 173.194.70.102 ping statistics ---
 3 packets transmitted, 0 packets received, 100.0% packet loss

 tcpdump listening on the host:

 # tcpdump -N -vv -i bridge0
 tcpdump: WARNING: bridge0: no IPv4 address assigned
 tcpdump: listening on bridge0, link-type EN10MB (Ethernet), capture size
 65535 bytes
 19:58:19.139767 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has static
 tell 192.168.0.137, length 46

 ---^ same on tap0


 What's wrong with that setup? Did I forget to set a proper route? Or is it
 a MAC address issue?


Can you ping the default router from your guest?

Also, I was bit puzzled that the default router is 5.9.157.129 on a
192.168.0.128/27 subnet. Should I read it as 192.168.0.129 instead?

best
Neel

 Thanks in advance,

 Sebastian
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To 

Re: Enable bvmconsole and bvmdebug on i386?

[Neel Natu]

Hi John,

On Fri, Feb 7, 2014 at 8:38 AM, John Baldwin j...@freebsd.org wrote:
 I was able to test bvmconsole and bvmdebug on an i386 guest today.  The only
 change required was to copy the entries from files.amd64 to files.i386.  Are
 there any objections to committing that?


None at all.

best
Neel

 --
 John Baldwin
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: BHyVe - ESXi comparison

[Neel Natu]

Hi Andrea,

On Tue, Jan 28, 2014 at 3:18 AM, Andrea Brancatelli
abrancate...@schema31.it wrote:
 Hello everybody.

 We did a very rough comparison betweend BHyVe and VMWare ESXi. Maybe you
 want to give it a read and let me know if I did write a bunch of sh!t :-)


Looks good to me :-)

 http://andrea.brancatelli.it/2014/01/28/freebsd-10-0-bhyve-vmware-esxi-5-5-comparison/

 I must say I'm very pleased with BHyVe performances! Very good work!!

 Thanks for your time.


Could you see if you can reproduce the hang with 20VMs with the host
running FreeBSD current?

Another thing you could do is compile the host with KTR and then when
you see the hangexecute the following command on the host: sudo
ktrdump -crto /tmp/ktrdump.out

kernel options to enable KTR:
options KTR
options KTR_ENTRIES=(4*1024*1024)
options KTR_MASK=(KTR_GEN)

best
Neel

 --




 *Andrea BrancatelliSchema 31 S.r.l. - Socio UnicoResponsabile ITROMA -
 FIRENZE - PALERMO ITALYTel: +39. 06.98.358.472*

 *Cell: +39 331.2488468Fax: +39. 055.71.880.466Società del Gruppo SC31
 ITALIA*
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Linux on BHyVe in 10.0-RELEASE

[Neel Natu]

Hi Jonas,

On Thu, Jan 23, 2014 at 9:34 AM, Jonas Bülow jonas.bu...@gmail.com wrote:
 There is no bhyve process when this happens.


I am at a loss to explain this.

Do you happen to have a console log or shell history that may help in
figuring this out?

Even better would be if you can reproduce this and I can debug this
locally based on your steps.

best
Neel

 /j

 23 jan 2014 kl. 18:07 skrev Neel Natu neeln...@gmail.com:

 Hi Jonas,

 On Thu, Jan 23, 2014 at 8:00 AM, Jonas Bülow jonas.bu...@gmail.com wrote:
 I will describe the steps when I manage to reproduce the installation.
 Right now I get stuck with undestroyable vm instances:

 root@bulow:/mnt/bhyve-script # ls /dev/vmm/

 lin0

 root@bulow:/mnt/bhyve-script # bhyvectl --destroy --vm=lin0

 errno = 22

 Hmm… can you tell what the bhyve process associated with lin0 is doing
 from 'top' or 'ps'?

 What happens if you send a SIGTERM to the bhyve process?

 best
 Neel


 On Thu, Jan 23, 2014 at 3:44 PM, Matthias Gamsjager 
 mgamsja...@gmail.comwrote:

 Could you elaborate what you did to get it working?


 On Thu, Jan 23, 2014 at 3:41 PM, Jonas Bülow jonas.bu...@gmail.comwrote:

 Success. I have a working ubuntu 13.10 running under bhyve on FreeBSD
 10.0-RELEASE.



 /J


 On Thu, Jan 23, 2014 at 11:37 AM, Aryeh Friedman
 aryeh.fried...@gmail.comwrote:




 On Thu, Jan 23, 2014 at 5:05 AM, Jonas Bülow jonas.bu...@gmail.com
 wrote:

 I do use the precanned bhyve-script vm0 (if that is what you mean).
 It
 explicitly mentions linux support. bhyveload can't be user for other
 images
 than FreeBSD so I don't know how to use your script for a linux OS

 Since I have not used bhyve-grub personally yet I can not recommend what
 the correct loader command line but if bhyve-script is in anyway
 related to
 vmrun.sh I would be highly suspicious of any command lines you manage to
 extract from it since they are often not the ones actual called.



 I assume the image is file backed as I have not assigned a dedicated
 partition to the vm (and I don't know how to do that).

 You can keep them anywhere no need for a special volume (mater of fact I
 was just reusing the name of one of the diretoriess that PC creates
 during
 install even without it being a different vol [just assumes any FS
 boundaries are the host OS's problem)... you can put the disk image any
 where (your home directory would be fine for example)
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Linux on BHyVe in 10.0-RELEASE

[Neel Natu]

Hi Jonas,

On Thu, Jan 23, 2014 at 1:02 PM, Jonas Bülow jonas.bu...@gmail.com wrote:
 I had one working ubuntu13.10 vm instance but I'm unable to reproduce it.
 When I try to start the vm I get:

 /usr/sbin/bhyve -c 1 -m 2048 -A -I -H -s 0,hostbridge -s
 2,virtio-blk,./vm/vm1/vm1.img -s 3,virtio-net,tap1 -s
 4,ahci-cd,./vm/vm1/vm1.iso -S 31,uart,stdio vm1

 vm exit[0]

   reason VMX

rip 0x


 inst_length 0


   error 0


   exit_reason 33



 qualification 0x


Looks like you are running into every possible problem there is :-)

Anyways, could you confirm that you are loading the guest kernel into
'vm1' using grub-bhyve before executing the command above?

best
Neel





 /J


 On Thu, Jan 23, 2014 at 5:00 PM, Jonas Bülow jonas.bu...@gmail.com wrote:

 I will describe the steps when I manage to reproduce the installation.
 Right now I get stuck with undestroyable vm instances:

 root@bulow:/mnt/bhyve-script # ls /dev/vmm/

 lin0

 root@bulow:/mnt/bhyve-script # bhyvectl --destroy --vm=lin0

 errno = 22


 On Thu, Jan 23, 2014 at 3:44 PM, Matthias Gamsjager 
 mgamsja...@gmail.comwrote:

 Could you elaborate what you did to get it working?


 On Thu, Jan 23, 2014 at 3:41 PM, Jonas Bülow jonas.bu...@gmail.comwrote:

 Success. I have a working ubuntu 13.10 running under bhyve on FreeBSD
 10.0-RELEASE.



 /J


 On Thu, Jan 23, 2014 at 11:37 AM, Aryeh Friedman
 aryeh.fried...@gmail.comwrote:

 
 
 
  On Thu, Jan 23, 2014 at 5:05 AM, Jonas Bülow jonas.bu...@gmail.com
 wrote:
 
  I do use the precanned bhyve-script vm0 (if that is what you mean).
 It
  explicitly mentions linux support. bhyveload can't be user for other
 images
  than FreeBSD so I don't know how to use your script for a linux OS
 
 
  Since I have not used bhyve-grub personally yet I can not recommend
 what
  the correct loader command line but if bhyve-script is in anyway
 related to
  vmrun.sh I would be highly suspicious of any command lines you manage
 to
  extract from it since they are often not the ones actual called.
 
 
 
  I assume the image is file backed as I have not assigned a dedicated
  partition to the vm (and I don't know how to do that).
 
 
  You can keep them anywhere no need for a special volume (mater of fact
 I
  was just reusing the name of one of the diretoriess that PC creates
 during
  install even without it being a different vol [just assumes any FS
  boundaries are the host OS's problem)... you can put the disk image any
  where (your home directory would be fine for example)
 
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org




 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve and legacy

[Neel Natu]

Hi John,

On Wed, Jan 22, 2014 at 2:15 PM, John Baldwin j...@freebsd.org wrote:
 Is there any interest in supporting more legacy setups via bhyve?  In
 particular, I'd like to take a whack at improving the PCI INTx support, but
 that can involve several things such as possibly implementing 8259A support
 and a PCI interrupt router vs always assuming that we have APICs.  If we do

I would love to see PCI INTx support so we can do legacy interrupts
for the virtio device models.

However, does that require going all the way back to 8259 style
interrupts? It should work fine with IOAPIC, no?

 want to support a more legacy route, is there interest in supporting a BIOS
 interface in the VM?  I know that one option is to go grab a BIOS ROM from
 something like qemu, but another option is to have the real-mode IDT vector to
 stub routines in a very small ROM that traps to the hypervisor to implement
 BIOS requests.  OTOH, that may turn out to be rather messy.

 Finally, I noticed a comment fly by about removing the need for bhyveload.
 One thing I have found useful recently is passing -H to bhyveload.
 Specifically, I can build a test kernel outside of the VM on the host and
 access it via the host0 filesystem in bhyveload so I can easily test kernels
 in the VM while still using the host as my development environment.  It would
 be nice to retain this ability in some fashion.


Yes, absolutely.

best
Neel

 --
 John Baldwin
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: can't to passthru bge(4) card

[Neel Natu]

Hi Sergey,

 On Jan 21, 2014, at 1:27 AM, Sergey Matveychuk s...@semmy.ru wrote:
 
 21.01.2014 8:56, Neel Natu пишет:
 I suspect that this is because the function number of the virtual PCI
 device and the physical PCI device are different.
 
 Could you try to use the following instead: -s 2:1,passthru,3/0/1
 
 Thank you. It works.
 
 I've tried 3:1,passthru,3/0/1 but it does not work. What rules for a slot 
 number to choose?
 

It depends on the PCI function that you are passing to the guest.

For some, e.g. Intel e1000, it does not matter. You can have it appear on a 
completely different function number in the guest as compared to the host.

For others, e.g. bge, as you discovered, the function numbers in the guest and 
host have to match.

Also, PCI requires that for a multi-function device, function 0 must be 
present. So, it is not possible to assign a device to a function  0 without 
also having some device present at function 0.

Hope that helps.

Best
Neel

 
 Alternatively you could try assigning the physical PCI device 3/0/0:
 -s 3:0,passthru,3/0/0
 
 3/0/0 is a first network card. I'd like to leave it for the host system.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: kldload vmm partial lockup

[Neel Natu]

Hi Joe,

On Tue, Jan 21, 2014 at 6:53 PM, Joe Maloney jpm...@gmail.com wrote:
 Neel,
 sorry it took me a day or two to get to test this.  The patch compiles
 cleanly however it fails now trying to build the vmm module.


No worries.

I committed the patch as r260972:
http://svnweb.freebsd.org/base?view=revisionrevision=260972

Could you get the latest HEAD and see how it works for you?

best
Neel

p.s. not sure why the patch is not compiling - perhaps ppt.h did not
get patched properly?

 Joe Maloney

 /usr/src/sys/modules/vmm/../../amd64/vmm/vmm.c:232:7: error: implicit
 declaration of function 'ppt_avail_devices' is invalid in C99
   [-Werror,-Wimplicit-function-declaration]
 if (ppt_avail_devices()  0)
 ^
 /usr/src/sys/modules/vmm/../../amd64/vmm/vmm.c:565:6: error: implicit
 declaration of function 'ppt_assigned_devices' is invalid in C99
   [-Werror,-Wimplicit-function-declaration]
 if (ppt_assigned_devices(vm) == 0) {
 ^
 /usr/src/sys/modules/vmm/../../amd64/vmm/vmm.c:565:6: note: did you mean
 'ppt_assign_device'?
 if (ppt_assigned_devices(vm) == 0) {
 ^~~~
 ppt_assign_device
 /usr/src/sys/modules/vmm/../../amd64/vmm/io/ppt.h:46:5: note:
 'ppt_assign_device' declared here
 int ppt_assign_device(struct vm *vm, int bus, int slot, int func);
 ^
 2 errors generated.
 *** Error code 1

 Stop.
 make[4]: stopped in /usr/src/sys/modules/vmm
 *** Error code 1

 Stop.
 make[3]: stopped in /usr/src/sys/modules
 *** Error code 1

 Stop.
 make[2]: stopped in /usr/obj/usr/src/sys/GENERIC
 *** Error code 1

 Stop.
 make[1]: stopped in /usr/src
 *** Error code 1

 Stop.
 make: stopped in /usr/src
 root@jmhome-pc:/usr/src #

 On Jan 19, 2014, at 8:43 PM, Neel Natu neeln...@gmail.com wrote:

 Hi Joe,

 On Sun, Jan 19, 2014 at 2:21 PM, Joe Maloney jpm...@gmail.com wrote:

 I’ve been trying to get bhyve to work on a system with the following specs:

 FreeBSD 10.0 64 bit
 ASUS P6T Deluxe
 Intel i7 920

 According to what I’ve researched my cpu should have the VT-D extensions.
 However many users of this motherboard have complained about ASUS not
 properly supporting VT-D with Xen and something about a problem with tables
 getting corrupt due to a problem with the bios.  On the third link below
 which is the Xen wiki it even states that this motherboard needs a bios
 update which is not available to the public.

 https://communities.intel.com/thread/28389

 http://vip.asus.com/forum/view.aspx?id=20090402224408018SLanguage=en-usboard_id=1

 http://wiki.xen.org/wiki/VTd_HowTo

 I am curious though if this would be what would be causing my problem with
 bhyve?  I suppose I am willing to just by a new motherboard if so.

 I can run kldload vmm and I see a bunch of text fly by and then something
 about uhci interrupt problem.  As soon as that happens the messages repeats
 my network drops and no keyboard input.  However the message repeats and it
 doesn’t appear that the system has fully locked up otherwise.  However on
 another system that works I notice that kldload VMM does not show this
 information it just loads the module.

 Is there a way I can gather more logs somehow so that I can determine what
 text is appearing before the uhci errors?  Perhaps some kind of dump
 procedure?  Or is there a way I can confirm that freebsd can properly see
 and utilize the VT-D extensions.  VirtualBox runs fine with the VT
 extensions enabled.  I also made sure VirtualBox wasn’t installed or loaded
 before I tried bhyve.  I’ve tried to research for a week or two before
 posting here.


 I have a patch to not initialize the iommu unless there are passthru
 devices explicitly configured for bhyve to use.

 It is available here:
 http://people.freebsd.org/~neel/patches/bhyve_iommu_init.patch

 Could you give it a spin and see if it helps with the ASUS system?

 best
 Neel

 Joe Maloney

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org


___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: can't to passthru bge(4) card

[Neel Natu]

Hi Sergey,

On Mon, Jan 20, 2014 at 11:43 AM, Sergey Matveychuk s...@semmy.ru wrote:
 Hi.

 I try to passthru bge:
 % pciconf -vl
 ...
 ppt0@pci0:3:0:1:class=0x02 card=0x169d103c chip=0x165714e4
 rev=0x01 hdr=0x00
 vendor = 'Broadcom Corporation'
 device = 'NetXtreme BCM5719 Gigabit Ethernet PCIe'
 class  = network
 subclass   = ethernet
 ...
 % cat /boot/loader.conf
 vmm_load=yes
 pptdevs=3/0/1

 % bhyve -c ${VM_CPUNUM} -m ${VM_MEMSIZE} -AI -HP -g0 \
 -s 0:0,hostbridge \
 -s 3:0,passthru,3/0/1 \
 -s 2:0,virtio-blk,${VM_DISK} \
 -S 31,uart,${VM_CONSOLE} \
 ${VM_NAME}


I suspect that this is because the function number of the virtual PCI
device and the physical PCI device are different.

Could you try to use the following instead: -s 2:1,passthru,3/0/1

Alternatively you could try assigning the physical PCI device 3/0/0:
-s 3:0,passthru,3/0/0

best
Neel

 And I got this on boot:

 bge0: Broadcom unknown BCM5719, ASIC rev. 0x5719001 mem
 0xc001-0xc001,0xc002-0xc002,0xc003-0xc003 irq 36 at
 device 3.0 on pci0
 bge0: APE FW version: NCSI v1.0.60.0
 bge0: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E
 bge0: Try again
 bge0: Try again
 bge0: Try again
 bge0: Try again
 bge0: attaching PHYs failed
 device_attach: bge0 attach returned 6

 I used this as an instruction: https://wiki.freebsd.org/bhyve/pci_passthru
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: kldload vmm partial lockup

[Neel Natu]

Hi Joe,

On Sun, Jan 19, 2014 at 2:21 PM, Joe Maloney jpm...@gmail.com wrote:
 I’ve been trying to get bhyve to work on a system with the following specs:

 FreeBSD 10.0 64 bit
 ASUS P6T Deluxe
 Intel i7 920

 According to what I’ve researched my cpu should have the VT-D extensions.  
 However many users of this motherboard have complained about ASUS not 
 properly supporting VT-D with Xen and something about a problem with tables 
 getting corrupt due to a problem with the bios.  On the third link below 
 which is the Xen wiki it even states that this motherboard needs a bios 
 update which is not available to the public.

 https://communities.intel.com/thread/28389

 http://vip.asus.com/forum/view.aspx?id=20090402224408018SLanguage=en-usboard_id=1

 http://wiki.xen.org/wiki/VTd_HowTo

 I am curious though if this would be what would be causing my problem with 
 bhyve?  I suppose I am willing to just by a new motherboard if so.

 I can run kldload vmm and I see a bunch of text fly by and then something 
 about uhci interrupt problem.  As soon as that happens the messages repeats 
 my network drops and no keyboard input.  However the message repeats and it 
 doesn’t appear that the system has fully locked up otherwise.  However on 
 another system that works I notice that kldload VMM does not show this 
 information it just loads the module.

 Is there a way I can gather more logs somehow so that I can determine what 
 text is appearing before the uhci errors?  Perhaps some kind of dump 
 procedure?  Or is there a way I can confirm that freebsd can properly see and 
 utilize the VT-D extensions.  VirtualBox runs fine with the VT extensions 
 enabled.  I also made sure VirtualBox wasn’t installed or loaded before I 
 tried bhyve.  I’ve tried to research for a week or two before posting here.


I have a patch to not initialize the iommu unless there are passthru
devices explicitly configured for bhyve to use.

It is available here:
http://people.freebsd.org/~neel/patches/bhyve_iommu_init.patch

Could you give it a spin and see if it helps with the ASUS system?

best
Neel

 Joe Maloney

 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: [PATCH] Rework DSDT generation in bhyve

[Neel Natu]

Hi John,

On Fri, Dec 27, 2013 at 9:05 AM, John Baldwin j...@freebsd.org wrote:
 While hacking on the power button support, I also started rototilling the DSDT
 generation code a bit.  My initial goal was to enumerate the LPC serial ports
 (COM1 and COM2) properly via ACPI.  I ended up doing the following:

 - Moved the info for the top-level PCI bus into the PCI emulation code and
   added ResourceProducer entries for the memory ranges decoded by the bus
   for memory BARs.
 - I added a framework to allow each PCI emulation driver to optionally write
   an entry into the DSDT under the \_SB_.PCI0 namespace.  The LPC driver uses
   this to write a node for the LPC bus (\_SB_.PCI0.ISA).
 - I added a linker set to allow any LPC devices to write entries into the
   DSDT below the LPC node.  I moved the existing block for the RTC out of
   acpi.c and into the RTC driver.  I added DSDT nodes for the AT PIC,
   the 8254 ISA timer, and the LPC UART devices.
 - I also added a SuperIO device under the LPC node to claim system
   resources (as is done in real hardware).  I added a linker set to allow
   various drivers to add IO or memory ranges that should be claimed by
   SuperIO and then added the extended RTC IO range and the registers used
   for ACPI power management as system resources.

 The end result is that for the stock VM created by vmrun.sh, the attimer0,
 uart0, and uart1 devices move from isa0 to acpi0.  There is also a PIC
 device that would be claimed by 'device atpic' (but the stock amd64 kernel
 doesn't include that).  The DSDT is also a bit more fleshed out, and also
 looks nice as devices are mostly laid out in the normal tree rather than
 using separate Scope() sections for each device.

 Note that I did add some helper routines for writing out DSDT lines and
 resource entries to try to simplify the code in the various DSDT handlers
 and make the code that generates DSDT lines a bit more readable (e.g. the
 implicit newlines make things more readable IMO).

 The patch is relative to the previous power button patch and is at
 http://www.FreeBSD.org/~jhb/patches/bhyve_dsdt.patch


This looks great!

This is not related to your patch (it has always been there), but I
was trying to understand the significance of this entry:
pci_lpc_write_dsdt(): OperationRegion (P40C, PCI_Config, 0x60, 0x04)

Any idea on why this is needed/interesting?

best
Neel

 Assuming these are ok, the next thing I might work on is cleaning up
 the PCI INTx interrupts.  They should really all be rounted to the I/O APIC
 intpins above 15 (and we should leave ISA IRQs 5, 10, and 11 alone in APIC
 mode and only fall back to that if we are using the 8259As and a real ELCR).
 The MP Table output would need some minor tweaking for that, and for ACPI we
 should generate a _PRT table under _SB_.PCI0 in pci_emul.c.

 --
 John Baldwin
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: [PATCH] Support for S5 (soft power off) in bhyve

[Neel Natu]

Hi John,


On Fri, Dec 27, 2013 at 8:46 AM, John Baldwin j...@freebsd.org wrote:
 On Thursday, December 26, 2013 11:37:47 am John Baldwin wrote:
 On Wednesday, December 25, 2013 10:43:27 am Michael Dexter wrote:
 
  John,
 
  This is awesome.
 
  Can this be triggered from the host to shut down an unresponsive VM?
 
  If so, syntax?

 It cannot.  However, we could add a virtual power button that acts like an
 external ACPI power button.  We could then add a flag to bhyvectl to
 trigger it.  I will look into that.

 I've implemented this.  Rather than doing it via bhyvectl, I turned SIGTERM
 sent to bhyve itself into the virtual power button.  Thus, if you
 'killall bhyve' or 'pkill bhyve' from the host, the guest will gracefully
 shutdown using S5.  You can still use kill -9 to kill a hung guest (similar
 to the 4 second power button override on real metal).

 To do this I fleshed out the ACPI power management support a bit more adding
 more of the implementation of PM1_EVT and PM1_CNT.  I also added an SMI_CMD
 register with commands to enable and disable ACPI (for now all that does is
 enable/disable the virtual power button as well as toggle SCI_EN in PM1_CNT).
 I also added support for raising ACPI's SCI based on settings in PM1_EVT.

 I extended mevent to handle EVFILT_SIGNAL events and use a mevent handler for
 SIGTERM to simulate pressing the power button.  This just implements the
 simple fixed-feature power button support via PM1_EVT rather than having a
 control method button (which would require implementing GPE support).

 While here, I made sure the SCI interrupt was active-lo / level-triggered,
 and I changed PCI interrupts in the MP Table to explicitly use active-lo
 since they were already explicitly using level-trigger.

 I also fixed the pmtmr code to use pthread_once instead of a racy static
 int.  That should probably be a separate commit though.


This looks great!

Thanks for adding this functionality.

best
Neel

 --- //depot/vendor/freebsd/src/usr.sbin/bhyve/acpi.c
 +++ //depot/user/jhb/bhyve/usr.sbin/bhyve/acpi.c
 @@ -85,10 +87,6 @@
  #define BHYVE_ASL_SUFFIX   .aml
  #define BHYVE_ASL_COMPILER /usr/sbin/iasl

 -#define BHYVE_PM1A_EVT_ADDR0x400
 -#define BHYVE_PM1A_CNT_ADDR0x404
 -#define BHYVE_PM_TIMER_ADDR0x408
 -
  static int basl_keep_temps;
  static int basl_verbose_iasl;
  static int basl_ncpu;
 @@ -285,11 +290,11 @@
 EFPRINTF(fp, [0001]\t\tSubtable Type : 02\n);
 EFPRINTF(fp, [0001]\t\tLength : 0A\n);
 EFPRINTF(fp, [0001]\t\tBus : 00\n);
 -   EFPRINTF(fp, [0001]\t\tSource : 09\n);
 -   EFPRINTF(fp, [0004]\t\tInterrupt : 0009\n);
 +   EFPRINTF(fp, [0001]\t\tSource : %02X\n, SCI_INT);
 +   EFPRINTF(fp, [0004]\t\tInterrupt : %08X\n, SCI_INT);
 EFPRINTF(fp, [0002]\t\tFlags (decoded below) : \n);
 -   EFPRINTF(fp, \t\t\tPolarity : 0\n);
 -   EFPRINTF(fp, \t\t\tTrigger Mode : 0\n);
 +   EFPRINTF(fp, \t\t\tPolarity : 3\n);
 +   EFPRINTF(fp, \t\t\tTrigger Mode : 3\n);
 EFPRINTF(fp, \n);

 /* Local APIC NMI is connected to LINT 1 on all CPUs */
 @@ -336,23 +341,27 @@
 basl_acpi_base + FACS_OFFSET);
 EFPRINTF(fp, [0004]\t\tDSDT Address : %08X\n,
 basl_acpi_base + DSDT_OFFSET);
 -   EFPRINTF(fp, [0001]\t\tModel : 00\n);
 +   EFPRINTF(fp, [0001]\t\tModel : 01\n);
 EFPRINTF(fp, [0001]\t\tPM Profile : 00 [Unspecified]\n);
 -   EFPRINTF(fp, [0002]\t\tSCI Interrupt : 0009\n);
 -   EFPRINTF(fp, [0004]\t\tSMI Command Port : \n);
 -   EFPRINTF(fp, [0001]\t\tACPI Enable Value : 00\n);
 -   EFPRINTF(fp, [0001]\t\tACPI Disable Value : 00\n);
 +   EFPRINTF(fp, [0002]\t\tSCI Interrupt : %04X\n,
 +   SCI_INT);
 +   EFPRINTF(fp, [0004]\t\tSMI Command Port : %08X\n,
 +   SMI_CMD);
 +   EFPRINTF(fp, [0001]\t\tACPI Enable Value : %02X\n,
 +   BHYVE_ACPI_ENABLE);
 +   EFPRINTF(fp, [0001]\t\tACPI Disable Value : %02X\n,
 +   BHYVE_ACPI_DISABLE);
 EFPRINTF(fp, [0001]\t\tS4BIOS Command : 00\n);
 EFPRINTF(fp, [0001]\t\tP-State Control : 00\n);
 EFPRINTF(fp, [0004]\t\tPM1A Event Block Address : %08X\n,
 -BHYVE_PM1A_EVT_ADDR);
 +   PM1A_EVT_ADDR);
 EFPRINTF(fp, [0004]\t\tPM1B Event Block Address : \n);
 EFPRINTF(fp, [0004]\t\tPM1A Control Block Address : %08X\n,
 -BHYVE_PM1A_CNT_ADDR);
 +   PM1A_CNT_ADDR);
 EFPRINTF(fp, [0004]\t\tPM1B Control Block Address : \n);
 EFPRINTF(fp, [0004]\t\tPM2 Control Block Address : \n);
 EFPRINTF(fp, [0004]\t\tPM Timer Block Address : %08X\n,
 -BHYVE_PM_TIMER_ADDR);
 +   IO_PMTMR);
 EFPRINTF(fp, [0004]\t\tGPE0 Block Address : \n);
 EFPRINTF(fp, [0004]\t\tGPE1 Block Address : \n);
 EFPRINTF(fp, [0001]\t\tPM1 Event Block Length : 04\n);
 @@ -385,7 +394,7 @@
  

Re: 11.0-CURRENT panic while running a bhyve instance

[Neel Natu]

Hi Markiyan,

Fixed in r259737:
http://svnweb.freebsd.org/changeset/base/259737

best
Neel

On Fri, Dec 13, 2013 at 2:35 PM, Markiyan Kushnir
markiyan.kush...@gmail.com wrote:
 2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


 Shared here:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing

 --
 Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-curr...@freebsd.org, freebsd-virtualization@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-curr...@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
 ___
 freebsd-curr...@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: 11.0-CURRENT panic while running a bhyve instance

[Neel Natu]

Hi Markiyan,

On Sat, Dec 14, 2013 at 6:00 AM, Markiyan Kushnir
markiyan.kush...@gmail.com wrote:
 2013/12/14 Markiyan Kushnir markiyan.kush...@gmail.com:
 2013/12/14 Markiyan Kushnir markiyan.kush...@gmail.com:
 2013/12/14 Neel Natu neeln...@gmail.com:
 Hi Markiyan,

 On Fri, Dec 13, 2013 at 2:35 PM, Markiyan Kushnir
 markiyan.kush...@gmail.com wrote:
 2013/12/13 John Baldwin j...@freebsd.org:
 On Friday, December 13, 2013 5:46:20 am Markiyan Kushnir wrote:
 Forgot to fill the Subject: header, re-posting it fixed.

 The mailing lists strips attachments, can you post it at a URL?


 Shared here:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnem5iYTVqLUxrcWo4cmlhdkM1c2lJa2dKak5R/edit?usp=sharing


 Thanks.

 It looks like something funky going on with the vcpu state. Do you
 know if there was any access to the VM via 'bhyvectl' close to the
 time of the panic?


 Well, I don't know if there was. I would set up the same scenario
 again + a script running on the host querying bhyvectl. May be I would
 catch it again. Please let me know if all it makes sense, and if so,
 how it can be made better.


 To make it clear -- I didn't run bhyvectl when the VM was running, so
 I can tell for sure that nobody was trying to interact with the VM
 during its run. My first thought was that it would make sense to
 (periodically) query state of the VM using bhyvectl to get info about
 what was going on when it comes close to the crash...

 --
 Markiyan.

 Ok, I was able to hit a similarly looking panic exactly when trying to
 run bhyvectl --vm=altroot-bhyve --get-all while the VM was busy with
 its own pkg delete -af. When the VM is mostly idle, bhyvectl
 --get-all goes smoothly.

 Now I'm thinking I might inadvertently run bhyvectl over my heavily
 running VM once when I was not at the desktop and wanted to see how
 things were going remotely from my office…


Thanks, that helps a lot because I can see how bhyvectl could perturb
the state of the vcpu and eventually lead to a panic.

I'll submit a fix for this ASAP.

best
Neel

 Here are core.txt of the two panics in a row:

 https://drive.google.com/file/d/0B9Q-zpUXxqCnMUxVdGwxSEs1dE0/edit?usp=sharing

 https://drive.google.com/file/d/0B9Q-zpUXxqCnREtuTXpabnQ5QXM/edit?usp=sharing

 --
 Markiyan.


 --
 Markiyan.

 best
 Neel

 --
 Markiyan.

 --
 Markiyan


 -- Forwarded message --
 From: Markiyan Kushnir markiyan.kush...@gmail.com
 Date: 2013/12/13
 Subject:
 To: freebsd-curr...@freebsd.org, freebsd-virtualization@freebsd.org


 I started some ports to compile inside a bhyve instance:

 root@vm:~ # uname -a
 FreeBSD vm.mkushnir.mooo.com 11.0-CURRENT FreeBSD 11.0-CURRENT #0
 r259250: Thu Dec 12 14:17:32 EET 2013
 r...@vm.mkushnir.zapto.org:/
 usr/obj/usr/src.svnup/sys/MAREK  amd64

 and left it running unattended. Approx. 2 hours later the host went to
 panic. The bhyve instance survived after the panic and I could be able
 to complete my ports compilation.

 core.txt attached (gzipped)
 ___
 freebsd-curr...@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to 
 freebsd-current-unsubscr...@freebsd.org


 --
 John Baldwin
 ___
 freebsd-curr...@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Panic starting a bhyve guest after resume

[Neel Natu]

Hi John,

On Fri, Dec 13, 2013 at 2:09 PM, John Baldwin j...@freebsd.org wrote:
 On Thursday, December 12, 2013 4:00:08 pm Neel Natu wrote:
 Hi John,

 On Thu, Dec 12, 2013 at 12:11 PM, John Baldwin j...@freebsd.org wrote:
  If I suspend and resume my laptop and then try to start a guest after the
  resume, I get an odd panic.  It generates a privileged instruction fault 
  (in
  kernel mode) for 'vmclear'.  I've checked CR4 and it claims that VMXE is 
  set.
  I dont have any other ideas off the top of my head on what I should be 
  poking
  at?  It looks like we read a bunch of MSRs in vmx_init(), but we don't 
  write
  to them, and all vmx_enable() does on each CPU is set VMXE in CR4 from 
  what I
  can tell.
 

 It also does a vmxon on each logical cpu which may also need to be
 done after a resume.

 Ah, yes it does.  That was sufficient both for starting a new guest after
 resume and even doing a suspend/resume while a guest was active (and the
 guest continued to run fine).  I have a hacky patch for this.  One, it
 includes both a suspend and resume hook for VMM, though for my testing I only
 needed a resume hook to invoke vmxon.  Second, the name of vmx_resume2()
 is a total hack (because vmx_resume() was already taken.  I think for now
 if I were to commit this, I'd just add the resme hook and maybe call the
 Intel method vmx_reset() or vmx_restore()?

 http://people.freebsd.org/~jhb/patches/bhyve_resume.patch


There seems to be a race after the APs are restarted and before
'vmm_resume_p()' where it would be problematic to execute a VMX
instruction.

Perhaps we should enable VMX on each cpu before they return to the
interrupted code?

best
Neel

 --
 John Baldwin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: [PATCH] Additional lapic support for bhyve

[Neel Natu]

Hi John,

On Wed, Dec 11, 2013 at 9:38 AM, John Baldwin j...@freebsd.org wrote:
 On Wednesday, December 11, 2013 2:19:14 am Neel Natu wrote:
 Hi John,

 On Tue, Dec 10, 2013 at 12:27 PM, John Baldwin j...@freebsd.org wrote:
  This patch extends the local APIC emulation a bit by adding the following
  things:
  - Add an ioctl and a bhyvectl command to trigger local interrupts on a
local APIC.  The fixed and NMI delivery modes are enabled.
  - Add support for the CMCI LVT entry.
  - Add the ability to report local APIC errors and trigger errors for
invalid vectors when sending IPIs or firing an external interrupt that
references an invalid IDT vector.
  - Silently force all reserved fields in LVT entries to zero when they
are written (removes the need for clearing those bits when triggering
an LVT interrupt).
  - Add entries to the MP Table and MADT to advertise the typical x86 LINT
configuration (ExtINT on LINT0 and NMI on LINT1).
  - Add a bhyvectl command to inject an NMI on an arbitrary CPU (this latter
should probably be a separate patch)
 
  In particular, while bhyectl --inject-nmi can inject an NMI on a single 
  vcpu,
  the more traditional way of signalling a system-wide error (such as SERR#
  or PERR#) is to assert the LINT1 pins on all CPUs.  This can now be done 
  via
  'bhyectl --vm=foo --cpu=-1 --assert-lapic-lvt=1' (cpu of -1 is a broadcast
  to all CPUs for the LVT ioctl).  The patch is at
  people.freebsd.org/~jhb/patches/bhyve_lapic.patch  I'm certainly open to
  suggestions on ways to make this be more consistent with the 
  style/design/flow
  of the existing code.
 

 This looks good.

 vlapic_fire_lvt() and vlapic_set_error() can call each other
 recursively if guest has programmed the ERROR_LVT vector incorrectly.
 This can probably be fixed by passing the LVT index to
 vlapic_fire_lvt() and treating LVT_ERROR specially.

 Oh, cute, and very true.  Another option might be to have set_error set
 a per vlapic flag and avoid posting the interrupt if it detects recursion.


Yup, that would work as well.

 A couple of nits:
 - Could you add a /* fallthrough */ comment after setting the 'mask'
 for the LINT_LVT entries in vlapic_set_lvt()?

 Sure.

 - Any reason that vlapic_set_error() and vlapic_fire_cmci() are not
 static to vlapic.c? Do you anticipate calling them from outside this
 file in the future?

 Yes.  I imagine the machine check injection code I plan to work on would
 be in a different file, and it would need to be able to trigger CMCI.

 For vlapic_set_error(), there is one error case we currently do not handle
 which is access to an invalid register (right now we ignore those).  For
 x2APIC we should be generating general protection fault.  For mmio access
 an error is supposed to be asserted instead.  I figured for the latter
 that it would be done in lapic_mmio_read() and lapic_mmio_write().


Sounds good.

best
Neel

 --
 John Baldwin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Bhyve virtio-net on Linux guest

[Neel Natu]

Hi,

 On Nov 7, 2013, at 4:21 AM, Tetsuya Mukawa mtetsu...@gmail.com wrote:
 
 Hi folks,
 
 I want to run a Linux guest on Bhyve.
 It seems very recently Bhyve supports a linux guest, so I am a bit worry
 about I can use a virtio-net.
 Someone, could you please let me know is it possible to use a virtio-net on
 Linux guest?
 

Yes it is possible to use virtio-net from a Linux guest inside a bhyve virtual 
machine.

Best
Neel

 --
 Tetsuya
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: [PATCH] Tidy usage messages for bhyve and bhyveload

[Neel Natu]

Hi John,

Thanks for the patch - submitted as r257018.

best
Neel

On Wed, Oct 23, 2013 at 7:36 AM, John Baldwin j...@freebsd.org wrote:

 These are just some cosmetic nits to the usage messages for bhyve and
 bhyveload so that they are more consistent with other tools in the base
 system (spaces between options, wrapping at 80 cols, etc.)

 Index: bhyve/bhyverun.c
 ===
 --- bhyve/bhyverun.c(revision 256946)
 +++ bhyve/bhyverun.c(working copy)
 @@ -37,6 +37,7 @@

  #include stdio.h
  #include stdlib.h
 +#include string.h
  #include err.h
  #include libgen.h
  #include unistd.h
 @@ -121,9 +122,8 @@
  {

  fprintf(stderr,
 -Usage: %s [-aehAHIPW][-g gdb port][-s pci][-S pci]
 -   [-c vcpus][-p pincpu][-m mem]
 -vmname\n
 +Usage: %s [-aehAHIPW] [-g gdb port] [-s pci] [-S
 pci]\n
 +  %*s [-c vcpus] [-p pincpu] [-m mem] vmname\n
-a: local apic is in XAPIC mode (default is
 X2APIC)\n
-A: create an ACPI table\n
-g: gdb port\n
 @@ -132,13 +132,13 @@
-H: vmexit from the guest on hlt\n
-I: present an ioapic to the guest\n
-P: vmexit from the guest on pause\n
 -  -W: force virtio to use single-vector MSI\n
 -  -e: exit on unhandled i/o access\n
 +  -W: force virtio to use single-vector MSI\n
 +  -e: exit on unhandled I/O access\n
-h: help\n
-s: slot,driver,configinfo PCI slot config\n
-S: slot,driver,configinfo legacy PCI slot
 config\n
-m: memory size in MB\n,
 -   progname);
 +   progname, (int)strlen(progname), );

 exit(code);
  }
 Index: bhyveload/bhyveload.c
 ===
 --- bhyveload/bhyveload.c   (revision 256946)
 +++ bhyveload/bhyveload.c   (working copy)
 @@ -71,6 +71,7 @@
  #include err.h
  #include fcntl.h
  #include getopt.h
 +#include libgen.h
  #include limits.h
  #include stdio.h
  #include stdlib.h
 @@ -568,8 +569,9 @@
  {

 fprintf(stderr,
 -   usage: %s [-m mem-size][-d disk-path] [-h host-path] 
 -   [-e name=value] vmname\n, progname);
 +   usage: %s [-m mem-size] [-d disk-path] [-h host-path]\n
 +  %*s [-e name=value] vmname\n, progname,
 +   (int)strlen(progname), );
 exit(1);
  }

 @@ -582,7 +584,7 @@
 int opt, error;
 char *disk_image;

 -   progname = argv[0];
 +   progname = basename(argv[0]);

 mem_size = 256 * MB;
 disk_image = NULL;

 --
 John Baldwin
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Use humanize_number() inside bhyveload?

[Neel Natu]

Hi Craig,

On Sun, Oct 6, 2013 at 11:31 PM, Craig Rodrigues rodr...@freebsd.orgwrote:

 On Fri, Oct 4, 2013 at 1:40 PM, Peter Grehan gre...@freebsd.org wrote:

  Hi Craig,
 
 
   Can we use expand_number() in bhyve_load?
 
 
   Please commit this change :)
 
  later,
 
  Peter.
 
 
 Here is a bigger patch, which uses expand_number() in bhyveload and bhyve,
 and also changes comments
 in vmrun.sh to reflect the change.


Thanks for the patch - submitted as r256176 with some modifications to
preserve backward compatibility.

best
Neel


 Index: share/examples/bhyve/vmrun.sh
 ===
 --- share/examples/bhyve/vmrun.sh (revision 256073)
 +++ share/examples/bhyve/vmrun.sh (working copy)
 @@ -31,7 +31,7 @@
  BHYVECTL=/usr/sbin/bhyvectl
  FBSDRUN=/usr/sbin/bhyve

 -DEFAULT_MEMSIZE=512
 +DEFAULT_MEMSIZE=512M
  DEFAULT_CPUS=2
  DEFAULT_TAPDEV=tap0

 @@ -47,7 +47,7 @@
   echo-g: listen for connection from kgdb at gdbport
   echo-i: force boot of the Installation CDROM image
   echo-I: Installation CDROM image location (default is
 ${DEFAULT_ISOFILE})
 - echo-m: memory size in MB (default is ${DEFAULT_MEMSIZE}MB)
 + echo-m: memory size (default is ${DEFAULT_MEMSIZE})
   echo-t: tap device for virtio-net (default is $DEFAULT_TAPDEV)
   echo 
   echoThis script needs to be executed with superuser privileges
 Index: usr.sbin/bhyve/Makefile
 ===
 --- usr.sbin/bhyve/Makefile (revision 256073)
 +++ usr.sbin/bhyve/Makefile (working copy)
 @@ -17,8 +17,8 @@

  NO_MAN=

 -DPADD= ${LIBVMMAPI} ${LIBMD} ${LIBPTHREAD}
 -LDADD= -lvmmapi -lmd -lpthread
 +DPADD= ${LIBVMMAPI} ${LIBMD} ${LIBUTIL} ${LIBPTHREAD}
 +LDADD= -lvmmapi -lmd -lutil -lpthread

  WARNS?= 2

 Index: usr.sbin/bhyve/bhyverun.c
 ===
 --- usr.sbin/bhyve/bhyverun.c (revision 256073)
 +++ usr.sbin/bhyve/bhyverun.c (working copy)
 @@ -37,7 +37,9 @@

  #include stdio.h
  #include stdlib.h
 +#include err.h
  #include libgen.h
 +#include libutil.h
  #include unistd.h
  #include assert.h
  #include errno.h
 @@ -541,6 +543,9 @@
   break;
  case 'm':
   memsize = strtoul(optarg, NULL, 0) * MB;
 + if (expand_number(optarg, memsize) == -1)
 + errx(EXIT_FAILURE,
 + invalid size argument `%s', optarg);
   break;
   case 'H':
   guest_vmexit_on_hlt = 1;
 Index: usr.sbin/bhyveload/Makefile
 ===
 --- usr.sbin/bhyveload/Makefile (revision 256073)
 +++ usr.sbin/bhyveload/Makefile (working copy)
 @@ -4,8 +4,8 @@
  SRCS= bhyveload.c
  MAN= bhyveload.8

 -DPADD+= ${LIBVMMAPI}
 -LDADD+= -lvmmapi
 +DPADD+= ${LIBVMMAPI} ${LIBUTIL}
 +LDADD+= -lvmmapi -lutil

  WARNS?= 3

 Index: usr.sbin/bhyveload/bhyveload.8
 ===
 --- usr.sbin/bhyveload/bhyveload.8 (revision 256073)
 +++ usr.sbin/bhyveload/bhyveload.8 (working copy)
 @@ -1,4 +1,4 @@
 -.\
 +\
  .\ Copyright (c) 2012 NetApp Inc
  .\ All rights reserved.
  .\
 @@ -60,13 +60,29 @@
  .Sh OPTIONS
  The following options are available:
  .Bl -tag -width indent
 -.It Fl m Ar mem-size
 +.It Fl m Ar mem-size Xo
 +.Sm off
 +.Op Cm K | k | M | m | G | g | T | t
 +.Xc
 +.Sm on
  .Ar mem-size
 -is the amount of memory allocated to the guest in units of megabytes.
 +is the amount of memory allocated to the guest.
  .Pp
 +The
 +.Ar mem-size
 +argument may be suffixed with one of
 +.Cm K ,
 +.Cm M ,
 +.Cm G
 +or
 +.Cm T
 +(either upper or lower case) to indicate a multiple of
 +Kilobytes, Megabytes, Gigabytes or Terabytes
 +respectively.
 +.Pp
  The default value of
  .Ar mem-size
 -is 256.
 +is 256M.
  .It Fl d Ar disk-path
  The
  .Ar disk-path
 @@ -83,7 +99,7 @@
  .Pa /freebsd/release.iso
  and has 1GB memory allocated to it:
  .Pp
 -.Dl bhyveload -m 1024 -d /freebsd/release.iso freebsd-vm
 +.Dl bhyveload -m 1G -d /freebsd/release.iso freebsd-vm
  .Sh SEE ALSO
  .Xr bhyve 4 ,
  .Xr bhyve 8 ,
 Index: usr.sbin/bhyveload/bhyveload.c
 ===
 --- usr.sbin/bhyveload/bhyveload.c (revision 256073)
 +++ usr.sbin/bhyveload/bhyveload.c (working copy)
 @@ -67,9 +67,11 @@
  #include dirent.h
  #include dlfcn.h
  #include errno.h
 +#include err.h
  #include fcntl.h
  #include getopt.h
  #include limits.h
 +#include libutil.h
  #include stdio.h
  #include stdlib.h
  #include string.h
 @@ -581,7 +583,9 @@
   break;

   case 'm':
 - mem_size = strtoul(optarg, NULL, 0) * MB;
 + if (expand_number(optarg, mem_size) == -1)
 + errx(EXIT_FAILURE,
 +invalid size argument `%s', optarg);
   break;

   case '?':
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 

Re: bhve bug report: attempting to physical DVD drive for install CD is unbootable

[Neel Natu]

Hi Aryeh,

The -d option was meant to specify the disk or the ISO file image so I am
not surprised that it doesn't work with /dev/cd0.

Is there any reason that you cannot use an ISO image file to boot the guest?

best
Neel

On Mon, Sep 30, 2013 at 3:16 PM, Aryeh Friedman aryeh.fried...@gmail.comwrote:

 With the foillowing script:

 truncate -s 20G /a
 ifconfig tap0 destroy
 ifconfig tap0 create
 ifconfig tap0 up
 sleep 5
 ifconfig bridge0 addm tap0 up
 /usr/sbin/bhyvectl --vm=a --destroy
 /usr/sbin/bhyveload -m 1024 -d /dev/cd0 a
 /usr/sbin/bhyve -c 1 -m 1024 -AI -H -P -g 0 -s 0:0,hostbridge -s
 1:0,virtio-net,tap0 -s 2:0,virtio-blk,/a -s 3:0,virtio-blk,/dev/cd0 -S
 31,uart,stdio a

 I get the following:

 FreeBSD/amd64 User boot, Revision 1.1
 (aryeh@testcloud, Wed Sep 11 22:56:07 EDT 2013)

 can't load 'kernel'
 OK ls
 open '/' failed: device not configured
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve vmrum vm_setup_memory: Inappropriate ioctl for device

[Neel Natu]

Hi Miguel,

On Thu, Sep 26, 2013 at 6:59 PM, Miguel Clara miguelmcl...@gmail.comwrote:

 Just wanted to start testing with bhyve, but I'm getting this error
 creating an exmaple VM:



 # ./vmrun.sh -c 1 vm1
 virtio disk device file ./diskdev does not exist.
 Creating it ...
 Launching virtual machine vm1 ...
 vm_setup_memory: Inappropriate ioctl for device


Your userspace is most likely out of sync with the kernel. Could you try to
re-run after a buildworld and installworld?

best
Neel


 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: 10-CURRENT code freezes and bhyve

[Neel Natu]

Hi Aryeh,

On Mon, Aug 26, 2013 at 12:14 AM, Aryeh Friedman
aryeh.fried...@gmail.comwrote:

 What is the schedule, functionality goal and such for bhyve in regards to
 the 10-CURRENT code freeze coming up (according to the rel_eng page) on
 9/17


- AMD/SVM support (ready to go)
- integrate bhyve nested page tables with amd64/pmap (code is being
reviewed)
- ahci device emulation (WIP, GSoC project)
- e1000 device emulation (WIP)

best
Neel

___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: headerless guest under bhyve

[Neel Natu]

Hi Aryeh,

On Mon, Jun 24, 2013 at 3:54 PM, Aryeh Friedman aryeh.fried...@gmail.comwrote:

 How do I make it so I do not need to allocate a xterm to being the
 console for a bhyve guest? (so far I have not been able to make it so
 I can break the connection to the console and the guest doesn't crash)


Yeah, this is because bhyve is exiting when it gets a SIGHUP.

Here is a patch that may help you sidestep this problem for now:

Index: bhyverun.c
===
--- bhyverun.c (revision 252047)
+++ bhyverun.c (working copy)
@@ -679,6 +679,17 @@

  vmname = argv[0];

+ {
+ struct sigaction sa;
+
+ sigemptyset(sa.sa_mask);
+ sa.sa_flags = 0;
+ sa.sa_handler = SIG_IGN;
+
+ err = sigaction(SIGHUP, sa, NULL);
+ assert(err == 0);
+ }
+

The right way to fix this is to have a configurable backend (terminal,
network port, pipe) for the virtual serial port.


 Related question when using vmrun.sh it seems like the vtnet on the
 guest becomes unusable after reboot of the guest (rebooting the host
 is the only solution I can find) is there a work around for this? (I
 tried toggling the state of if_tap and if_bridge)


If you could provide some more details I will try to reproduce it. For e.g.
IP address assigned to the guest, host networking setup etc.

best
Neel


 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve guest dies on building java/openjdk6

[Neel Natu]

Hi Aryeh,

On Mon, Jun 24, 2013 at 4:25 PM, Aryeh Friedman
aryeh.fried...@gmail.com wrote:

 As soon as java/openjdk6 starts its build process (fdesc and procfs
 both are present as per pkg-message) bhyve dies with:

 vm exit[0]
 reason  VMX
 rip 0x0008032322ac

 inst_length 2

 error   0

 exit_reason 10


Thanks for reporting this. The exit reason indicates bhyve is not
happy with the cpuid leaf that the guest is querying.

Tycho Nightingale has a patch to the cpuid emulation that should fix
the issue you are seeing.

Could you give it a spin?

Index: x86.c
===
--- x86.c   (revision 251672)
+++ x86.c   (working copy)
@@ -45,7 +45,7 @@

 #defineCPUID_VM_HIGH   0x4000

-static const char bhyve_id[12] = BHyVE BHyVE ;
+static const char bhyve_id[12] = bhyve bhyve ;

 int
 x86_emulate_cpuid(struct vm *vm, int vcpu_id,
@@ -80,12 +80,10 @@
case CPUID__:
case CPUID__0002:
case CPUID__0003:
-   case CPUID__000A:
cpuid_count(*eax, *ecx, regs);
break;

case CPUID_8000_:
-   case CPUID_8000_0001:
case CPUID_8000_0002:
case CPUID_8000_0003:
case CPUID_8000_0004:
@@ -94,6 +92,15 @@
cpuid_count(*eax, *ecx, regs);
break;

+   case CPUID_8000_0001:
+   /*
+* Hide rdtscp/ia32_tsc_aux until we know how
+* to deal with them.
+*/
+   cpuid_count(*eax, *ecx, regs);
+   regs[3] = ~AMDID_RDTSCP;
+   break;
+
case CPUID_8000_0007:
cpuid_count(*eax, *ecx, regs);
/*
@@ -151,6 +158,11 @@
regs[2] = ~CPUID2_MON;

/*
+* Hide the performance and debug features.
+*/
+   regs[2] = ~CPUID2_PDCM;
+
+   /*
 * Hide thermal monitoring
 */
regs[3] = ~(CPUID_ACPI | CPUID_TM);
@@ -161,6 +173,11 @@
 */
regs[3] = ~(CPUID_MCA | CPUID_MCE | CPUID_MTRR);

+/*
+* Hide the debug store capability.
+*/
+   regs[3] = ~CPUID_DS;
+
/*
 * Disable multi-core.
 */
@@ -180,6 +197,7 @@

case CPUID__0006:
case CPUID__0007:
+   case CPUID__000A:
/*
 * Handle the access, but report 0 for
 * all options
@@ -203,12 +221,17 @@
case 0x4000:
regs[0] = CPUID_VM_HIGH;
bcopy(bhyve_id, regs[1], 4);
-   bcopy(bhyve_id, regs[2], 4);
-   bcopy(bhyve_id, regs[3], 4);
+   bcopy(bhyve_id + 4, regs[2], 4);
+   bcopy(bhyve_id + 8, regs[3], 4);
break;
+
default:
-   /* XXX: Leaf 5? */
-   return (0);
+   /*
+* The leaf value has already been clamped, so
+* simply pass this through
+*/
+   cpuid_count(*eax, *ecx, regs);
+   break;
}

*eax = regs[0];

best
Neel

 qualification   0x
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve hangs on init on recent 10-CURRENT

[Neel Natu]

Hi Dan,

On Sun, Jun 23, 2013 at 7:02 AM, Dan Mack m...@macktronics.com wrote:
 Is this just me?  When I start a new bhyve instance, it hangs on /sbin/init.  
  top shows bhyve consuming about 1 host core of CPU.   This is on a real 
 recent current.


I don't see it on my system with this:
FreeBSD 10.0-CURRENT (GENERIC) #0: Sun Jun 23 02:55:37 UTC 2013

My system has a much lower clock speed that yours - 2GHz versus
3.2GHz, but other than that I kept everything else similar to your
setup.

Is this a new system or were you able to use this successfully in the
past to run bhyve?

best
Neel



 root@darkstor:/vms/porter # ./vmrun.sh porter
 Launching virtual machine porter ...
 Consoles: userboot

 FreeBSD/amd64 User boot, Revision 1.1
 (r...@darkstor.macktronics.com, Sat Jun 22 22:31:58 CDT 2013)
 Loading /boot/defaults/loader.conf
 /boot/kernel/kernel text=0x6c21c7 data=0xa43b8+0xfa338 
 syms=[0x8+0xe76a8+0x8+0x147ee8]
 \
   __      _ _
  |  | |  _ \ / |  __ \
  | |___ _ __ ___  ___ | |_) | (___ | |  | |
  |  ___| '__/ _ \/ _ \|  _  \___ \| |  | |
  | |   | | |  __/  __/| |_) |) | |__| |
  | |   | | |||| |  |  |
  |_|   |_|  \___|\___||/|_/|_/````
  s` `.---...--.```   -/
  +Welcome to FreeBSD---+ +o   .--` /y:`  +.
  | |  yo`:.:o  `+-
  |  1. Back to Main Menu [Backspace]   |   y/   -/`   -o/
  |  2. Load System [D]efaults  |  .-  ::/sy+:.
  | |  / `--  /
  |  Boot Options:  | `:  :`
  |  3. Safe [M]ode... off  | `:  :`
  |  4. [S]ingle User. off  |  /  /
  |  5. [V]erbose. On   |  .--.
  | |   --  -.
  | |`:`  `:`
  | |  .-- `--.
  | | .---..
  +-+


 Booting...
 GDB: no debug ports present
 KDB: debugger backends: ddb
 KDB: current backend: ddb
 SMAP type=01 base= len=000a
 SMAP type=01 base=0010 len=8930
 Table 'APIC' at 0xf0500
 APIC: Found table at 0xf0500
 APIC: Using the MADT enumerator.
 MADT: Found CPU APIC ID 0 ACPI ID 0: enabled
 SMP: Added CPU 0 (AP)
 MADT: Found CPU APIC ID 1 ACPI ID 1: enabled
 SMP: Added CPU 1 (AP)
 Copyright (c) 1992-2013 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 10.0-CURRENT #9 r251956: Tue Jun 18 12:25:25 CDT 2013
 r...@darkstor.macktronics.com:/usr/obj/usr/src/sys/MACKGEN amd64
 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
 Preloaded elf kernel /boot/kernel/kernel at 0x80e91000.
 Hypervisor: Origin = BHyVBHyVBHyV
 Calibrating TSC clock ... TSC clock: 3299774008 Hz
 CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (3299.77-MHz K8-class CPU)
   Origin = GenuineIntel  Id = 0x306a9  Family = 0x6  Model = 0x3a  Stepping 
 = 9
   
 Features=0x8fa3ab7fFPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,PBE
   
 Features2=0xe3bae257SSE3,PCLMULQDQ,DTES64,DS_CPL,SMX,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,F16C,RDRAND,HV
   AMD Features=0x28100800SYSCALL,NX,RDTSCP,LM
   AMD Features2=0x1LAHF
   TSC: P-state invariant
 real memory  = 2302672896 (2196 MB)
 Physical memory chunk(s):
 0x1000 - 0x0009bfff, 634880 bytes (155 pages)
 0x0010 - 0x001f, 1048576 bytes (256 pages)
 0x00eb1000 - 0x85bc1fff, 2228293632 bytes (544017 pages)
 avail memory = 2133233664 (2034 MB)
 Event timer LAPIC quality 400
 ACPI APIC Table: BHYVE  BVMADT  
 INTR: Adding local APIC 1 as a target
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 FreeBSD/SMP: 2 package(s) x 1 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
 x86bios:  IVT 0x00-0x0004ff at 0xfe00
 x86bios: SSEG 0x001000-0x001fff at 0xff800022e000
 x86bios:  ROM 0x0a-0x0fefff at 0xfe0a
 APIC: CPU 0 has ACPI ID 0
 APIC: CPU 1 has ACPI ID 1
 random device not loaded; using insecure entropy
 ULE: setup cpu 0
 ULE: setup cpu 1
 ACPI: RSDP 0xf0400 00024 (v02 BHYVE )
 ACPI: XSDT 0xf0480 00034 (v01 BHYVE  BVXSDT   0001 INTL 20130517)
 ACPI: APIC 0xf0500 00052 (v01 BHYVE  BVMADT   0001 INTL 20130517)
 ACPI: FACP 0xf0600 

Re: bhyveload vm_create: Device not configured

[Neel Natu]

Hi Lars,

On Tue, Apr 30, 2013 at 2:06 AM, Eggert, Lars l...@netapp.com wrote:

 Hi,

 On Apr 30, 2013, at 9:38, Eggert, Lars l...@netapp.com wrote:
  I'm starting to play around with bhyve, and I'm trying to byhveload a
 disk image that I otherwise use to netboot some headless servers. When I
 tried that, I get:
 
  root@five:~ # bhyveload -d /home/elars/dst/rootfs test
  vm_create: Device not configured

 turns out that because I used the loader to kldload vmm.ko, I did not see
 the following error message, which is thrown when loading it when booted:

 vmx_init: processor does not support desired primary processor-based
 controls

 The CPU in this box reports as:

 CPU: Intel(R) Xeon(R) CPU   E5240  @ 3.00GHz (3000.17-MHz K8-class
 CPU)
   Origin = GenuineIntel  Id = 0x1067a  Family = 0x6  Model = 0x17
  Stepping = 10

 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE

 Features2=0xc0ce3bdSSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,OSXSAVE
   AMD Features=0x20100800SYSCALL,NX,LM
   AMD Features2=0x1LAHF
   TSC: P-state invariant, performance statistics

 Shouldn't this be sufficient to support bhyve?


Unfortunately not - this processor does not support extended page tables.

See http://ark.intel.com/Products/VirtualizationTechnology

Although this also shows that we need better error reporting ... one more
thing on the todo list.

best
Neel


 Thanks,
 Lars
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: FW: AMD and BHyve?

[Neel Natu]

Hi Craig,

On Sun, Apr 14, 2013 at 5:03 PM, Craig Rodrigues rodr...@crodrigues.orgwrote:

 On Sat, Apr 13, 2013 at 2:43 PM, Anish akgu...@gmail.com wrote:

  Hi Craig,
 
   AMD Features=0xea500800SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!
   AMD Features2=0x11fLAHF,CMP,*SVM*,ExtAPIC,CR8,Prefetch
 
  Yes, the AMD processor you have support AMD-V or SVM and I got BHyVe
  working on AMD processors with SVM Nested Page Table/RVI support,
  equivalent of Intel Extended Page Table.

 OK, cool.

 
 
  Neel and Peter have created projects/bhyve_svm for submitting AMD patches
  for BHyVe. It doesn't have complete patches yet, only vmcb related files.

 I looked at
 http://svnweb.freebsd.org/base/projects/bhyve_svm/sys/amd64/vmm/amd/amdv.c
 but it looks like the code will still give me an error if I try to kldload
 vmm.ko on my box.


Yes, that is expected.

The bhyve_svm branch does not have all the bits to support AMD just yet.

best
Neel



 Do you have a patch?


Thanks.

 --
 Craig
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: two more small improvements

[Neel Natu]

Hi Chris,

Thanks for the patches. Committed as:
http://svnweb.freebsd.org/base?view=revisionrevision=249396
http://svnweb.freebsd.org/base?view=revisionrevision=249351

best
Neel



On Fri, Apr 5, 2013 at 6:27 PM, Chris Torek to...@torek.net wrote:

 When I first went to run bhyve on one hardware box I had the
 entire system crash.  This turned out to be due to the BIOS
 disabling VMX.  A message came out on the console, but the scripts
 continued anyway, and then, boom.  I first looked at the wrong bit
 of code for fixing this, but I *think* I improved it anyway. :-)
 That's the first patch below.

 The second patch below (note: I'll put them somewhere on torek.net
 if that's better than just including them here) fixes the crash.
 The easiest way to reproduce it is to attempt to create a VM
 within a bhyve VM (bh1 is a running VM):

 [before patch -- note, some wraparound fixed]

 root@bh1:~ # kldload vmm
 vmx_init: processor does not support VMX operation
 module_register_init: MOD_LOAD (vmm, 0x81a133b0, 0) error 6
 root@bh1:~ # sysctl hw.vmm
 hw.vmm.destroy: beavis
 hw.vmm.create: beavis
 hw.vmm.pages_allocated: 0
 root@bh1:~ # sysctl hw.vmm.create
 hw.vmm.create: beavis
 root@bh1:~ # sysctl hw.vmm.create=
 hw.vmm.create: beavisvm exit[1]
 reason  VMX
 rip 0x81a1932c
 inst_length 6
 error   0
 exit_reason 50
 qualification   0xfff0
 vm exit[0]
 reason  VMX
 rip 0x81a1932c
 inst_length 6
 error   0
 exit_reason 50
 qualification   0xfff0

 [after patch]

 root@bh1:~ # kldload vmm
 vmx_init: processor does not support VMX operation
 module_register_init: MOD_LOAD (vmm, 0x81a133d0, 0) error 6
 root@bh1:~ # sysctl hw.vmm
 hw.vmm.destroy: beavis
 hw.vmm.create: beavis
 hw.vmm.pages_allocated: 0
 root@bh1:~ # sysctl hw.vmm.create
 hw.vmm.create: beavis
 root@bh1:~ # sysctl hw.vmm.create=
 hw.vmm.create: beavis
 sysctl: hw.vmm.create=: Device not configured
 root@bh1:~ #

 Chris

 changeset:   1962:7e204f321854
 user:Chris Torek chris.to...@gmail.com
 date:Fri Apr 05 19:10:09 2013 -0600
 files:   sys/amd64/vmm/intel/vmx.c
 description:
 improve test for VMX-mode

 We only need VMX extensions to be enabled, not locked.
 Moreover, we should not care about any VMX-in-SMX settings.


 diff --git a/sys/amd64/vmm/intel/vmx.c b/sys/amd64/vmm/intel/vmx.c
 --- a/sys/amd64/vmm/intel/vmx.c
 +++ b/sys/amd64/vmm/intel/vmx.c
 @@ -436,14 +436,33 @@
 return (ENXIO);
 }

 +/*
 + * Bits in MSR_IA32_FEATURE_CONTROL register.
 + *
 + * XXX move these to machine/specialreg.h
 + */
 +#defineMSR_IA32_FEAT_CTL_LOCK  0x01/* locks the
 control reg */
 +#defineMSR_IA32_FEAT_CTL_VMX_SMX_EN0x02/* enable VMXON in
 SMX mode */
 +#defineMSR_IA32_FEAT_CTL_VMX_EN0x04/* enable VMXON
 (non-SMX) */
 /*
 -* Verify that MSR_IA32_FEATURE_CONTROL lock and VMXON enable bits
 -* are set (bits 0 and 2 respectively).
 +* Verify that VMXON is allowed.
 +*
 +* According to Intel docs, we just need VMX_EN to be
 +* set.  If the LOCK bit is not set we can set or clear
 +* VMX_EN ourselves.  Once the LOCK bit is set no more
 +* changes are possible without a processor reset.
 +*
 +* Existing BIOSes currently set-and-lock the feature, but
 +* this code should work with BIOSes that don't lock it.
  */
 feature_control = rdmsr(MSR_IA32_FEATURE_CONTROL);
 -   if ((feature_control  0x5) != 0x5) {
 -   printf(vmx_init: VMX operation disabled by BIOS\n);
 -   return (ENXIO);
 +   if ((feature_control  MSR_IA32_FEAT_CTL_VMX_EN) == 0) {
 +   if (feature_control  MSR_IA32_FEAT_CTL_LOCK) {
 +   printf(vmx_init: VMX operation disabled by
 BIOS\n);
 +   return (ENXIO);
 +   }
 +   feature_control |= MSR_IA32_FEAT_CTL_VMX_EN;
 +   wrmsr(MSR_IA32_FEATURE_CONTROL, feature_control);
 }

 /* Check support for primary processor-based VM-execution controls
 */

 changeset:   1963:6cea2a2ed727
 tag: tip
 user:Chris Torek chris.to...@gmail.com
 date:Fri Apr 05 19:14:05 2013 -0600
 files:   sys/amd64/include/vmm.h sys/amd64/vmm/vmm.c
 sys/amd64/vmm/vmm_dev.c
 description:
 prevent host OS crash when VMX is disabled

 If VMX is disabled in the BIOS, vmm_init correctly returns an
 error, but this still leaves the module loaded and running, and a
 later sysctl to create a virtual machine crashed 

Re: suspend/resume on BHyVe

[Neel Natu]

Hi Lori,

On Mon, Mar 25, 2013 at 11:53 AM, Iori　YONEJI fivo.11235...@gmail.com wrote:
 Hello.

 I'm thinking of adding suspend/resume features on BHyVe.


Fantastic!

 For this, I think those below must be implemented.

   - virtual machine state command interface
   - saving registers per CPU
   - dumping physical memory
   - saving virt-io and other device emulation state


 To save registers, the sysctl used in bhyvectl (vmmctl command
 previously) is helpful,
 however, it's interface is no good because getting register value
 cause a sysctl call
 so to context-switch per one register, and for getting all registers,
 it's not efficient.
 I think it's more preferable to make a struct to set or unset boolean
 fields per register,
 to tell which registers kernel should return, and kernel returns those
 state with struct vmxctx.

 struct vmxctx is such.
  struct vmxctx {
 register_t  tmpstk[32]; /* vmx_return() stack */
 register_t  tmpstktop;

 register_t  guest_rdi;  /* Guest state */
 register_t  guest_rsi;
 :
 :


 And, considering memory dump, /dev/vmm/vmname is a file that is a map
 of guest memory,
 so memory dump doesn't seem hard, just stop vm, write back all guest
 cache, and copy
 memory file to a regular file.

 Finally, I don't know much about device state, but I think there must
 some state to be saved, like
 network stack.

 I'm not sure I wrote former, so I appreciate your ideas and suggestions.


I think that you are on the right track.

A brute force way of figuring out all the state must be saved is to
look at all the initialization functions that are called when a vm and
a vcpu are created. So, this would be vm_create() and vcpu_init() in
the kernel module.

There is also the hardware assist state that is maintained by the
processor (VT-x or SVM) and this includes things like guest
interruptibility, guest run state etc. I am assuming that it would be
sufficient to save the VMCS page after telling the processor to flush
any state it may be caching on chip.

There is also emulated pci bus, virtio devices and legacy isa device
state that would need to be saved by the userspace 'bhyve' process.

And finally there is the matter of how to communicate with 'bhyve'
process that it needs to suspend the virtual machine and write its
state to disk - perhaps a signal would be good enough place to start.

This certainly sounds like an interesting and challenging project and
we would be happy to help in any way we can.

best
Neel

 Thanks, Iori.
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve: simplify memory assignment to VMs

[Neel Natu]

Hi,

On Mon, Mar 11, 2013 at 11:08 AM, Neel Natu neeln...@gmail.com wrote:
 Hi,

 I am going to commit a change soon that simplifies how memory is
 assigned to a VM from a command line perspective.

 At the moment if you want to create a VM with say 8GB of memory you
 need to express this as -m 2048 -M 6144 or -m 1024 -M 7168 or some
 split that adds up to 8192MB. This is very flexible but is also very
 confusing to most users who don't know or care about the location of
 the PCI hole.

 I am going to change this so that the memory size is specified
 entirely with the -m memsize option and the -M option will
 disappear altogether.

 Of course, this means that scripts using -M will break and hence
 this heads up.

 I did consider making this a backward compatible change but given the
 stage of development of bhyve it seemed early to start accumulating
 compatibility crud.


Committed as r248477:
http://svnweb.freebsd.org/base?view=revisionrevision=248477

Please update your vmrun.sh scripts from /usr/share/examples/bhyve/vmrun.sh

best
Neel

 best
 Neel
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve in -current 4/14/13 can no longer support FreeBSD stable install

[Neel Natu]

Hi Alfred,

On Fri, Mar 15, 2013 at 11:21 PM, Alfred Perlstein bri...@mu.org wrote:
 On 3/15/13 8:03 PM, Neel Natu wrote:

 Hi Alfred,

 On Fri, Mar 15, 2013 at 9:48 AM, Alfred Perlstein bri...@mu.org wrote:

 Hey guys,

 I've been trying to get bhyve to install FreeBSD-stable for the past day
 on
 a -current machine:

 FreeBSD dan 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r248291: Fri Mar 15
 00:58:03 PDT 2013 root@dan:/usr/obj/usr/trees/head/sys/GENERIC amd64


 The stable snapshot I'm using to install is from here:

 ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-STABLE-amd64-20130216-r246877-bootonly.iso

 What will happen is during the extract process the install will hang.  No
 network IO happens and on the host I see bhyve's CPU hit 100% for each
 core
 assigned.

 Are you installing over the virtio-net interface?


 Yes, it is a bootonly image.  Using the virtio-net seems to cause the
 machine to go sidewise relatively quickly.


This should be fixed with r248368:
http://svnweb.freebsd.org/base?view=revisionrevision=248368

best
Neel

 thank you,
 -Alfred
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve in -current 4/14/13 can no longer support FreeBSD stable install

[Neel Natu]

Hi Alfred,

On Fri, Mar 15, 2013 at 9:48 AM, Alfred Perlstein bri...@mu.org wrote:
 Hey guys,

 I've been trying to get bhyve to install FreeBSD-stable for the past day on
 a -current machine:

 FreeBSD dan 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r248291: Fri Mar 15
 00:58:03 PDT 2013 root@dan:/usr/obj/usr/trees/head/sys/GENERIC amd64


 The stable snapshot I'm using to install is from here:
 ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-STABLE-amd64-20130216-r246877-bootonly.iso

 What will happen is during the extract process the install will hang.  No
 network IO happens and on the host I see bhyve's CPU hit 100% for each core
 assigned.


Are you installing over the virtio-net interface?

best
Neel

 For reference this is the bhyve command invoked via Neel's vmrun.sh.

 0 3915 3908   0 103  0 2122208 103428 -R+2 1048:45.75
 /usr/sbin/bhyve -c 2 -m 2048 -M 0 -AI -H -P -g 0 -s 0:0,hostbridge -s
 1:0,virtio-net,tap0 -s 2:0,virtio-blk,../freebsd-stable.img -s
 3:0,virtio-blk,../FreeBSD-9.1-STABLE-amd64-20130216-r246877-bootonly.iso -S
 31,uart,stdio freebsd-stable

 I tried to gdb the bhyve process but that was bad news.  Is there a set of
 steps usually taken to help pinpoint what is going on when we this state?

 thank you,
 -Alfred
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: vtnet cannot allocate interrupts

[Neel Natu]

Hi Marat,

On Thu, Feb 14, 2013 at 1:03 AM, Marat Bakeev haw...@hawara.com wrote:
 On 14/02/2013 06:20, Neel Natu wrote:
 You'll need to set 'hw.pci.honor_msi_blacklist' to '0' at the loader
 prompt or in /boot/loader.conf.
 Great! This fixes the 9.1-release guest! Thank you.


Once you update your host to r246846 you won't need to set the loader variable.
http://svnweb.freebsd.org/base?view=revisionrevision=246846

best
Neel


 I am working on a fix to bhyve that should let you boot the guest
 without needing the workaround. I hope to submit it in a day or two.

 best
 Neel
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: vtnet cannot allocate interrupts

[Neel Natu]

Hi Marat,

On Mon, Feb 11, 2013 at 10:56 AM, Marat Bakeev haw...@hawara.com wrote:
 11.02.2013 22:34, Bryan Venteicher пишет:

 Recent bhyve changed to use MSIX instead of MSI. I wonder if that
 is causing this. If you do:

 $ BHYVE_USE_MSI=true bhyve ...

 does it work?


 No, it won`t work, but it seems i had msix disabled in loader.conf, as in
 example from here - http://www.emulsoft.com/bhyve-ufs.txt.
 I enabled msix, and the vtnet device is now loading fine.


Glad to hear.

 Do I need other options in loader.conf, like these?:

 smbios.bios.vendor=BHYVE
 console=userboot
 hw.pci.honor_msi_blacklist=0


No, you don't need these anymore.

Here is a simple set of instructions that should get you started:
http://people.freebsd.org/~neel/bhyve/bhyve_instructions.txt

They do not have all the bells-and-whistles as some of the other
HOWTOs but it should get you a working VM in a short amount of time.

best
Neel


 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org