Re: bhyve VM crashed

[Takuya ASADA]

Did you execute bhyveload before bhyve?


2014-04-09 4:26 GMT+09:00 Steve Wills swi...@freebsd.org:

 On Tue, Apr 08, 2014 at 11:43:27AM -0700, Peter Grehan wrote:
  Hi Steve,
 
   I experienced a bhyve VM crash with this message:
  
   vm exit[1]
reason  VMX
rip 0x80c7490b
inst_length 3
status  0
exit_reason 2
 
That's a triple-fault :(
 
(the exit reasons are the VMX ones in Appendix C, Vol 3 of the Intel
 SDM).
 
Any idea what was happening at the time ?

 The VM was building ports. I could try to deduce which ports, but I
 wouldn't be
 100% sure.

  Also, for the guest, do you
  have the config that was being used (e.g. bhyve command line) ?

 /usr/sbin/bhyve -c 4 -m 12G -A -H -P -g 0 -s 0:0,hostbridge -s 1:0,lpc -s
 2:0,virtio-net,tap1,mac=58:9c:fc:01:01:02 -s
 3:0,virtio-blk,/vm/100amd64/disk.img -l com1,/dev/nmdm1A 100amd64

 Thanks,
 Steve
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: CFT: vmrc Virtual Machine rc script

[Takuya ASADA]

Hi Michael,

I tried openbsd6, ubuntu7, centos9 by remove comment out on install.sh, and
did provision command like this:
$ sudo /usr/local/etc/rc.d/vm provision openbsd6

Then I get an error:
vm: provisioning VM openbsd6
/usr/local/vm//openbsd6/openbsd6.conf: 21: Syntax error: ( unexpected

Same as another images.

Where this error coming from?

Note: I'm on FreeBSD-CURRENT, not 10.0.


2014-01-27 12:09 GMT+02:00 Michael Dexter edi...@callfortesting.org:


 Hello all,

 For the one-week anniversary of bhyve's release, I wish to share the
 next iteration of my virtual machine scripts that have evolved from a
 2005 jail rc/disk image framework to a FreeBSD 9.0 bhyve binary and VM
 build script to a 10.* sequential provision script. It's primary purpose
 is to facilitate the exposure of bugs in bhyve and related FreeBSD
 components but you may find it useful for continuous VM use or for
 integration into your own scripts.

 Because bhyve is only the latest multiplicity solution on FreeBSD, this
 rc script includes basic jail and qemu support and could include chroot,
 simh and GXemul support. Of these, QEMU is intended to help prepare OSs
 like OpenBSD that require VirtIO and console adjustments prior to
 booting under bhyve.

 Synopsis:

 Unpack vmrc.tar
 Run 'sh install.sh' or perform the steps manually
 Verify your host_nic and host_zpool in /usr/local/etc/vm.conf
 /usr/local/etc/rc.d/vm(Show usage)
 /usr/local/etc/rc.d/vm provision vm0  (Note: /usr/local/vm/vm0/vm0.conf)
 (Fetches OS install media, formats disk, installs to disk)
 /usr/local/etc/rc.d/vm start vm0  (Loads and boots vm0)

 Alternatively:
 /usr/local/etc/rc.d/vm iso vm0(Fetches and boots and ISO image)

 ISO mode will prepare a disk image to install the OS to and includes
 fetch support for FreeBSD, Ubuntu and CentOS.

 Manual scripts exist in install/ to fetch FreeNAS, pfSense and OpenBSD.

 Status:

 Know issues and known desired features are at the end of
 instructions.txt plus various fix: notes throughout the script. This
 is not mature enough for packaging yet but does some things quite
 reliably and is ready for feedback. Because development is ongoing, I
 invite you to mail me about issues directly and I can post it to
 something like GitHub if there is interest.

 Please read the instructions.txt to avoid FAQ's and I have done my best
 to make the script itself readable.


 Site: http://bhyve.org/vmrc/

 Download: http://bhyve.org/vmrc/vmrc.tar (latest version)


 Thank you Neel, Peter, John and Tycho for your hard work on bhyve.

 Thank you Devin for greatly improving this script and Allan at
 ScaleEngine for testing it and providing resources for bhyve's overall
 improvement.


 Thank you for testing bhyve!

 Michael
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: CFT: vmrc Virtual Machine rc script

[Takuya ASADA]

2014-02-27 16:25 GMT+02:00 Matt Donovan kitchet...@gmail.com:

 The error is coming from the openbsd.conf line 21

The error is due to a syntax errror in your openbs.conf. Can you post your
 configuration as it appears you have an extra ).


It's true, but I even never changes it. Which means the configuration file
is may broken by default.

This line:

grub_boot_cmd=echo -e \kopenbsd -h com0 (hd0,openbsd1)/bsd\nboot\ |
grub-bhyve -r hd0 -m ${host_vmroot}/${vm_name}/device.map -M $vm_ram
$vm_name
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Linux on BHyVe in 10.0-RELEASE

[Takuya ASADA]

Hi,

I want to know which Linux distribution is supported on BHyVe
(+grub2-bhyve) in 10.0-RELEASE.
I tried to these distributions, but I haven't succeeded to install any
choice.
On these tests, I used modified vmrun.sh script for all test, source code
is here:
https://gist.github.com/syuu1228/8552418

* Fedora 20
Fedora 20 has disk image for VM, I just tried to boot it from the image:

./linuxrun.sh -d Fedora-x86_64-20-20131211.1-sda.raw fedora

grub linux (hd1,msdos1)/boot/vmlinuz-3.11.10-301.fc20.x86_64
grub initrd (hd1,msdos1)/boot/initramfs-3.11.10-301.fc20.x86_64.img
grub boot

but bhyve dies by unhandled rdmsr:
[1.676041] device-mapper: uevent: version 1.0.3
[1.676556] device-mapper: ioctl: 4.25.0-ioctl (2013-06-26) initialised:
dm-de...@redhat.com
vm exit rdmsr 0xe8, cpu 0

* CentOS 6.5
Tried to install with following commands but no luck:

./linuxrun.sh centos

grub linux (hd0)/isolinux/vmlinuz
grub initrd (hd0)/isolinux/initrd.img
grub boot

Got following error:

NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 32610k freed
Unknown WRMSR code 391, val 200f, cpu 0

* Ubuntu 13.10
Almost boot-up but couldn't mount cdrom.

./linuxrun.sh -d ubuntu.img -I ubuntu-13.10-desktop-amd64.iso ubuntu

grub linux (hd0)/casper/vmlinuz.efi
grub initrd (hd0)/casper/initrd.lz
grub boot

Got following error, maybe it can prevents by passing root= argument?

[6.380384] Btrfs loaded
Scanning for Btrfs filesystems
done.
mount: can't read '/etc/fstab': No such file or directory
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev
failed: No such file or directory
done.
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have requested /sbin/init.
No init found. Try passing init= bootarg.

BusyBox v1.20.2 (Ubuntu 1:1.20.0-8.1ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)


Any suggestions?
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Linux on BHyVe in 10.0-RELEASE

[Takuya ASADA]

Hmm, looks like neel@ adding ignoring unimplemented MSR access patch on
-CURRENT.
Does it means we can't use (some version of) Linux on 10.0-RELEASE?

https://github.com/freebsd/freebsd/commit/5dab6f9ed3816204ff73fd075abfaba1ed5c7673


2014/1/22 Takuya ASADA s...@dokukino.com

 Hi,

 I want to know which Linux distribution is supported on BHyVe
 (+grub2-bhyve) in 10.0-RELEASE.
 I tried to these distributions, but I haven't succeeded to install any
 choice.
 On these tests, I used modified vmrun.sh script for all test, source code
 is here:
 https://gist.github.com/syuu1228/8552418

 * Fedora 20
 Fedora 20 has disk image for VM, I just tried to boot it from the image:

 ./linuxrun.sh -d Fedora-x86_64-20-20131211.1-sda.raw fedora

 grub linux (hd1,msdos1)/boot/vmlinuz-3.11.10-301.fc20.x86_64
 grub initrd (hd1,msdos1)/boot/initramfs-3.11.10-301.fc20.x86_64.img
 grub boot

 but bhyve dies by unhandled rdmsr:
 [1.676041] device-mapper: uevent: version 1.0.3
 [1.676556] device-mapper: ioctl: 4.25.0-ioctl (2013-06-26)
 initialised: dm-de...@redhat.com
 vm exit rdmsr 0xe8, cpu 0

 * CentOS 6.5
 Tried to install with following commands but no luck:

 ./linuxrun.sh centos

 grub linux (hd0)/isolinux/vmlinuz
 grub initrd (hd0)/isolinux/initrd.img
 grub boot

 Got following error:

 NET: Registered protocol family 1
 Trying to unpack rootfs image as initramfs...
 Freeing initrd memory: 32610k freed
 Unknown WRMSR code 391, val 200f, cpu 0

 * Ubuntu 13.10
 Almost boot-up but couldn't mount cdrom.

 ./linuxrun.sh -d ubuntu.img -I ubuntu-13.10-desktop-amd64.iso ubuntu

 grub linux (hd0)/casper/vmlinuz.efi
 grub initrd (hd0)/casper/initrd.lz
 grub boot

 Got following error, maybe it can prevents by passing root= argument?

 [6.380384] Btrfs loaded
 Scanning for Btrfs filesystems
 done.
 mount: can't read '/etc/fstab': No such file or directory
 Begin: Running /scripts/local-bottom ... done.
 done.
 Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev
 failed: No such file or directory
 done.
 mount: mounting /sys on /root/sys failed: No such file or directory
 mount: mounting /proc on /root/proc failed: No such file or directory
 Target filesystem doesn't have requested /sbin/init.
 No init found. Try passing init= bootarg.

 BusyBox v1.20.2 (Ubuntu 1:1.20.0-8.1ubuntu1) built-in shell (ash)
 Enter 'help' for a list of built-in commands.

 (initramfs)


 Any suggestions?

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Linux on BHyVe in 10.0-RELEASE

[Takuya ASADA]

Oh, I didn't know bhyve-script.tar is still on bhyve.org (I didn't notice
there's Tools tab on the page).
I'll try it.


2014/1/22 Matt Donovan kitchet...@gmail.com

 Hello

 On Jan 21, 2014 8:38 PM, Takuya ASADA s...@dokukino.com wrote:
 
  Hmm, looks like neel@ adding ignoring unimplemented MSR access patch on
  -CURRENT.
  Does it means we can't use (some version of) Linux on 10.0-RELEASE?
 
 
 https://github.com/freebsd/freebsd/commit/5dab6f9ed3816204ff73fd075abfaba1ed5c7673
 
 
  2014/1/22 Takuya ASADA s...@dokukino.com
 
   Hi,
  
   I want to know which Linux distribution is supported on BHyVe
   (+grub2-bhyve) in 10.0-RELEASE.
   I tried to these distributions, but I haven't succeeded to install any
   choice.
   On these tests, I used modified vmrun.sh script for all test, source
 code
   is here:
   https://gist.github.com/syuu1228/8552418
  
   * Fedora 20
   Fedora 20 has disk image for VM, I just tried to boot it from the
 image:
  
   ./linuxrun.sh -d Fedora-x86_64-20-20131211.1-sda.raw fedora
  
   grub linux (hd1,msdos1)/boot/vmlinuz-3.11.10-301.fc20.x86_64
   grub initrd (hd1,msdos1)/boot/initramfs-3.11.10-301.fc20.x86_64.img
   grub boot
  
   but bhyve dies by unhandled rdmsr:
   [1.676041] device-mapper: uevent: version 1.0.3
   [1.676556] device-mapper: ioctl: 4.25.0-ioctl (2013-06-26)
   initialised: dm-de...@redhat.com
   vm exit rdmsr 0xe8, cpu 0
  
   * CentOS 6.5
   Tried to install with following commands but no luck:
  
   ./linuxrun.sh centos
  
   grub linux (hd0)/isolinux/vmlinuz
   grub initrd (hd0)/isolinux/initrd.img
   grub boot
  
   Got following error:
  
   NET: Registered protocol family 1
   Trying to unpack rootfs image as initramfs...
   Freeing initrd memory: 32610k freed
   Unknown WRMSR code 391, val 200f, cpu 0
  
   * Ubuntu 13.10
   Almost boot-up but couldn't mount cdrom.
  
   ./linuxrun.sh -d ubuntu.img -I ubuntu-13.10-desktop-amd64.iso ubuntu
  
   grub linux (hd0)/casper/vmlinuz.efi
   grub initrd (hd0)/casper/initrd.lz
   grub boot
  
   Got following error, maybe it can prevents by passing root= argument?
  
   [6.380384] Btrfs loaded
   Scanning for Btrfs filesystems
   done.
   mount: can't read '/etc/fstab': No such file or directory
   Begin: Running /scripts/local-bottom ... done.
   done.
   Begin: Running /scripts/init-bottom ... mount: mounting /dev on
 /root/dev
   failed: No such file or directory
   done.
   mount: mounting /sys on /root/sys failed: No such file or directory
   mount: mounting /proc on /root/proc failed: No such file or directory
   Target filesystem doesn't have requested /sbin/init.
   No init found. Try passing init= bootarg.
  
   BusyBox v1.20.2 (Ubuntu 1:1.20.0-8.1ubuntu1) built-in shell (ash)
   Enter 'help' for a list of built-in commands.
  
   (initramfs)
  
  
   Any suggestions?
  
 

 I have gotten Centos 6.5 to boot however I used the bhyve scripts which
 can be gotten at bhyve.org. haven't tried any other currently

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Linux on BHyVe in 10.0-RELEASE

[Takuya ASADA]

Append results using bhyyve-scripts.

* CentOS 6.5 with bhyve-scripts
Dies at same place: Unknown WRMSR code 391, val 200f, cpu 0

* Ubuntu 13.10 with bhyve-scripts
Dies at same place: (initramfs)
Anyway I'll test with ahci-cd, which is point-outed by Peter.

* Debian 7.3 with bhyve-scripts
Able to install, bootup!
But serial console sometimes outputs AT+GCAP or something, looks like AT
command?
Maybe I mis-configured Debian during installation?

* OpenBSD 5.4 with bhyve-scripts
Hangs at virtio0 initialization:

scsibus0 at vioblk0: 2 targets
sd0 at scsibus0 targ 0 lun 0: VirtIO, Block Device,  SCSI3 0/direct fixed
sd0: 10240MB, 512 bytes/sector, 20971520 sectors
virtio0: couldn't map interrupt


So, only Debian had worked on my environment for now.


2014/1/22 Takuya ASADA s...@dokukino.com

 Hi,

 I want to know which Linux distribution is supported on BHyVe
 (+grub2-bhyve) in 10.0-RELEASE.
 I tried to these distributions, but I haven't succeeded to install any
 choice.
 On these tests, I used modified vmrun.sh script for all test, source code
 is here:
 https://gist.github.com/syuu1228/8552418

 * Fedora 20
 Fedora 20 has disk image for VM, I just tried to boot it from the image:

 ./linuxrun.sh -d Fedora-x86_64-20-20131211.1-sda.raw fedora

 grub linux (hd1,msdos1)/boot/vmlinuz-3.11.10-301.fc20.x86_64
 grub initrd (hd1,msdos1)/boot/initramfs-3.11.10-301.fc20.x86_64.img
 grub boot

 but bhyve dies by unhandled rdmsr:
 [1.676041] device-mapper: uevent: version 1.0.3
 [1.676556] device-mapper: ioctl: 4.25.0-ioctl (2013-06-26)
 initialised: dm-de...@redhat.com
 vm exit rdmsr 0xe8, cpu 0

 * CentOS 6.5
 Tried to install with following commands but no luck:

 ./linuxrun.sh centos

 grub linux (hd0)/isolinux/vmlinuz
 grub initrd (hd0)/isolinux/initrd.img
 grub boot

 Got following error:

 NET: Registered protocol family 1
 Trying to unpack rootfs image as initramfs...
 Freeing initrd memory: 32610k freed
 Unknown WRMSR code 391, val 200f, cpu 0

 * Ubuntu 13.10
 Almost boot-up but couldn't mount cdrom.

 ./linuxrun.sh -d ubuntu.img -I ubuntu-13.10-desktop-amd64.iso ubuntu

 grub linux (hd0)/casper/vmlinuz.efi
 grub initrd (hd0)/casper/initrd.lz
 grub boot

 Got following error, maybe it can prevents by passing root= argument?

 [6.380384] Btrfs loaded
 Scanning for Btrfs filesystems
 done.
 mount: can't read '/etc/fstab': No such file or directory
 Begin: Running /scripts/local-bottom ... done.
 done.
 Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev
 failed: No such file or directory
 done.
 mount: mounting /sys on /root/sys failed: No such file or directory
 mount: mounting /proc on /root/proc failed: No such file or directory
 Target filesystem doesn't have requested /sbin/init.
 No init found. Try passing init= bootarg.

 BusyBox v1.20.2 (Ubuntu 1:1.20.0-8.1ubuntu1) built-in shell (ash)
 Enter 'help' for a list of built-in commands.

 (initramfs)


 Any suggestions?

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Linux on BHyVe in 10.0-RELEASE

[Takuya ASADA]

2014/1/22 Takuya ASADA s...@dokukino.com

 Append results using bhyyve-scripts.

 * CentOS 6.5 with bhyve-scripts
 Dies at same place: Unknown WRMSR code 391, val 200f, cpu 0


This issue was resolved by back-porting -CURRENT patch:

svn co svn://svn.freebsd.org/base/head
cd head
svn diff -r259634:r259635  ~/msr.diff
cd /usr/src
patch -p0  ~/msr.diff
cd usr.sbin/bhyve
make
make install

Then CentOS 6.5 worked on my machine without any problems.

Since we modified source code maybe we should not call it worked on
10.0-RELEASE, actually it's good for temporally workaround.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Linux on BHyVe in 10.0-RELEASE

[Takuya ASADA]

Thanks, I'll test your image.


2014/1/22 Peter Grehan gre...@freebsd.org

 Hi Takuya,


  * Debian 7.3 with bhyve-scripts
 Able to install, bootup!
 But serial console sometimes outputs AT+GCAP or something, looks like AT
 command?
 Maybe I mis-configured Debian during installation?


  I think I saw this but it eventually popped up a login prompt. Guessing
 it was attempting to detect if a modem was present.


  * OpenBSD 5.4 with bhyve-scripts
 Hangs at virtio0 initialization:

 scsibus0 at vioblk0: 2 targets
 sd0 at scsibus0 targ 0 lun 0: VirtIO, Block Device,  SCSI3 0/direct
 fixed
 sd0: 10240MB, 512 bytes/sector, 20971520 sectors
 virtio0: couldn't map interrupt


  OpenBSD 5.4 doesn't have the change to use MSI interrupts for virtio
 devices (http://www.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/
 virtio_pci.c 1.7)

  I built a 5.4/amd64 flashrd image with this change at
http://people.freebsd.org/~grehan/flashimg.amd64-20131014.bz2

  5.5 isn't displaying serial console output - I've yet to debug this;
 probably an issue with grub-bhyve.

 later,

 Peter.


___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Standalone binary loader for BHyVe

[Takuya ASADA]

Hi,

I implemented standalone binary loader for BHyVe.
This allows to load custom program on guest memory area, and run it from
specified entry point address.
I wrote this for test code of BHyVe suspend/resume project(Yoneji Iori's
work), but it might be useful for some other purposes.

This is a patch:
http://people.freebsd.org/~syuu/bhyve_standalone_guest.diff

Most of the code is borrowed from userboot, but userboot looks like only
for FreeBSD kernel, so I copied those code to bhyveload instead of
modifying userboot APIs.

Here's a example of usage:
 echo main:  test.S
 echo inb \$0x50, %al  test.S
 as -o test.o test.S
 objcopy -I elf64-x86-64-freebsd -O binary test.o test.bin
 sudo bhyveload -m 256 -S test.bin:0x7c00 test
 sudo bhyve -m 256 -e test
 # bhyve says Unhandled inb 0x0050, because there's no inb 0x50 handler.

Another example of usage:
 svn co svn://svn.freebsd.org/base/user/syuu/bhyve_standalone_guest_test1
 cd bhyve_standalone_guest_test1
 make
 make run
 # because test1 is COM1 test program, bhyve continues to show asterisk on
the console.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: suspend/resume on BHyVe

[Takuya ASADA]

You can dump whole struct just a binary array(don't look into it in your
program), and get sizeof(vmxctx) via ioctl return. It should enough to
suspend/resume.
Then if AMD SVM mode will have different values on the structure, your
implementation won't break.

Does it make sense?  Peter, Neel

And, AMD SVM's vmxctx will not have much difference than Intel's one, I
guess.
(VMCS/VMCB layout are differ, though. But it's different topic.)

2013年4月2日火曜日 Iori YONEJI fivo.11235...@gmail.com:

 Hi all,

 I couldn't spare much time to consider about this topic because I have
 to move to new house, but now I'm OK.

 I'm implementing registers-save feature including:
   1. libvmmapi with get_vmxctx
   2. kernel supports of 1.
   3. user-land hack using 1, that saves registers and some
 configurations into (possibly msgpack) binary file.

 But I noticed that vmxctx is depending on Intel VT-x, and we also need
 to support AMD SVM,
 so I need common struct registers.

 I heard about SVM support but it's not in my repository copy(Revision:
 248994),
 how much do I have to care about AMD?

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

suspend/resume on BHyVe

[Takuya ASADA]

Hi,

I had some discussion with Iori about the project since last year, and now I'm
suggesting him to apply Google Summer of Code'13 with the project.
(GSoC'13 will start next month)

  For this, I think those below must be implemented.
 
- virtual machine state command interface
- saving registers per CPU
- dumping physical memory
- saving virt-io and other device emulation state
 
 
  To save registers, the sysctl used in bhyvectl (vmmctl command
  previously) is helpful,



 Maybe he meant ioctl.



 however, it's interface is no good because getting register value
  cause a sysctl call
  so to context-switch per one register, and for getting all registers,
  it's not efficient.

 I think it's more preferable to make a struct to set or unset boolean
  fields per register,
  to tell which registers kernel should return, and kernel returns those
  state with struct vmxctx.
 
  struct vmxctx is such.
   struct vmxctx {
  register_t  tmpstk[32]; /* vmx_return() stack */
  register_t  tmpstktop;
 
  register_t  guest_rdi;  /* Guest state */
  register_t  guest_rsi;
  :
  :


It looks we don't really have to take care register values on VMCS on here,
just registers on vmxctx is enough(described below).
Then, how about to add vmxctx dumping ioctl?


 
 
  And, considering memory dump, /dev/vmm/vmname is a file that is a map
  of guest memory,
  so memory dump doesn't seem hard, just stop vm, write back all guest
  cache, and copy
  memory file to a regular file.
 
  Finally, I don't know much about device state, but I think there must
  some state to be saved, like
  network stack.
 
  I'm not sure I wrote former, so I appreciate your ideas and suggestions.
 

 I think that you are on the right track.

 A brute force way of figuring out all the state must be saved is to
 look at all the initialization functions that are called when a vm and
 a vcpu are created. So, this would be vm_create() and vcpu_init() in
 the kernel module.

 There is also the hardware assist state that is maintained by the
 processor (VT-x or SVM) and this includes things like guest
 interruptibility, guest run state etc. I am assuming that it would be
 sufficient to save the VMCS page after telling the processor to flush
 any state it may be caching on chip.


 I think, just dump whole VMCS page after calling VMCLEAR instruction is
easiest way to do this.
(I also considered to dump only necessary values on VMCS by VMREAD
instruction, but maybe it's easy to break guest state mistakenly, and we
don't get advantage by doing that way.)

Then maybe we need VMCS dumping ioctl here.

There is also emulated pci bus, virtio devices and legacy isa device
 state that would need to be saved by the userspace 'bhyve' process.


What is the necessary operation for virtio devices to suspend/resume?
Maybe dump all rings of the devices?
It doesn't have registers, right?


 And finally there is the matter of how to communicate with 'bhyve'
 process that it needs to suspend the virtual machine and write its
 state to disk - perhaps a signal would be good enough place to start.


How about this idea:
bhyvectl sends VM_SUSPEND ioctl.
If the guests is in VMX non-root mode, VM_SUSPEND ioctl handler sends IPI
to interrupt the guest thread.
Then the guest thread breaks vmx_run() loop, exit to userland with exitcode
VM_EXITCODE_SUSPEND.

Or, if the guests is not in VM_RUN ioctl but performing userland work(such
as running virtio host-side driver), maybe you just need to wait bhyve
process sends VM_RUN ioctl.
When bhyve sends VM_RUN ioctl, vmm.ko should not perform VMEnter.
It should just returns VM_EXITCODE_SUSPEND.

On both cases, vmm.ko returns VM_EXITCODE_SUSPEND at the end.
Then bhyve process can perform suspend action in VM_EXITCODE_SUSPEND
handler.

I think this is simple.


 This certainly sounds like an interesting and challenging project and
 we would be happy to help in any way we can.

 best
 Neel

  Thanks, Iori.
  ___
  freebsd-virtualization@freebsd.org mailing list
  http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
  To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
 ___
 freebsd-virtualization@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
 To unsubscribe, send any mail to 
 freebsd-virtualization-unsubscr...@freebsd.org
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: suspend/resume on BHyVe

[Takuya ASADA]

Hi,

 A brute force way of figuring out all the state must be saved is to
 look at all the initialization functions that are called when a vm and
 a vcpu are created. So, this would be vm_create() and vcpu_init() in
 the kernel module.

I think these things are need to dump by resume action:

- guest memory area
- struct vmxctx
- VMCS page
- VLAPIC registers
- virtio device state (dumping virtio ring?)
- current configuration of bhyve process (with was configured by an
arguments passed to bhyve process)

Maybe there's more.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Howto close BHyVe console

[Takuya ASADA]

Hi,

Just for a little question,
I could run BHyVe on my PC with neel's guest image by following CFT
page(http://callfortesting.org/bhyve/), it just worked without
problem, but I couldn't shutdown guest OS safely.
shutdown command in guest OS doesn't poweroff the VM, and I have no
idea to close bhyve console.
Is there any shortcut-key or commands to shutdown it?

I also tried vmmctl --vm=vm name --destroy, but it suddenly causes panic.
kill bhyve pid  vmmctl --vm=vm name --destroy had worked, so I
guess  the panic reason is because /usr/sbin/bhyve still holding
/dev/vmm/vm name.

Takuya ASADA
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org