Directly reserve an interrupt IDT entry for Hyper-V
Hello, Sending to Xen, drivers and virtualization mailing lists since this might be of interest to the folks on these aliases. I am working for Microsoft to improve the performance of FreeBSD running on Hyper-V. Right now I am adding a feature in the vmbus driver which could handle the host-guest channel communications on all vCPUs simultaneously. In order to achieve this, the hypervisor will send same interrupt concurrently on all the vCPUs. The traditional way on FreeBSD to set up interrupt handling for devicse, such as calling bus_alloc_resource() to reserve an IRQ line, and then calling bus_setup_intr() to create a vector, doesn't seem to work in this case. It seems if the interrupt is routed via legacy IRQ, it can only be active on one vCPU at a time. In order to allow the same interrupt to be handled on all vCPUs concurrently, all I need is an IDT entry, not an IRQ line. I checked current FreeBSD code. It looks to me Xen directly uses the vector number IDT_EVTCHN (0x93) to achieve the same purpose. I am proposing both Xen and Hyper-V share this same vector. Following is a little bit detail of my proposal for the changes in the current kernel. 1. In machdep.c: #ifdef XENHVM setidt(IDT_EVTCHN, IDTVEC(xen_intr_upcall), SDT_SYSIGT, SEL_UPL, 0); #else setidt(IDT_EVTCHN, IDTVEC(hv_vmbus_intr), SDT_SYSIGT, SEL_UPL, 0); #endif 2. Apic_vector.S Add IDTVEC(hv_vmbus_intr) to call Hyper-V vmbus interrupt service routine. Any thoughts, objections and feedbacks are all welcome. Thanks, Wei Hu Open Source Technology Center Microsoft China ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: libvirt and rebooting of a bhyve VM
Allan Jude wrote: On 2014-08-19 11:50, John Nielsen wrote: On Aug 19, 2014, at 9:40 AM, Roman Bogorodskiy no...@freebsd.org wrote: Craig Rodrigues wrote: Roman, I am using libvirt and bhyve according to this XML: http://libvirt.org/drvbhyve.html and it works great. I gave a presentation at BAFUG on this: http://www.slideshare.net/CraigRodrigues1/libvirt-bhyve I have one question. If I reboot the bhyve VM started with libvirt with shutdown -r now, the VM shuts down, but it does not restart. How can I get the machine to reboot with shutdown -r now when started with libvirt? Hi Craig, Unfortunately, I'm not sure how to get the reboot working. Moreover, I get the same behaviour when starting bhyve manually -- when I do a reboot, bhyve(8) exits as soon as the system is ready to restart. So looks like that's a default bhyve behaviour or I'm missing something? Wasn't changing this the intention of r267216 (MFCed as r270071)? Roman, was your 10-STABLE built after that revision? JN ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org Yes, this revision adds the ability to 'reboot'. This does not exit bhyve at all, so there is no exit level You can 'reboot' a bhyve externally using: bhyvectl --force-reset --vm=xxx or force a poweroff with: bhyvectl --force-reset --vm=xxx SIGTERM simulates an ACPI shutdown the bhyvectl --destroy is to clean up, the instance remains even after a clean shutdown. In the past, it had to be destroyed and recreated to start the VM again, although now with reboot support, that may not be required. I have updated to 10-STABLE that includes commit r270071. I can see bhyveload has --force-reset flag. However, when I boot a VM and do 'reboot' there, bhyve(8) exits with exit code 0. The same happens when I do bhyvectl --force-reset. Any ideas how to debug why bhyve(8) exits in this case? Roman Bogorodskiy pgpuzYzrwjScz.pgp Description: PGP signature
Re: Directly reserve an interrupt IDT entry for Hyper-V
On 20/08/14 11:19, Wei Hu wrote: Hello, Sending to Xen, drivers and virtualization mailing lists since this might be of interest to the folks on these aliases. I am working for Microsoft to improve the performance of FreeBSD running on Hyper-V. Right now I am adding a feature in the vmbus driver which could handle the host-guest channel communications on all vCPUs simultaneously. In order to achieve this, the hypervisor will send same interrupt concurrently on all the vCPUs. The traditional way on FreeBSD to set up interrupt handling for devicse, such as calling bus_alloc_resource() to reserve an IRQ line, and then calling bus_setup_intr() to create a vector, doesn't seem to work in this case. It seems if the interrupt is routed via legacy IRQ, it can only be active on one vCPU at a time. In order to allow the same interrupt to be handled on all vCPUs concurrently, all I need is an IDT entry, not an IRQ line. I checked current FreeBSD code. It looks to me Xen directly uses the vector number IDT_EVTCHN (0x93) to achieve the same purpose. I am proposing both Xen and Hyper-V share this same vector. Following is a little bit detail of my proposal for the changes in the current kernel. 1. In machdep.c: #ifdef XENHVM setidt(IDT_EVTCHN, IDTVEC(xen_intr_upcall), SDT_SYSIGT, SEL_UPL, 0); #else setidt(IDT_EVTCHN, IDTVEC(hv_vmbus_intr), SDT_SYSIGT, SEL_UPL, 0); #endif 2. Apic_vector.S Add IDTVEC(hv_vmbus_intr) to call Hyper-V vmbus interrupt service routine. Any thoughts, objections and feedbacks are all welcome. Hello, I don't think using the same IDT vector is the right approach, I would just pick a different IDT vector and use that for Hyper-V. Using the same IDT vector (like your suggestion above) would prevent shipping a kernel with with both Hyper-V and Xen support (like it's done now in GENERIC). Roger. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: libvirt and rebooting of a bhyve VM
On Tuesday, August 19, 2014 2:33:19 pm Craig Rodrigues wrote:
On Tue, Aug 19, 2014 at 10:32 AM, John Baldwin j...@freebsd.org wrote:
The exit code is different. An ACPI shutdown uses an exit code of 1
whereas a
reboot uses an exit code of 0 IIRC.
Hi,
I have a CURRENT system, and ran some tests doing a shutdown -r and
shutdown -p verified that you are right,
with respect to the exit codes.
Looking at src/usr.sbin/bhyve/bhyverun.c, I see that
fbsdrun_start_thread() - vm_loop()
Inside vm_loop(), there is this line:
rc = (*handler[exitcode])(ctx, vmexit[vcpu], vcpu);
That line ends up calling vmexit_suspend().
In vmexit_suspend() there is this:
switch (how) {
case VM_SUSPEND_RESET:
exit(0);
case VM_SUSPEND_POWEROFF:
exit(1);
case VM_SUSPEND_HALT:
exit(2);
case VM_SUSPEND_TRIPLEFAULT:
exit(3);
default:
fprintf(stderr, vmexit_suspend: invalid reason %d\n, how);
exit(100);
}
I think that:
(1) vmrun.sh should be changed a little bit to accomodate this new logic
(2) the bhyve man page should document these return codes.
vmrun.sh already handles this. It breaks out of the loop for any non-zero
exit code and only continues the loop for exit code 0.
It probably would be good to document the return codes in the man page, but
eventually they will all go away. The long term plan is to make bhyve operate
more like other hypervisors in which case a single process invocation will
mimic a power-on to power-off cycle of a machine and you wouldn't need
vmrun.sh at all (you would just run bhyve with appropriate args / config
file). In that case, the return codes will change since it won't exit for
reboots anymore.
--
John Baldwin
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: Directly reserve an interrupt IDT entry for Hyper-V
On Wednesday, August 20, 2014 9:31:54 am Roger Pau Monné wrote: On 20/08/14 11:19, Wei Hu wrote: Hello, Sending to Xen, drivers and virtualization mailing lists since this might be of interest to the folks on these aliases. I am working for Microsoft to improve the performance of FreeBSD running on Hyper-V. Right now I am adding a feature in the vmbus driver which could handle the host-guest channel communications on all vCPUs simultaneously. In order to achieve this, the hypervisor will send same interrupt concurrently on all the vCPUs. The traditional way on FreeBSD to set up interrupt handling for devicse, such as calling bus_alloc_resource() to reserve an IRQ line, and then calling bus_setup_intr() to create a vector, doesn't seem to work in this case. It seems if the interrupt is routed via legacy IRQ, it can only be active on one vCPU at a time. In order to allow the same interrupt to be handled on all vCPUs concurrently, all I need is an IDT entry, not an IRQ line. I checked current FreeBSD code. It looks to me Xen directly uses the vector number IDT_EVTCHN (0x93) to achieve the same purpose. I am proposing both Xen and Hyper-V share this same vector. Following is a little bit detail of my proposal for the changes in the current kernel. 1. In machdep.c: #ifdef XENHVM setidt(IDT_EVTCHN, IDTVEC(xen_intr_upcall), SDT_SYSIGT, SEL_UPL, 0); #else setidt(IDT_EVTCHN, IDTVEC(hv_vmbus_intr), SDT_SYSIGT, SEL_UPL, 0); #endif 2. Apic_vector.S Add IDTVEC(hv_vmbus_intr) to call Hyper-V vmbus interrupt service routine. Any thoughts, objections and feedbacks are all welcome. Hello, I don't think using the same IDT vector is the right approach, I would just pick a different IDT vector and use that for Hyper-V. Using the same IDT vector (like your suggestion above) would prevent shipping a kernel with with both Hyper-V and Xen support (like it's done now in GENERIC). Roger. Hmm, can't you make this a runtime check to only call setidt() if you detect you are under the appropriate hypervisor? Also, bhyve currently has a hackish way of requesting a free IDT slot. Perhaps it would be best if I added little API to reserve an IDT slot assuming that callers could accept a dynamic IDT vector rather than a static one. -- John Baldwin ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Directly reserve an interrupt IDT entry for Hyper-V
On 20/08/14 17:31, John Baldwin wrote: On Wednesday, August 20, 2014 9:31:54 am Roger Pau Monné wrote: Hello, I don't think using the same IDT vector is the right approach, I would just pick a different IDT vector and use that for Hyper-V. Using the same IDT vector (like your suggestion above) would prevent shipping a kernel with with both Hyper-V and Xen support (like it's done now in GENERIC). Roger. Hmm, can't you make this a runtime check to only call setidt() if you detect you are under the appropriate hypervisor? Also, bhyve currently has a hackish way of requesting a free IDT slot. Perhaps it would be best if I added little API to reserve an IDT slot assuming that callers could accept a dynamic IDT vector rather than a static one. That would work for Xen. The IDT vector doesn't need to be fixed since it's registered with Xen when the system boots. Roger. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: libvirt and rebooting of a bhyve VM
On Tue, Aug 19, 2014, Craig Rodriques wrote: On Tue, Aug 19, 2014 at 12:18 PM, Allan Jude allanjude at freebsd.org wrote: The name of the vm is in the title of the bhyve process, but yes, it Hmm, OK, that would require me to grep the processes and send SIGTERM to the right pid. That's not ideal, especially if I start lots of VM's, but it is workable. I would think that hacking in support to the bhyve program to use pidfile_write() in libutil, with something like: /var/run/bhyve.vmname.pid as the default pathname for the pidfile would make this a lot easier to manage from a script. -Kurt ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Directly reserve an interrupt IDT entry for Hyper-V
Hi John, On Wed, Aug 20, 2014 at 8:31 AM, John Baldwin j...@freebsd.org wrote: On Wednesday, August 20, 2014 9:31:54 am Roger Pau Monné wrote: On 20/08/14 11:19, Wei Hu wrote: Hello, Sending to Xen, drivers and virtualization mailing lists since this might be of interest to the folks on these aliases. I am working for Microsoft to improve the performance of FreeBSD running on Hyper-V. Right now I am adding a feature in the vmbus driver which could handle the host-guest channel communications on all vCPUs simultaneously. In order to achieve this, the hypervisor will send same interrupt concurrently on all the vCPUs. The traditional way on FreeBSD to set up interrupt handling for devicse, such as calling bus_alloc_resource() to reserve an IRQ line, and then calling bus_setup_intr() to create a vector, doesn't seem to work in this case. It seems if the interrupt is routed via legacy IRQ, it can only be active on one vCPU at a time. In order to allow the same interrupt to be handled on all vCPUs concurrently, all I need is an IDT entry, not an IRQ line. I checked current FreeBSD code. It looks to me Xen directly uses the vector number IDT_EVTCHN (0x93) to achieve the same purpose. I am proposing both Xen and Hyper-V share this same vector. Following is a little bit detail of my proposal for the changes in the current kernel. 1. In machdep.c: #ifdef XENHVM setidt(IDT_EVTCHN, IDTVEC(xen_intr_upcall), SDT_SYSIGT, SEL_UPL, 0); #else setidt(IDT_EVTCHN, IDTVEC(hv_vmbus_intr), SDT_SYSIGT, SEL_UPL, 0); #endif 2. Apic_vector.S Add IDTVEC(hv_vmbus_intr) to call Hyper-V vmbus interrupt service routine. Any thoughts, objections and feedbacks are all welcome. Hello, I don't think using the same IDT vector is the right approach, I would just pick a different IDT vector and use that for Hyper-V. Using the same IDT vector (like your suggestion above) would prevent shipping a kernel with with both Hyper-V and Xen support (like it's done now in GENERIC). Roger. Hmm, can't you make this a runtime check to only call setidt() if you detect you are under the appropriate hypervisor? Also, bhyve currently has a hackish way of requesting a free IDT slot. Perhaps it would be best if I added little API to reserve an IDT slot assuming that callers could accept a dynamic IDT vector rather than a static one. Yup, it'll be good to get rid of vmm_ipi.c. best Neel -- John Baldwin ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: libvirt and rebooting of a bhyve VM
On Wed, Aug 20, 2014 at 8:15 AM, John Baldwin j...@freebsd.org wrote:
It probably would be good to document the return codes in the man page, but
eventually they will all go away. The long term plan is to make bhyve operate
more like other hypervisors in which case a single process invocation will
mimic a power-on to power-off cycle of a machine and you wouldn't need
vmrun.sh at all (you would just run bhyve with appropriate args / config
file). In that case, the return codes will change since it won't exit for
reboots anymore.
Hi,
I wasn't aware of the longer term direction of bhyve. If the return
codes may change
in the future, I don't know if it is a good idea to document them in
the man page.
However, documenting the return codes in comments in the vmrun.sh
script might be OK, since it
is just an example that can be changed easily.
What do you think of these changes to vmrun.sh:
(1) Move bhyvectl --destroy outside of the while loop. It doesn't
need to run if the VM has been reset.
(2) Write some comments about return code of bhyve process.
--
Craig
--- share/examples/bhyve/vmrun.sh.orig 2014-08-20 11:38:09.788714905 -0700
+++ share/examples/bhyve/vmrun.sh 2014-08-20 11:43:14.158706821 -0700
@@ -173,9 +173,9 @@
virtio_diskdev=$disk_dev0
-while [ 1 ]; do
- ${BHYVECTL} --vm=${vmname} --destroy /dev/null 21
+${BHYVECTL} --vm=${vmname} --destroy /dev/null 21
+while [ 1 ]; do
file ${virtio_diskdev} | grep boot sector /dev/null
rc=$?
if [ $rc -ne 0 ]; then
@@ -237,6 +237,14 @@
-l com1,${console} \
${installer_opt}\
${vmname}
+ #
+ # bhyve returns the following status codes:
+ # 0 - VM has been reset
+ # 1 - VM has been powered off
+ # 2 - VM has been halted
+ # 3 - VM generated a triple fault
+ # all other non-zero status codes are errors
+ #
if [ $? -ne 0 ]; then
break
fi
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
freebsd-virtualization-unsubscr...@freebsd.org
Re: libvirt and rebooting of a bhyve VM
Kurt Lidl wrote: On Tue, Aug 19, 2014, Craig Rodriques wrote: On Tue, Aug 19, 2014 at 12:18 PM, Allan Jude allanjude at freebsd.org wrote: The name of the vm is in the title of the bhyve process, but yes, it Hmm, OK, that would require me to grep the processes and send SIGTERM to the right pid. That's not ideal, especially if I start lots of VM's, but it is workable. I would think that hacking in support to the bhyve program to use pidfile_write() in libutil, with something like: /var/run/bhyve.vmname.pid as the default pathname for the pidfile would make this a lot easier to manage from a script. IMHO, as for the pidfile, it's the thing that could be easily done in the script itself. What would be really good from my point of view is to have a control socket (Unix socket) and support for some sort of a protocol encoded in a machine readable format. So it could be used to request VM information, perform actions on it, doing a dynamic configuration (e.g. plugging devices etc). Also, it'd be easier to monitor a socket presence than a pid. Of course, all this could be done by extending bhyvectl as well. This way seems less convenient from me because from the scripting point of view command line application features are much harder to probe than doing the same via a defined protocol. Roman Bogorodskiy pgp1BzDQvTI3v.pgp Description: PGP signature
