Re: Event channels in KVM?
Anthony Liguori wrote: Kapadia, Vivek wrote: I came across this thread looking for an efficient event channel mechanism between two guests (running on different cpu cores). While I can use available emulated IO mechanism (guest1-host kernel driver-Qemu1-Qemu2) in conjunction with interrupt mechanism (Qemu2-host kernel driver-guest2) in KVM, this involves several context switches. Xen handles notifications in hypervisor via hypercall and hence is likely more efficient. They almost certainly aren't more efficient. An event channel notification involves a hypercall to the hypervisor. When using VT, the performance difference between a vmcall exit vs. a pio exit is quite small (especially compared to the overhead of the exit). We're talking in the order of nanoseconds compared to microseconds. What makes KVM particularly different from Xen is that in KVM, the PIO operation results in a direct transition to QEMU. In Xen, typically event channel notifications result in a bit being set in a bitmap which then results in an interrupt injection depending on the next opportunity the hypervisor has to schedule/run the receiving domain. This is not deterministic and can potentially be a very long period of time. Event channels are inherently asynchronous whereas PIO notifications in KVM are synchronous. Since the scheduler isn't involved and control never leaves the CPU, the KVM PIO notifications are actually extremely efficient. IMHO, it's one of KVM's best design features. If you make the pio operation wake up another guest, then the operation becomes asynchronous. There's really no fundamental different between Xen and kvm here, and both will require the same number of context switches (one) to transfer control. Handling a pio that is completely internal to the guest is different (Xen has to schedule dom0 or the stub domain), but that's not related to interguest communications. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Event channels in KVM?
I came across this thread looking for an efficient event channel mechanism between two guests (running on different cpu cores). While I can use available emulated IO mechanism (guest1-host kernel driver-Qemu1-Qemu2) in conjunction with interrupt mechanism (Qemu2-host kernel driver-guest2) in KVM, this involves several context switches. Xen handles notifications in hypervisor via hypercall and hence is likely more efficient. Is there a way I can perform direct notification (guest1-host kernel driver-guest2) in kvm? Thanks, Vivek. Matt Anger wrote: I was referring to the bounce from host kernel to qemu and then back to the host kernel for my BE driver. Xen: guest - guest kernel driver- host kernel driver For both situations I need a FE and BE driver, but for KVM I need to modify QEMU and teach it how to pass the virtio calls to my Host driver. In Xen BE driver and Host driver are the same and I don't have to recompile any part of Xen. You can implement (host) kernel level device model too which will have same result with Xen, i.e. skip Qemu. I don't think Xen KVM has fundamental difference here, but different implementation style. Xen implement BE driver in driver domain kernel, but KVM like to implement in user level host process. Different people has different favorite, I am neutral actually :) Thx, eddie -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Event channels in KVM?
Kapadia, Vivek wrote: I came across this thread looking for an efficient event channel mechanism between two guests (running on different cpu cores). While I can use available emulated IO mechanism (guest1-host kernel driver-Qemu1-Qemu2) in conjunction with interrupt mechanism (Qemu2-host kernel driver-guest2) in KVM, this involves several context switches. Xen handles notifications in hypervisor via hypercall and hence is likely more efficient. They almost certainly aren't more efficient. An event channel notification involves a hypercall to the hypervisor. When using VT, the performance difference between a vmcall exit vs. a pio exit is quite small (especially compared to the overhead of the exit). We're talking in the order of nanoseconds compared to microseconds. What makes KVM particularly different from Xen is that in KVM, the PIO operation results in a direct transition to QEMU. In Xen, typically event channel notifications result in a bit being set in a bitmap which then results in an interrupt injection depending on the next opportunity the hypervisor has to schedule/run the receiving domain. This is not deterministic and can potentially be a very long period of time. Event channels are inherently asynchronous whereas PIO notifications in KVM are synchronous. Since the scheduler isn't involved and control never leaves the CPU, the KVM PIO notifications are actually extremely efficient. IMHO, it's one of KVM's best design features. It used to be in HVM that since things like PIO operations are inherently synchronous, and there's not point in a VM waiting around for the asynchronous event channel notification to result in qemu-dm invocation,, there was a very special code path in the hypervisor to ensure that Domain-0 was scheduled immediately when receiving an event channel notification from an HVM domain. This was an important optimization because event channel notification latency was prohibitively high. Now in the context of the stub domain, I'm not sure what changes they've made. In the earliest prototypes of stub domain, the same short cutting logic was maintained but the stub domain was executed instead of domain-0. Is there a way I can perform direct notification (guest1-host kernel driver-guest2) in kvm? Between guests, we don't have a notification framework today. You can use IPC from two QEMU processes and I'd expect that to perform pretty well. I'm not sure you can get much advantages from doing things in the kernel because you cannot avoid the heavy weight exit. Regards, Anthony Liguori -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
FW: Event channels in KVM?
Anthony Liguori anthony at codemonkey.ws writes: Kapadia, Vivek wrote: I came across this thread looking for an efficient event channel mechanism between two guests (running on different cpu cores). While I can use available emulated IO mechanism (guest1-host kernel driver-Qemu1-Qemu2) in conjunction with interrupt mechanism (Qemu2 -host kernel driver-guest2) in KVM, this involves several context switches. Xen handles notifications in hypervisor via hypercall and hence is likely more efficient. They almost certainly aren't more efficient. An event channel notification involves a hypercall to the hypervisor. When using VT, the performance difference between a vmcall exit vs. a pio exit is quite small (especially compared to the overhead of the exit). We're talking in the order of nanoseconds compared to microseconds. I agree. The hypercall and PIO exit won't be significantly different, but what I am referring to here is a particular case of guest to guest notification and number of context switches involved in the process. While I cannot avoid heavy weight exit either way (exit via hypercall or PIO), by handling the notification in hypervisor, I would think we can reduce the overhead due to extra the context switches. What makes KVM particularly different from Xen is that in KVM, the PIO operation results in a direct transition to QEMU. In Xen, typically event channel notifications result in a bit being set in a bitmap which then results in an interrupt injection depending on the next opportunity the hypervisor has to schedule/run the receiving domain. This is not deterministic and can potentially be a very long period of time. I have guests pinned to cpu cores, and hence the notifications will be synchronous. Is there a way I can perform direct notification (guest1-host kernel driver-guest2) in kvm? Between guests, we don't have a notification framework today. You can use IPC from two QEMU processes and I'd expect that to perform pretty well. I'm not sure you can get much advantages from doing things in the kernel because you cannot avoid the heavy weight exit. Has KVM future plans to support guest-to-guest notification framework? If not, how easy or difficult would it be today to write simple guest-to-guest notification in KVM host driver? Any hints would be helpful. Thanks. Regards, Anthony Liguori -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Event channels in KVM?
Thanks for the info, I've been looking into it by trying to look around kvm source code. Apparently I have to write a kernel driver for the guest os and then also write backend driver and modify qemu to use it? Is that correct? That seems ugly, especially since now my io goes guest-guest kernel driver-host kernel kvm-qemu-host kernel driver With Xen event-channels my driver gets the event notification directly with no user space intervention removing the middle 2 steps and I don't have to touch any of Xen's codebase. My driver just registers for an event-channel. Thanks -Matt On Fri, Sep 19, 2008 at 12:14 PM, Anthony Liguori [EMAIL PROTECTED] wrote: Javier Guerra wrote: On Fri, Sep 19, 2008 at 1:31 PM, Anthony Liguori [EMAIL PROTECTED] wrote: Matt Anger wrote: Does KVM have any interface similar to event-channels like Xen does? Basically a way to send notifications between the host and guest. virtio is the abstraction we use. But virtio is based on the standard hardware interfaces of the PC--PIO, MMIO, and interrupts. this is rather low-level, it would be nice to have a multiplatform interface to this abstraction. That's exactly the purpose of virtio. virtio is a high-level, cross platform interface. It's been tested on x86, PPC, s390, and I believe ia64. It also works in lguest. It happens to use PIO, MMIO, and interrupts on x86 under KVM but other virtio implementations exist for other platforms. just for kicks, i've found and printed Rusty's paper about it. hope it's current :-) The other good thing to look at is the lguest documentation. You can skip to just the virtio bits if you're so inclined. It's really quite thoroughly documented. Regards, Anthony Liguori -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Event channels in KVM?
Matt Anger wrote: Thanks for the info, I've been looking into it by trying to look around kvm source code. Apparently I have to write a kernel driver for the guest os and then also write backend driver and modify qemu to use it? Is that correct? That seems ugly, especially since now my io goes guest-guest kernel driver-host kernel kvm-qemu-host kernel driver With Xen event-channels my driver gets the event notification directly with no user space intervention removing the middle 2 steps and I Why virtio needs start from guest user space while event channel not? It is exactly same, event channel can start from user space or kernel space, same for virtio. The only difference is guest-host communication needs to go thru Qemu (host process) in your description model, while Xen handle event channel in hypervisor. But this can be enahnced also IMO by employing a kernel level device model which does inter-communication for CPU if performance is critical, but mostly probably not. don't have to touch any of Xen's codebase. My driver just registers for an event-channel. But you still need the counter part side to handle it. I think you may want to reuse Xen BE side code which is a wrong assumption. Thanks -Matt On Fri, Sep 19, 2008 at 12:14 PM, Anthony Liguori [EMAIL PROTECTED] wrote: Javier Guerra wrote: On Fri, Sep 19, 2008 at 1:31 PM, Anthony Liguori [EMAIL PROTECTED] wrote: Matt Anger wrote: Does KVM have any interface similar to event-channels like Xen does? Basically a way to send notifications between the host and guest. virtio is the abstraction we use. But virtio is based on the standard hardware interfaces of the PC--PIO, MMIO, and interrupts. this is rather low-level, it would be nice to have a multiplatform interface to this abstraction. That's exactly the purpose of virtio. virtio is a high-level, cross platform interface. It's been tested on x86, PPC, s390, and I believe ia64. It also works in lguest. It happens to use PIO, MMIO, and interrupts on x86 under KVM but other virtio implementations exist for other platforms. just for kicks, i've found and printed Rusty's paper about it. hope it's current :-) The other good thing to look at is the lguest documentation. You can skip to just the virtio bits if you're so inclined. It's really quite thoroughly documented. Regards, Anthony Liguori -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Event channels in KVM?
I was referring to the bounce from host kernel to qemu and then back to the host kernel for my BE driver. Xen: guest - guest kernel driver- host kernel driver For both situations I need a FE and BE driver, but for KVM I need to modify QEMU and teach it how to pass the virtio calls to my Host driver. In Xen BE driver and Host driver are the same and I don't have to recompile any part of Xen. -Matt On Mon, Sep 22, 2008 at 5:30 PM, Dong, Eddie [EMAIL PROTECTED] wrote: Matt Anger wrote: Thanks for the info, I've been looking into it by trying to look around kvm source code. Apparently I have to write a kernel driver for the guest os and then also write backend driver and modify qemu to use it? Is that correct? That seems ugly, especially since now my io goes guest-guest kernel driver-host kernel kvm-qemu-host kernel driver With Xen event-channels my driver gets the event notification directly with no user space intervention removing the middle 2 steps and I Why virtio needs start from guest user space while event channel not? It is exactly same, event channel can start from user space or kernel space, same for virtio. The only difference is guest-host communication needs to go thru Qemu (host process) in your description model, while Xen handle event channel in hypervisor. But this can be enahnced also IMO by employing a kernel level device model which does inter-communication for CPU if performance is critical, but mostly probably not. don't have to touch any of Xen's codebase. My driver just registers for an event-channel. But you still need the counter part side to handle it. I think you may want to reuse Xen BE side code which is a wrong assumption. Thanks -Matt On Fri, Sep 19, 2008 at 12:14 PM, Anthony Liguori [EMAIL PROTECTED] wrote: Javier Guerra wrote: On Fri, Sep 19, 2008 at 1:31 PM, Anthony Liguori [EMAIL PROTECTED] wrote: Matt Anger wrote: Does KVM have any interface similar to event-channels like Xen does? Basically a way to send notifications between the host and guest. virtio is the abstraction we use. But virtio is based on the standard hardware interfaces of the PC--PIO, MMIO, and interrupts. this is rather low-level, it would be nice to have a multiplatform interface to this abstraction. That's exactly the purpose of virtio. virtio is a high-level, cross platform interface. It's been tested on x86, PPC, s390, and I believe ia64. It also works in lguest. It happens to use PIO, MMIO, and interrupts on x86 under KVM but other virtio implementations exist for other platforms. just for kicks, i've found and printed Rusty's paper about it. hope it's current :-) The other good thing to look at is the lguest documentation. You can skip to just the virtio bits if you're so inclined. It's really quite thoroughly documented. Regards, Anthony Liguori -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Event channels in KVM?
Matt Anger wrote: I was referring to the bounce from host kernel to qemu and then back to the host kernel for my BE driver. Xen: guest - guest kernel driver- host kernel driver For both situations I need a FE and BE driver, but for KVM I need to modify QEMU and teach it how to pass the virtio calls to my Host driver. In Xen BE driver and Host driver are the same and I don't have to recompile any part of Xen. You can implement (host) kernel level device model too which will have same result with Xen, i.e. skip Qemu. I don't think Xen KVM has fundamental difference here, but different implementation style. Xen implement BE driver in driver domain kernel, but KVM like to implement in user level host process. Different people has different favorite, I am neutral actually :) Thx, eddie -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Event channels in KVM?
Does KVM have any interface similar to event-channels like Xen does? Basically a way to send notifications between the host and guest. Thanks, -Matt -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Event channels in KVM?
Matt Anger wrote: Does KVM have any interface similar to event-channels like Xen does? Basically a way to send notifications between the host and guest. virtio is the abstraction we use. But virtio is based on the standard hardware interfaces of the PC--PIO, MMIO, and interrupts. Regards, Anthony Liguori Thanks, -Matt -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
