Re: [Qemu-devel] Will the ELI incorporated in theKVM?
GaoYi gaoyi...@gmail.com wrote on 20/09/2012 08:42:51 AM: The CPU isolation in Hitachi patches is just to improve the real time performance of GUEST. The core of it, direct IRQ delivery, is very similar to that of ELI. For the ELI patches, (1) Since EOI part of ELI is already supported by the Intel Sandy Bridge CPUs and requires modifications on GUEST code, it should not be included in the KVM. Only the ELI delivery part, which plays a vital role in performance improvement, should be considered. Giving to the guest direct access to the EOI MSR (if x2APIC is available) is what we call ELI completion. Note this mechanism is not so simple, there are some cases (which are not part of the critical path) where ELI must trap EOIs. For the APLOS paper evaluation we didn't have CPUs with x2APIC so we simulated the behavior changing the guest code. In any case, as you can see in the paper, the big part of the improvement comes from ELI delivery. ELI completion improvement will be even smaller with the latest KVM EOI optimizations for the memory based LAPIC. (2) It should be provided in the kvm-kmod or qemu-kvm ( not just for some linux kernel as Hitachi patches do), to make this part independent of linux kernel version. Exactly, ELI only modifies the kvm kernel module and qemu-kvm but we should also modify VFIO for newer kvm versions.
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
On 2012-09-20 08:58, Abel Gordon wrote: GaoYi gaoyi...@gmail.com wrote on 20/09/2012 08:42:51 AM: The CPU isolation in Hitachi patches is just to improve the real time performance of GUEST. The core of it, direct IRQ delivery, is very similar to that of ELI. For the ELI patches, (1) Since EOI part of ELI is already supported by the Intel Sandy Bridge CPUs and requires modifications on GUEST code, it should not be included in the KVM. Only the ELI delivery part, which plays a vital role in performance improvement, should be considered. Giving to the guest direct access to the EOI MSR (if x2APIC is available) is what we call ELI completion. Note this mechanism is not so simple, there are some cases (which are not part of the critical path) where ELI must trap EOIs. For the APLOS paper evaluation we didn't have CPUs with x2APIC so we simulated the behavior changing the guest code. In any case, as you can see in the paper, the big part of the improvement comes from ELI delivery. ELI completion improvement will be even smaller with the latest KVM EOI optimizations for the memory based LAPIC. (2) It should be provided in the kvm-kmod or qemu-kvm ( not just for some linux kernel as Hitachi patches do), to make this part independent of linux kernel version. Exactly, ELI only modifies the kvm kernel module and qemu-kvm but we should also modify VFIO for newer kvm versions. Again: If you think the feature is non-invasive, send patches against the kernel and QEMU. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
On Thu, Sep 20, 2012 at 09:14:31AM +0200, Jan Kiszka wrote: On 2012-09-20 08:58, Abel Gordon wrote: GaoYi gaoyi...@gmail.com wrote on 20/09/2012 08:42:51 AM: The CPU isolation in Hitachi patches is just to improve the real time performance of GUEST. The core of it, direct IRQ delivery, is very similar to that of ELI. For the ELI patches, (1) Since EOI part of ELI is already supported by the Intel Sandy Bridge CPUs and requires modifications on GUEST code, it should not be included in the KVM. Only the ELI delivery part, which plays a vital role in performance improvement, should be considered. Giving to the guest direct access to the EOI MSR (if x2APIC is available) is what we call ELI completion. Note this mechanism is not so simple, there are some cases (which are not part of the critical path) where ELI must trap EOIs. For the APLOS paper evaluation we didn't have CPUs with x2APIC so we simulated the behavior changing the guest code. In any case, as you can see in the paper, the big part of the improvement comes from ELI delivery. ELI completion improvement will be even smaller with the latest KVM EOI optimizations for the memory based LAPIC. (2) It should be provided in the kvm-kmod or qemu-kvm ( not just for some linux kernel as Hitachi patches do), to make this part independent of linux kernel version. Exactly, ELI only modifies the kvm kernel module and qemu-kvm but we should also modify VFIO for newer kvm versions. Again: If you think the feature is non-invasive, send patches against the kernel and QEMU. And explain why it is better than what modern HW provides. -- Gleb.
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
On 09/20/2012 10:19 AM, Gleb Natapov wrote: Again: If you think the feature is non-invasive, send patches against the kernel and QEMU. And explain why it is better than what modern HW provides. If it's non-invasive (and easily maintainable), it doesn't have to be better. -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
On Thu, Sep 20, 2012 at 12:18:11PM +0300, Avi Kivity wrote: On 09/20/2012 10:19 AM, Gleb Natapov wrote: Again: If you think the feature is non-invasive, send patches against the kernel and QEMU. And explain why it is better than what modern HW provides. If it's non-invasive (and easily maintainable), it doesn't have to be better. If it were all that it would have been applied already. -- Gleb.
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
[putting Avi on CC as the final decision maker] On 2012-09-19 15:37, Muli Ben-Yehuda wrote: On Tue, Sep 18, 2012 at 03:08:40PM +0200, Jan Kiszka wrote: On 2012-09-18 14:50, GaoYi wrote: Hi Jan, I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html. I wonder whether this mechanism will be incorporated in KVM someday. Likely not. Both Intel and AMD will soon ship hardware that obsoletes this invasive and imperfect software solution, see also [1]. Hi Jan, I think this decision may be short-sighted. First, it remains to be seen how good the upcoming hardware support for interrupt virtualization will be once it ships. Second, history has been pretty consistent in that it takes several years for hardware to catch up to software, and even when it does, software sometimes still wins. (There are workloads where shadow page tables still outperform EPT/NPT, to give a concrete example). Although ELI works today, I'd be happy to hear what parts of it you find invasive and imperfect (what software ever is perfect?) It's imperfect as you need to dedicate a core to pure guest-mode load and cannot run userspace on that core (cannot walk through userspace-based device models e.g.). And it requires that magic bar to map the shadow IDT into the guest (hmm, I think Hitachi avoided this). It's invasive as it has to change Linux to maintain those isolated slave CPUs. That is, of course, based on code that was published by Hitachi. Yours may differ but will still have to solve the same problems. Back then in our discussion, it was unclear if and when real hardware will actually show up. Now the spec is already there, hardware should follow within months (typically). I'm not aware of limitations regarding zero-exit virtualization so far. But please share any insights you collected from studying the specs or recent patches! Jan -- Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
On 09/19/2012 05:10 PM, Jan Kiszka wrote: Although ELI works today, I'd be happy to hear what parts of it you find invasive and imperfect (what software ever is perfect?) It's imperfect as you need to dedicate a core to pure guest-mode load and cannot run userspace on that core (cannot walk through userspace-based device models e.g.). And it requires that magic bar to map the shadow IDT into the guest (hmm, I think Hitachi avoided this). It's invasive as it has to change Linux to maintain those isolated slave CPUs. That is, of course, based on code that was published by Hitachi. Yours may differ but will still have to solve the same problems. Back then in our discussion, it was unclear if and when real hardware will actually show up. Now the spec is already there, hardware should follow within months (typically). I'm not aware of limitations regarding zero-exit virtualization so far. But please share any insights you collected from studying the specs or recent patches! This matches my thoughts on the matter exactly. I would add that ELI adds a maintenance burden of supporting rarely used code, as ELI can hardly be called mainstream. On the other hand, the new hardware support will be used by every guest running on new hardware, for IPIs, virtual interrupts, and assigned device interrupts. -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
On 2012-09-19 16:43, Abel Gordon wrote: It's imperfect as you need to dedicate a core to pure guest-mode load and cannot run userspace on that core (cannot walk through userspace-based device models e.g.). That's not correct. For the evaluation, we dedicated a core for each guest to maximize the performance but this is not a requirement. You can over-commit cores and share them across multiple VMs. In this case, you will be sharing the cycles among all the VMs and the performance per VM will be degraded. In addition, if you share a core, an interrupt for a guest may be raised while the corresponding VCPU is not running. Depending on the workload, most of the interrupts will be generated while the VCPU runs or not. and cannot run userspace on that core (cannot walk through userspace-based device models e.g.). That's not correct. We can run any thread (including user-space threads) on the same core we run the VMs (VCPU threads). In fact, we did that for the ELI evaluation: we shared a single core to run the VCPU thread and ALL the host threads (including qemu I/O thread). And it requires that magic bar to map the shadow IDT into the guest (hmm, I think Hitachi avoided this). Hitachi uses a different technique which seems to have the 2 disadvantages you previously mentioned while ELI doesn't have these disadvantages. It's invasive as it has to change Linux to maintain those isolated slave CPUs. That is, of course, based on code that was published by Hitachi. Yours may differ but will still have to solve the same problems. ELI does not require isolated/slave CPUs. OK. Show patches. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
It's imperfect as you need to dedicate a core to pure guest-mode load and cannot run userspace on that core (cannot walk through userspace-based device models e.g.). That's not correct. For the evaluation, we dedicated a core for each guest to maximize the performance but this is not a requirement. You can over-commit cores and share them across multiple VMs. In this case, you will be sharing the cycles among all the VMs and the performance per VM will be degraded. In addition, if you share a core, an interrupt for a guest may be raised while the corresponding VCPU is not running. Depending on the workload, most of the interrupts will be generated while the VCPU runs or not. and cannot run userspace on that core (cannot walk through userspace-based device models e.g.). That's not correct. We can run any thread (including user-space threads) on the same core we run the VMs (VCPU threads). In fact, we did that for the ELI evaluation: we shared a single core to run the VCPU thread and ALL the host threads (including qemu I/O thread). And it requires that magic bar to map the shadow IDT into the guest (hmm, I think Hitachi avoided this). Hitachi uses a different technique which seems to have the 2 disadvantages you previously mentioned while ELI doesn't have these disadvantages. It's invasive as it has to change Linux to maintain those isolated slave CPUs. That is, of course, based on code that was published by Hitachi. Yours may differ but will still have to solve the same problems. ELI does not require isolated/slave CPUs.
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
On Tue, Sep 18, 2012 at 03:08:40PM +0200, Jan Kiszka wrote: On 2012-09-18 14:50, GaoYi wrote: Hi Jan, I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html. I wonder whether this mechanism will be incorporated in KVM someday. Likely not. Both Intel and AMD will soon ship hardware that obsoletes this invasive and imperfect software solution, see also [1]. Hi Jan, I think this decision may be short-sighted. First, it remains to be seen how good the upcoming hardware support for interrupt virtualization will be once it ships. Second, history has been pretty consistent in that it takes several years for hardware to catch up to software, and even when it does, software sometimes still wins. (There are workloads where shadow page tables still outperform EPT/NPT, to give a concrete example). Although ELI works today, I'd be happy to hear what parts of it you find invasive and imperfect (what software ever is perfect?) Cheers, Muli
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
The CPU isolation in Hitachi patches is just to improve the real time performance of GUEST. The core of it, direct IRQ delivery, is very similar to that of ELI. For the ELI patches, (1) Since EOI part of ELI is already supported by the Intel Sandy Bridge CPUs and requires modifications on GUEST code, it should not be included in the KVM. Only the ELI delivery part, which plays a vital role in performance improvement, should be considered. (2) It should be provided in the kvm-kmod or qemu-kvm ( not just for some linux kernel as Hitachi patches do), to make this part independent of linux kernel version. Yi 2012/9/19 Jan Kiszka jan.kis...@siemens.com On 2012-09-19 16:43, Abel Gordon wrote: It's imperfect as you need to dedicate a core to pure guest-mode load and cannot run userspace on that core (cannot walk through userspace-based device models e.g.). That's not correct. For the evaluation, we dedicated a core for each guest to maximize the performance but this is not a requirement. You can over-commit cores and share them across multiple VMs. In this case, you will be sharing the cycles among all the VMs and the performance per VM will be degraded. In addition, if you share a core, an interrupt for a guest may be raised while the corresponding VCPU is not running. Depending on the workload, most of the interrupts will be generated while the VCPU runs or not. and cannot run userspace on that core (cannot walk through userspace-based device models e.g.). That's not correct. We can run any thread (including user-space threads) on the same core we run the VMs (VCPU threads). In fact, we did that for the ELI evaluation: we shared a single core to run the VCPU thread and ALL the host threads (including qemu I/O thread). And it requires that magic bar to map the shadow IDT into the guest (hmm, I think Hitachi avoided this). Hitachi uses a different technique which seems to have the 2 disadvantages you previously mentioned while ELI doesn't have these disadvantages. It's invasive as it has to change Linux to maintain those isolated slave CPUs. That is, of course, based on code that was published by Hitachi. Yours may differ but will still have to solve the same problems. ELI does not require isolated/slave CPUs. OK. Show patches. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Re: [Qemu-devel] Will the ELI incorporated in theKVM?
On 2012-09-18 14:50, GaoYi wrote: Hi Jan, I have followed a previous thread about ELI proposed by Abel Gordon, http://www.spinics.net/lists/kvm/msg73907.html. I wonder whether this mechanism will be incorporated in KVM someday. Likely not. Both Intel and AMD will soon ship hardware that obsoletes this invasive and imperfect software solution, see also [1]. Jan [1] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/97715 -- Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
