Re: performance trouble
On 27.03.2012 19:06, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 06:16:11 PM Peter Lieven wrote: On 27.03.2012 18:12, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 05:58:01 PM Peter Lieven wrote: On 27.03.2012 17:44, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote: On 27.03.2012 14:29, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote: On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-ne t- 1v cpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/f f5 42 633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? Yes. first feedback: performance seems to be amazing. i cannot confirm that it breaks hv_spinlocks, hv_vapic and hv_relaxed. why did you assume this? I didn't mean that hv_refcnt will break any other hyper-v features. I just want to say that turning hv_refcnt on (as any other hv_ option) will crash Win8 on boot-up. yes, i got it meanwhile ;-) let me know what you think should be done to further test the refcnt implementation. i would suggest to return at least 0x if msr 0x4021 is read. IIRC Win7(W2k8R2) only reads this MSR. Win8 reads and writes. you mean win7 only writes, don't
Re: performance trouble
On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? peter -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tue, Mar 27, 2012 at 11:23:33AM +0200, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. Upstream SeaBIOS handles HPET properly now. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? Gleb mentioned that it properly handled in upstream, otherwise just comment the entire HPET section in acpi-dsdt.dsl file. the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? Yes, but don't forget about kvm patch as well. peter -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 27.03.2012 12:40, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? Gleb mentioned that it properly handled in upstream, otherwise just comment the entire HPET section in acpi-dsdt.dsl file. i have upstream bios installed. so -no-hpet should disable hpet completely. can you give a hint, what bits 1 and 9 must be set to on in leaf 0x4003 means? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? Yes, but don't forget about kvm patch as well. ok, i will try my best. would you consider your patch a quick hack or do you think it would be worth to be uploaded to the upstream repository? peter peter -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
On 27.03.2012 12:40, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
On 27.03.2012 11:23, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
On 26.03.2012 20:36, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
On 22.03.2012 10:38, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
On 22.03.2012 09:48, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
On 21.03.2012 12:10, David Cure wrote:
hello,
Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
ecrivait :
Try to addfeature policy='disable' name='hypervisor'/
to cpu definition in XML and check command line.
ok I try this but I can't usecpu model to map
the
host cpu
(my libvirt is 0.9.8) so I use :
cpu match='exact'
modelOpteron_G3/model
feature policy='disable' name='hypervisor'/
/cpu
(the physical server use Opteron CPU).
The log is here :
http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-
cp u.tx t.gz
And now with only 1 vcpu, the response time is 8.5s,
great
improvment. We keep this configuration for production : we
check the response time when some other users are
connected.
please keep in mind, that setting -hypervisor, disabling
hpet and only one vcpu
makes windows use tsc as clocksource. you have to make
sure, that your vm is not switching between physical
sockets on your system and that you have constant_tsc
feature to have a stable tsc between the cores in the same
socket. its also likely that the vm will crash when live
migrated.
All true. I asked to try -hypervisor only to verify where we
loose performance. Since you get good result with it
frequent access to PM timer is probably the reason. I do
not recommend using -hypervisor for production!
@gleb: do you know whats the state of in-kernel hyper-v
timers?
Vadim is working on it. I'll let him answer.
It would be nice to have synthetic timers supported. But, at
the moment, I'm only researching this feature.
So it will take months at least?
I would say weeks.
Is there a way, we could contribute and help you with this?
Hi Peter,
You are welcome to add an appropriate handler.
I think Vadim refers to this HV MSR
http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%
28 v=vs .85 %29.aspx
This one is pretty simple to support. Please see attachments for
more details. I was thinking about synthetic timers
http://msdn.microsoft.com/en-
us/library/windows/hardware/ff542758(v=vs.85).aspx
is this what microsoft qpc uses as clocksource in hyper-v?
Yes, it should be enough for Win7 / W2K8R2.
To clarify the thing that microsoft qpc uses is what is implemented by
the patch Vadim attached to his previous email. But I believe that
additional qemu patch is needed for Windows to actually use it.
You are right.
bits 1 and 9 must be set to on in leaf 0x4003 and HPET
should be completely removed from ACPI.
could you advise how to do this and/or make a patch?
Gleb mentioned that it properly handled in upstream,
otherwise just comment the entire HPET section in
acpi-dsdt.dsl file.
i have upstream bios installed. so -no-hpet should disable hpet completely.
can you give a hint, what
bits 1 and 9 must be set to on in leaf 0x4003 means?
I mean the following code:
+if (hyperv_ref_counter_enabled()) {
+c-eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+c-eax |= 0x200;
+}
Please see attached file for more information.
the stuff you send yesterday is for qemu, right? would
it be possible to use it in qemu-kvm also?
Yes, but don't forget about kvm patch as well.
ok, i will try my best. would you consider your patch a quick hack
or do you think it would be worth to be uploaded to the upstream
repository?
It was just a brief attempt from my side, mostly inspirited by our with Gleb
conversation, to see what it worth to turn this option on.
It is not fully tested. It will crash Win8 (as well as the rest of the
currently introduced hyper-v features).
I wouldn't commit this code without comprehensive testing.
Vadim.
peter
peter
--
Gleb.
diff --git a/target-i386/cpuid.c b/target-i386/cpuid.c
index c2edb64..5c85492 100644
--- a/target-i386/cpuid.c
+++
Re: performance trouble
On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? peter -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu modelto map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? peter -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote: On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu modelto map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? Yes. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 27.03.2012 14:29, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote: On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? Yes. ok, will try it. can you give me a short advice if the patch has to applied to qemu-kvm or qemu latest from git? peter -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 27.03.2012 14:29, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote: On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? Yes. first feedback: performance seems to be amazing. i cannot confirm that it breaks hv_spinlocks, hv_vapic and hv_relaxed. why did you assume this? no more pmtimer reads. i can now almost fully utililizy a 1GBit interface with a file transfer while there was not one cpu core fully utilized as observed with pmtimer. some live migration tests revealed that it did not crash even under load. @vadim: i think we need a proper patch for the others to test this ;-) what i observed: is it right, that HV_X64_MSR_TIME_REF_COUNT is missing in msrs_to_save[] in x86/x86.c of the kernel module? thanks for you help, peter -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 27.03.2012 13:43, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
On 27.03.2012 12:40, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
On 27.03.2012 11:23, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
On 26.03.2012 20:36, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
On 22.03.2012 10:38, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
On 22.03.2012 09:48, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
On 21.03.2012 12:10, David Cure wrote:
hello,
Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
ecrivait :
Try to addfeature policy='disable' name='hypervisor'/
to cpu definition in XML and check command line.
ok I try this but I can't usecpu modelto map
the
host cpu
(my libvirt is 0.9.8) so I use :
cpu match='exact'
modelOpteron_G3/model
feature policy='disable' name='hypervisor'/
/cpu
(the physical server use Opteron CPU).
The log is here :
http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-
cp u.tx t.gz
And now with only 1 vcpu, the response time is 8.5s,
great
improvment. We keep this configuration for production : we
check the response time when some other users are
connected.
please keep in mind, that setting -hypervisor, disabling
hpet and only one vcpu
makes windows use tsc as clocksource. you have to make
sure, that your vm is not switching between physical
sockets on your system and that you have constant_tsc
feature to have a stable tsc between the cores in the same
socket. its also likely that the vm will crash when live
migrated.
All true. I asked to try -hypervisor only to verify where we
loose performance. Since you get good result with it
frequent access to PM timer is probably the reason. I do
not recommend using -hypervisor for production!
@gleb: do you know whats the state of in-kernel hyper-v
timers?
Vadim is working on it. I'll let him answer.
It would be nice to have synthetic timers supported. But, at
the moment, I'm only researching this feature.
So it will take months at least?
I would say weeks.
Is there a way, we could contribute and help you with this?
Hi Peter,
You are welcome to add an appropriate handler.
I think Vadim refers to this HV MSR
http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%
28 v=vs .85 %29.aspx
This one is pretty simple to support. Please see attachments for
more details. I was thinking about synthetic timers
http://msdn.microsoft.com/en-
us/library/windows/hardware/ff542758(v=vs.85).aspx
is this what microsoft qpc uses as clocksource in hyper-v?
Yes, it should be enough for Win7 / W2K8R2.
To clarify the thing that microsoft qpc uses is what is implemented by
the patch Vadim attached to his previous email. But I believe that
additional qemu patch is needed for Windows to actually use it.
You are right.
bits 1 and 9 must be set to on in leaf 0x4003 and HPET
should be completely removed from ACPI.
could you advise how to do this and/or make a patch?
Gleb mentioned that it properly handled in upstream,
otherwise just comment the entire HPET section in
acpi-dsdt.dsl file.
i have upstream bios installed. so -no-hpet should disable hpet completely.
can you give a hint, what
bits 1 and 9 must be set to on in leaf 0x4003 means?
I mean the following code:
+if (hyperv_ref_counter_enabled()) {
+c-eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+c-eax |= 0x200;
+}
Please see attached file for more information.
the stuff you send yesterday is for qemu, right? would
it be possible to use it in qemu-kvm also?
Yes, but don't forget about kvm patch as well.
ok, i will try my best. would you consider your patch a quick hack
or do you think it would be worth to be uploaded to the upstream
repository?
It was just a brief attempt from my side, mostly inspirited by our with Gleb
conversation, to see what it worth to turn this option on.
It is not fully tested. It will crash Win8 (as well as the rest of the
currently introduced hyper-v features).
i can confirm that windows 8 installer does not start and resets the
vm continously. it tries to access hv msr 0x4021
http://msdn.microsoft.com/en-us/library/windows/hardware/ff542648%28v=vs.85%29.aspx
it is possible to tell the guest that the host is not iTSC (how they
call it) capable. i will try to hack
a patch for this.
peter
I wouldn't commit this code without
Re: performance trouble
On Tuesday, March 27, 2012 04:44:51 PM Peter Lieven wrote:
On 27.03.2012 13:43, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
On 27.03.2012 12:40, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
On 27.03.2012 11:23, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
On 26.03.2012 20:36, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
On 22.03.2012 10:38, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
On 22.03.2012 09:48, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven
wrote:
On 21.03.2012 12:10, David Cure wrote:
hello,
Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
ecrivait :
Try to addfeature policy='disable' name='hypervisor'/
to cpu definition in XML and check command line.
ok I try this but I can't usecpu modelto map
the
host cpu
(my libvirt is 0.9.8) so I use :
cpu match='exact'
modelOpteron_G3/model
feature policy='disable' name='hypervisor'/
/cpu
(the physical server use Opteron CPU).
The log is here :
http://www.roullier.net/Report/report-3.2-vhost-net-1vcp
u- cp u.tx t.gz
And now with only 1 vcpu, the response time is 8.5s,
great
improvment. We keep this configuration for production :
we check the response time when some other users are
connected.
please keep in mind, that setting -hypervisor, disabling
hpet and only one vcpu
makes windows use tsc as clocksource. you have to make
sure, that your vm is not switching between physical
sockets on your system and that you have constant_tsc
feature to have a stable tsc between the cores in the
same socket. its also likely that the vm will crash when
live migrated.
All true. I asked to try -hypervisor only to verify where
we loose performance. Since you get good result with it
frequent access to PM timer is probably the reason. I do
not recommend using -hypervisor for production!
@gleb: do you know whats the state of in-kernel hyper-v
timers?
Vadim is working on it. I'll let him answer.
It would be nice to have synthetic timers supported. But,
at the moment, I'm only researching this feature.
So it will take months at least?
I would say weeks.
Is there a way, we could contribute and help you with this?
Hi Peter,
You are welcome to add an appropriate handler.
I think Vadim refers to this HV MSR
http://msdn.microsoft.com/en-us/library/windows/hardware/ff54263
3% 28 v=vs .85 %29.aspx
This one is pretty simple to support. Please see attachments for
more details. I was thinking about synthetic timers
http://msdn.microsoft.com/en-
us/library/windows/hardware/ff542758(v=vs.85).aspx
is this what microsoft qpc uses as clocksource in hyper-v?
Yes, it should be enough for Win7 / W2K8R2.
To clarify the thing that microsoft qpc uses is what is implemented
by the patch Vadim attached to his previous email. But I believe
that additional qemu patch is needed for Windows to actually use
it.
You are right.
bits 1 and 9 must be set to on in leaf 0x4003 and HPET
should be completely removed from ACPI.
could you advise how to do this and/or make a patch?
Gleb mentioned that it properly handled in upstream,
otherwise just comment the entire HPET section in
acpi-dsdt.dsl file.
i have upstream bios installed. so -no-hpet should disable hpet
completely. can you give a hint, what
bits 1 and 9 must be set to on in leaf 0x4003 means?
I mean the following code:
+if (hyperv_ref_counter_enabled()) {
+c-eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+c-eax |= 0x200;
+}
Please see attached file for more information.
the stuff you send yesterday is for qemu, right? would
it be possible to use it in qemu-kvm also?
Yes, but don't forget about kvm patch as well.
ok, i will try my best. would you consider your patch a quick hack
or do you think it would be worth to be uploaded to the upstream
repository?
It was just a brief attempt from my side, mostly inspirited by our with
Gleb conversation, to see what it worth to turn this option on.
It is not fully tested. It will crash Win8 (as well as the rest of the
currently introduced hyper-v features).
i can confirm that windows 8 installer does not start and resets the
vm continously. it tries
Re: performance trouble
On 27.03.2012 17:37, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 04:44:51 PM Peter Lieven wrote:
On 27.03.2012 13:43, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
On 27.03.2012 12:40, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
On 27.03.2012 11:23, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
On 26.03.2012 20:36, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
On 22.03.2012 10:38, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
On 22.03.2012 09:48, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven
wrote:
On 21.03.2012 12:10, David Cure wrote:
hello,
Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
ecrivait :
Try to addfeature policy='disable' name='hypervisor'/
to cpu definition in XML and check command line.
ok I try this but I can't usecpu model to map
the
host cpu
(my libvirt is 0.9.8) so I use :
cpu match='exact'
modelOpteron_G3/model
feature policy='disable' name='hypervisor'/
/cpu
(the physical server use Opteron CPU).
The log is here :
http://www.roullier.net/Report/report-3.2-vhost-net-1vcp
u- cp u.tx t.gz
And now with only 1 vcpu, the response time is 8.5s,
great
improvment. We keep this configuration for production :
we check the response time when some other users are
connected.
please keep in mind, that setting -hypervisor, disabling
hpet and only one vcpu
makes windows use tsc as clocksource. you have to make
sure, that your vm is not switching between physical
sockets on your system and that you have constant_tsc
feature to have a stable tsc between the cores in the
same socket. its also likely that the vm will crash when
live migrated.
All true. I asked to try -hypervisor only to verify where
we loose performance. Since you get good result with it
frequent access to PM timer is probably the reason. I do
not recommend using -hypervisor for production!
@gleb: do you know whats the state of in-kernel hyper-v
timers?
Vadim is working on it. I'll let him answer.
It would be nice to have synthetic timers supported. But,
at the moment, I'm only researching this feature.
So it will take months at least?
I would say weeks.
Is there a way, we could contribute and help you with this?
Hi Peter,
You are welcome to add an appropriate handler.
I think Vadim refers to this HV MSR
http://msdn.microsoft.com/en-us/library/windows/hardware/ff54263
3% 28 v=vs .85 %29.aspx
This one is pretty simple to support. Please see attachments for
more details. I was thinking about synthetic timers
http://msdn.microsoft.com/en-
us/library/windows/hardware/ff542758(v=vs.85).aspx
is this what microsoft qpc uses as clocksource in hyper-v?
Yes, it should be enough for Win7 / W2K8R2.
To clarify the thing that microsoft qpc uses is what is implemented
by the patch Vadim attached to his previous email. But I believe
that additional qemu patch is needed for Windows to actually use
it.
You are right.
bits 1 and 9 must be set to on in leaf 0x4003 and HPET
should be completely removed from ACPI.
could you advise how to do this and/or make a patch?
Gleb mentioned that it properly handled in upstream,
otherwise just comment the entire HPET section in
acpi-dsdt.dsl file.
i have upstream bios installed. so -no-hpet should disable hpet
completely. can you give a hint, what
bits 1 and 9 must be set to on in leaf 0x4003 means?
I mean the following code:
+if (hyperv_ref_counter_enabled()) {
+c-eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+c-eax |= 0x200;
+}
Please see attached file for more information.
the stuff you send yesterday is for qemu, right? would
it be possible to use it in qemu-kvm also?
Yes, but don't forget about kvm patch as well.
ok, i will try my best. would you consider your patch a quick hack
or do you think it would be worth to be uploaded to the upstream
repository?
It was just a brief attempt from my side, mostly inspirited by our with
Gleb conversation, to see what it worth to turn this option on.
It is not fully tested. It will crash Win8 (as well as the rest of the
currently introduced hyper-v features).
i can confirm that windows 8 installer does not start and resets the
vm continously. it tries to access hv msr 0x4021
Win8 needs more comprehensive Hyper-V support.
yes it seems. i read your comment wrong. i was believing the
hv_refcnt breaks the other hv_features
Re: performance trouble
On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote: On 27.03.2012 14:29, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote: On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1v cpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542 633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? Yes. first feedback: performance seems to be amazing. i cannot confirm that it breaks hv_spinlocks, hv_vapic and hv_relaxed. why did you assume this? I didn't mean that hv_refcnt will break any other hyper-v features. I just want to say that turning hv_refcnt on (as any other hv_ option) will crash Win8 on boot-up. Cheers, Vadim. no more pmtimer reads. i can now almost fully utililizy a 1GBit interface with a file transfer while there was not one cpu core fully utilized as observed with pmtimer. some live migration tests revealed that it did not crash even under load. @vadim: i think we need a proper patch for the others to test this ;-) what i observed: is it right, that
Re: performance trouble
On Tuesday, March 27, 2012 05:39:10 PM Peter Lieven wrote:
On 27.03.2012 17:37, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 04:44:51 PM Peter Lieven wrote:
On 27.03.2012 13:43, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
On 27.03.2012 12:40, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
On 27.03.2012 11:23, Vadim Rozenfeld wrote:
On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
On 26.03.2012 20:36, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld
wrote:
On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
On 22.03.2012 10:38, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
On 22.03.2012 09:48, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov
wrote:
On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven
wrote:
On 21.03.2012 12:10, David Cure wrote:
hello,
Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
ecrivait :
Try to addfeature policy='disable'
name='hypervisor'/ to cpu definition in XML and
check command line.
ok I try this but I can't usecpu model to
map
the
host cpu
(my libvirt is 0.9.8) so I use :
cpu match='exact'
modelOpteron_G3/model
feature policy='disable'
name='hypervisor'/
/cpu
(the physical server use Opteron CPU).
The log is here :
http://www.roullier.net/Report/report-3.2-vhost-net-1v
cp u- cp u.tx t.gz
And now with only 1 vcpu, the response time is 8.5s,
great
improvment. We keep this configuration for production
: we check the response time when some other users
are connected.
please keep in mind, that setting -hypervisor,
disabling hpet and only one vcpu
makes windows use tsc as clocksource. you have to make
sure, that your vm is not switching between physical
sockets on your system and that you have constant_tsc
feature to have a stable tsc between the cores in the
same socket. its also likely that the vm will crash
when live migrated.
All true. I asked to try -hypervisor only to verify
where we loose performance. Since you get good result
with it frequent access to PM timer is probably the
reason. I do not recommend using -hypervisor for
production!
@gleb: do you know whats the state of in-kernel hyper-v
timers?
Vadim is working on it. I'll let him answer.
It would be nice to have synthetic timers supported. But,
at the moment, I'm only researching this feature.
So it will take months at least?
I would say weeks.
Is there a way, we could contribute and help you with this?
Hi Peter,
You are welcome to add an appropriate handler.
I think Vadim refers to this HV MSR
http://msdn.microsoft.com/en-us/library/windows/hardware/ff542
63 3% 28 v=vs .85 %29.aspx
This one is pretty simple to support. Please see attachments
for more details. I was thinking about synthetic timers
http://msdn.microsoft.com/en-
us/library/windows/hardware/ff542758(v=vs.85).aspx
is this what microsoft qpc uses as clocksource in hyper-v?
Yes, it should be enough for Win7 / W2K8R2.
To clarify the thing that microsoft qpc uses is what is
implemented by the patch Vadim attached to his previous email.
But I believe that additional qemu patch is needed for Windows to
actually use it.
You are right.
bits 1 and 9 must be set to on in leaf 0x4003 and HPET
should be completely removed from ACPI.
could you advise how to do this and/or make a patch?
Gleb mentioned that it properly handled in upstream,
otherwise just comment the entire HPET section in
acpi-dsdt.dsl file.
i have upstream bios installed. so -no-hpet should disable hpet
completely. can you give a hint, what
bits 1 and 9 must be set to on in leaf 0x4003 means?
I mean the following code:
+if (hyperv_ref_counter_enabled()) {
+c-eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+c-eax |= 0x200;
+}
Please see attached file for more information.
the stuff you send yesterday is for qemu, right? would
it be possible to use it in qemu-kvm also?
Yes, but don't forget about kvm patch as well.
ok, i will try my best. would you consider your patch a quick hack
or do you think it would be worth to be uploaded to the upstream
repository?
It was just a brief attempt from my side, mostly inspirited by our with
Gleb conversation, to see what it worth to turn this option on.
It is not fully tested. It will crash Win8 (as well as the rest of the
currently
Re: performance trouble
On 27.03.2012 17:44, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote: On 27.03.2012 14:29, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote: On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1v cpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542 633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? Yes. first feedback: performance seems to be amazing. i cannot confirm that it breaks hv_spinlocks, hv_vapic and hv_relaxed. why did you assume this? I didn't mean that hv_refcnt will break any other hyper-v features. I just want to say that turning hv_refcnt on (as any other hv_ option) will crash Win8 on boot-up. yes, i got it meanwhile ;-) let me know what you think should be done to further test the refcnt implementation. i would suggest to return at least 0x if msr 0x4021 is read. peter Cheers, Vadim. no more pmtimer reads. i can now almost fully utililizy a 1GBit interface with a file transfer while there was not one cpu core fully utilized as observed with pmtimer. some live migration tests revealed that it did not crash even under load. @vadim: i think we need a proper patch for the others to test
Re: performance trouble
On Tuesday, March 27, 2012 05:58:01 PM Peter Lieven wrote: On 27.03.2012 17:44, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote: On 27.03.2012 14:29, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote: On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net- 1v cpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff5 42 633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? Yes. first feedback: performance seems to be amazing. i cannot confirm that it breaks hv_spinlocks, hv_vapic and hv_relaxed. why did you assume this? I didn't mean that hv_refcnt will break any other hyper-v features. I just want to say that turning hv_refcnt on (as any other hv_ option) will crash Win8 on boot-up. yes, i got it meanwhile ;-) let me know what you think should be done to further test the refcnt implementation. i would suggest to return at least
Re: performance trouble
On 27.03.2012 18:12, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 05:58:01 PM Peter Lieven wrote: On 27.03.2012 17:44, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote: On 27.03.2012 14:29, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote: On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net- 1v cpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff5 42 633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? Yes. first feedback: performance seems to be amazing. i cannot confirm that it breaks hv_spinlocks, hv_vapic and hv_relaxed. why did you assume this? I didn't mean that hv_refcnt will break any other hyper-v features. I just want to say that turning hv_refcnt on (as any other hv_ option) will crash Win8 on boot-up. yes, i got it meanwhile ;-) let me know what you think should be done to further test the refcnt implementation. i would suggest to return at least 0x if msr 0x4021 is read. IIRC Win7(W2k8R2) only reads this MSR. Win8 reads and writes. you mean win7 only writes, don't you? at least you put a break in set_msr_hyperv for this msr. i just thought that it would be ok to return the
Re: performance trouble
On Tuesday, March 27, 2012 06:16:11 PM Peter Lieven wrote: On 27.03.2012 18:12, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 05:58:01 PM Peter Lieven wrote: On 27.03.2012 17:44, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote: On 27.03.2012 14:29, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote: On 27.03.2012 14:26, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote: On 27.03.2012 12:00, Gleb Natapov wrote: On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote: On 27.03.2012 11:23, Vadim Rozenfeld wrote: On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-ne t- 1v cpu-cp u.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/f f5 42 633%28 v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. To clarify the thing that microsoft qpc uses is what is implemented by the patch Vadim attached to his previous email. But I believe that additional qemu patch is needed for Windows to actually use it. You are right. bits 1 and 9 must be set to on in leaf 0x4003 and HPET should be completely removed from ACPI. could you advise how to do this and/or make a patch? the stuff you send yesterday is for qemu, right? would it be possible to use it in qemu-kvm also? No, they are for kernel. i meant the qemu.diff file. Yes, I missed the second attachment. if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu? Looks like it. ok, so it would be interesting if it helps to avoid the pmtimer reads we observed earlier. right? Yes. first feedback: performance seems to be amazing. i cannot confirm that it breaks hv_spinlocks, hv_vapic and hv_relaxed. why did you assume this? I didn't mean that hv_refcnt will break any other hyper-v features. I just want to say that turning hv_refcnt on (as any other hv_ option) will crash Win8 on boot-up. yes, i got it meanwhile ;-) let me know what you think
Re: performance trouble
On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/to cpu definition in XML and check command line. ok I try this but I can't usecpu modelto map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Peter -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/to cpu definition in XML and check command line. ok I try this but I can't usecpu modelto map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs.85%29.aspx -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
On 22.03.2012 10:38, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
On 22.03.2012 09:48, Vadim Rozenfeld wrote:
On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
On 21.03.2012 12:10, David Cure wrote:
hello,
Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
Try to addfeature policy='disable' name='hypervisor'/to
cpu definition in XML and check command line.
ok I try this but I can't usecpu modelto map the host cpu
(my libvirt is 0.9.8) so I use :
cpu match='exact'
modelOpteron_G3/model
feature policy='disable' name='hypervisor'/
/cpu
(the physical server use Opteron CPU).
The log is here :
http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.tx
t.gz
And now with only 1 vcpu, the response time is 8.5s, great
improvment. We keep this configuration for production : we check
the response time when some other users are connected.
please keep in mind, that setting -hypervisor, disabling hpet and
only one vcpu
makes windows use tsc as clocksource. you have to make sure, that
your vm is not switching between physical sockets on your system
and that you have constant_tsc feature to have a stable tsc
between the cores in the same socket. its also likely that the
vm will crash when live migrated.
All true. I asked to try -hypervisor only to verify where we loose
performance. Since you get good result with it frequent access to
PM timer is probably the reason. I do not recommend using
-hypervisor for production!
@gleb: do you know whats the state of in-kernel hyper-v timers?
Vadim is working on it. I'll let him answer.
It would be nice to have synthetic timers supported. But, at the
moment, I'm only researching this feature.
So it will take months at least?
I would say weeks.
Is there a way, we could contribute and help you with this?
Hi Peter,
You are welcome to add an appropriate handler.
I think Vadim refers to this HV MSR
http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs.85
%29.aspx
This one is pretty simple to support. Please see attachments for more details.
I was thinking about synthetic timers http://msdn.microsoft.com/en-
us/library/windows/hardware/ff542758(v=vs.85).aspx
--
Gleb.
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 74c9edf..fafc8ff 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -535,6 +535,7 @@ struct kvm_arch {
/* fields used by HYPER-V emulation */
u64 hv_guest_os_id;
u64 hv_hypercall;
+u64 hv_ref_count;
atomic_t reader_counter;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index c9d99e5..4562581 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1393,6 +1393,7 @@ static bool kvm_hv_msr_partition_wide(u32 msr)
switch (msr) {
case HV_X64_MSR_GUEST_OS_ID:
case HV_X64_MSR_HYPERCALL:
+case HV_X64_MSR_TIME_REF_COUNT:
r = true;
break;
}
@@ -1432,6 +1433,7 @@ static int set_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data)
if (__copy_to_user((void __user *)addr, instructions, 4))
return 1;
kvm-arch.hv_hypercall = data;
+kvm-arch.hv_ref_count = get_kernel_ns();
break;
}
default:
@@ -1467,6 +1469,9 @@ static int set_msr_hyperv(struct kvm_vcpu *vcpu, u32 msr, u64 data)
return kvm_hv_vapic_msr_write(vcpu, APIC_ICR, data);
case HV_X64_MSR_TPR:
return kvm_hv_vapic_msr_write(vcpu, APIC_TASKPRI, data);
+case 0x4021:
+break;
default:
pr_unimpl(vcpu, HYPER-V unimplemented wrmsr: 0x%x
data 0x%llx\n, msr, data);
@@ -1842,6 +1847,10 @@ static int get_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
case HV_X64_MSR_HYPERCALL:
data = kvm-arch.hv_hypercall;
break;
+case HV_X64_MSR_TIME_REF_COUNT:
+data = get_kernel_ns() - kvm-arch.hv_ref_count;
+break;
default:
pr_unimpl(vcpu, Hyper-V unhandled rdmsr: 0x%x\n, msr);
return 1;
@@ -1873,6 +1882,9 @@ static int get_msr_hyperv(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
case HV_X64_MSR_APIC_ASSIST_PAGE:
data = vcpu-arch.hv_vapic;
break;
default:
pr_unimpl(vcpu, Hyper-V unhandled rdmsr: 0x%x\n, msr);
return 1;
diff --git a/target-i386/cpuid.c b/target-i386/cpuid.c
index c2edb64..5c85492 100644
--- a/target-i386/cpuid.c
+++ b/target-i386/cpuid.c
@@ -778,6 +778,9 @@ static int
Re: performance trouble
On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs.85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? i will check tomorrow. thanks vadim -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/to cpu definition in XML and check command line. ok I try this but I can't usecpu modelto map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. Best regards, Vadim. Peter -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote: On 26.03.2012 20:36, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote: On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote: On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote: On 22.03.2012 10:38, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.tx t.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. Is there a way, we could contribute and help you with this? Hi Peter, You are welcome to add an appropriate handler. I think Vadim refers to this HV MSR http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs .85 %29.aspx This one is pretty simple to support. Please see attachments for more details. I was thinking about synthetic timers http://msdn.microsoft.com/en- us/library/windows/hardware/ff542758(v=vs.85).aspx is this what microsoft qpc uses as clocksource in hyper-v? Yes, it should be enough for Win7 / W2K8R2. Cheers, Vadim. i will check tomorrow. thanks vadim -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. peter David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 22.03.2012 08:53, Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. @avi, gleb: another option would be to revisit the old in-kernel pm-timer implementation and check if its feasible to use this as an alternative. it would also help non hyper-v aware systems (i think bsd and old windows like xp). i rebased this old implementation and can confirm that it does also solve the performance slow down. peter peter David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Thu, Mar 22, 2012 at 09:53:45AM +0200, Gleb Natapov ecrivait : All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! so if I leave cpu as previous (not defined) and only disable hpet and use 1 vcpu, it's ok for production ? Is there a workaround for this PM access ? David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
hello, Le Thu, Mar 22, 2012 at 08:57:08AM +0100, Peter Lieven ecrivait : @avi, gleb: another option would be to revisit the old in-kernel pm-timer implementation and check if its feasible to use this as an alternative. it would also help non hyper-v aware systems (i think bsd and old windows like xp). i rebased this old implementation and can confirm that it does also solve the performance slow down. Are some patches available for testing ? David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven ecrivait : please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. ok, yet I only have disable hpet and use 1vcpu. for the switching I need to pin the vcpu on 1 physical proc, right ? for constant_tsc, how can I check if I use it ? for live migration : what is the feature that cause trouble : -hypervisor, hpet, vcpu or all ? David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 22.03.2012 09:31, David Cure wrote: Le Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven ecrivait : please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. ok, yet I only have disable hpet and use 1vcpu. you have to use 1 vcpu on 32-bit windows. 64-bit seems to work with more than 1 vcpu. why all those limitations: windows avoids using tsc in a hypervisor which is a good decision. problem is it falls back to pm_timer or hpet. both of them are very expensive in emulation currently because kvm exits kernel mode and the userspace qemu-kvm handles this. i have done experiments where i saw ~20.000 userpace exits just for pmtimer reads. this made up ~30-40% of the whole processing power. every call to a QPC timer in windows causes a pm_timer/hpet read. especially each i/o request seems to cause a QPC timer read and which is odd as well a lazy fpu call (but this is a differnt issue) which also is very expensive to emulate (currently). for the switching I need to pin the vcpu on 1 physical proc, right ? you need 1 vcpu for 32-bit windows and disabling hypervisor to cheat windows and make it think it runs on real hardware. it then uses tsc. for constant_tsc, how can I check if I use it ? cat /proc/cpuinfo on the host. there should be a flag 'constant_tsc'. it might be that also rdtscp is necessary. for live migration : what is the feature that cause trouble : -hypervisor, hpet, vcpu or all ? using tsc as clocksource is the problem not the features themselves. peter David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. peter David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 22.03.2012 09:33, David Cure wrote: Le Thu, Mar 22, 2012 at 09:53:45AM +0200, Gleb Natapov ecrivait : All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! so if I leave cpu as previous (not defined) and only disable hpet and use 1 vcpu, it's ok for production ? this is ok, but windows will use pm timer so you will have bad performance. Is there a workaround for this PM access ? there exists old patches from 2010 for in-kernel pmtimer. they work, but only partly. problem here is windows enables the pmtimer overflow interrupt which this patch did not address (amongst other things). i simply ignored it and windows ran nevertheless. but i would not do this in production because i do not know which side effects it might have. there are to possible solutions: a) a real in-kernel pmtimer implementation (which does also help other systems not only windows) b) hyper-v support in-kernel at least partly (for the timing stuff). this is being worked on by Vadim. Peter David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? What do the others think, would it be feasible to make a proper in-kernel pmtimer solution in the meantime. I think Windows guest performance is very important for the success of KVM. Peter peter David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote: On 22.03.2012 09:48, Vadim Rozenfeld wrote: On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote: On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote: On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. All true. I asked to try -hypervisor only to verify where we loose performance. Since you get good result with it frequent access to PM timer is probably the reason. I do not recommend using -hypervisor for production! @gleb: do you know whats the state of in-kernel hyper-v timers? Vadim is working on it. I'll let him answer. It would be nice to have synthetic timers supported. But, at the moment, I'm only researching this feature. So it will take months at least? I would say weeks. What do the others think, would it be feasible to make a proper in-kernel pmtimer solution in the meantime. I think Windows guest performance is very important for the success of KVM. Peter peter David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to add feature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't use cpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 21.03.2012 12:10, David Cure wrote: hello, Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait : Try to addfeature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. ok I try this but I can't usecpu model to map the host cpu (my libvirt is 0.9.8) so I use : cpu match='exact' modelOpteron_G3/model feature policy='disable' name='hypervisor'/ /cpu (the physical server use Opteron CPU). The log is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz And now with only 1 vcpu, the response time is 8.5s, great improvment. We keep this configuration for production : we check the response time when some other users are connected. please keep in mind, that setting -hypervisor, disabling hpet and only one vcpu makes windows use tsc as clocksource. you have to make sure, that your vm is not switching between physical sockets on your system and that you have constant_tsc feature to have a stable tsc between the cores in the same socket. its also likely that the vm will crash when live migrated. @gleb: do you know whats the state of in-kernel hyper-v timers? peter David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
hello, Le Mon, Mar 19, 2012 at 12:51:34PM +0200, Gleb Natapov ecrivait : Can you run the same test on Linux guest and on Windows vm with 1 cpu? I see a lot of IPIs between vcpus. I run the same slowly fonction (to remember the guest is Windows2008R2 64bits). The report is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu.txt.gz and the response time is ~13s. thanks for tracking this trouble, David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tue, Mar 20, 2012 at 10:32:20AM +0100, David Cure wrote: hello, Le Mon, Mar 19, 2012 at 12:51:34PM +0200, Gleb Natapov ecrivait : Can you run the same test on Linux guest and on Windows vm with 1 cpu? I see a lot of IPIs between vcpus. I run the same slowly fonction (to remember the guest is Windows2008R2 64bits). The report is here : http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu.txt.gz and the response time is ~13s. This is without -no-hpet :) Also I hope you are not measuring time while tracing since it supposed to be slower. Can you please try to run with -cpu host,-hypervisor and -no-hpet and see if times are better. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Tue, Mar 20, 2012 at 11:45:41AM +0200, Gleb Natapov ecrivait : This is without -no-hpet :) Also I hope you are not measuring time while ha yes, sorry for the mistake. tracing since it supposed to be slower. Can you please try to run with -cpu host,-hypervisor and -no-hpet and see if times are better. I use libvirt to start the VM, do you know how to modify the xml file to have this option generated (I know for the -no-hpet, but not for other) ? If not, I capture the command and modify it to run directly. David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tue, Mar 20, 2012 at 12:18:39PM +0100, David Cure wrote: Le Tue, Mar 20, 2012 at 11:45:41AM +0200, Gleb Natapov ecrivait : This is without -no-hpet :) Also I hope you are not measuring time while ha yes, sorry for the mistake. tracing since it supposed to be slower. Can you please try to run with -cpu host,-hypervisor and -no-hpet and see if times are better. I use libvirt to start the VM, do you know how to modify the xml file to have this option generated (I know for the -no-hpet, but not for other) ? If not, I capture the command and modify it to run directly. I was pointed to here: http://berrange.com/posts/2010/02/15/guest-cpu-model-configuration-in-libvirt-with-qemukvm/ Try to add feature policy='disable' name='hypervisor'/ to cpu definition in XML and check command line. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Fri, Mar 16, 2012 at 11:13:31AM +0100, David Cure wrote: hello, sorry for the delay, Le Thu, Feb 23, 2012 at 10:38:07AM +0200, Gleb Natapov ecrivait : Ah, I guess the reason is that it records events only of IO thread. You need to trace all vcpu threads too. Not sure trace-cmd allows more then one -P option though. I manage to have the physical server with only one VM with the slowly function and take trace during the slowly function. I upload trace in http://www.roullier.net/Report/ with : o report.txt.3.1.gz : with kernel 3.1 o report.txt.3.2.gz : with kernel 3.2 o report.txt.vhost-net-3.1.gz : with kernel 3.1 and vhost-net o report.txt.vhost-net.3.2.gz : with kernel 3.2 and vhost-net With 3.2 + vhost-net we have 10.5s (to remember 8s with vmware esxi 4). Can you run the same test on Linux guest and on Windows vm with 1 cpu? I see a lot of IPIs between vcpus. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
hello, sorry for the delay, Le Thu, Feb 23, 2012 at 10:38:07AM +0200, Gleb Natapov ecrivait : Ah, I guess the reason is that it records events only of IO thread. You need to trace all vcpu threads too. Not sure trace-cmd allows more then one -P option though. I manage to have the physical server with only one VM with the slowly function and take trace during the slowly function. I upload trace in http://www.roullier.net/Report/ with : o report.txt.3.1.gz : with kernel 3.1 o report.txt.3.2.gz : with kernel 3.2 o report.txt.vhost-net-3.1.gz : with kernel 3.1 and vhost-net o report.txt.vhost-net.3.2.gz : with kernel 3.2 and vhost-net With 3.2 + vhost-net we have 10.5s (to remember 8s with vmware esxi 4). David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Wed, Feb 22, 2012 at 05:33:56PM +0100, David Cure wrote: Le Sun, Feb 19, 2012 at 11:13:15AM +0200, Gleb Natapov ecrivait : http://www.roullier.net/report-no-hpet.txt.gz How have you acquired this trace? It does not trace all kvm events only those that have set_irq in them. I run : trace-cmd record -b 2 -e kvm -P process_pid Ah, I guess the reason is that it records events only of IO thread. You need to trace all vcpu threads too. Not sure trace-cmd allows more then one -P option though. Nothing particularly strange here. ~1745 IRQs are injected into the guest each second. RTC clock is configured to 1kH and other devices contribute ~721 irqs (MSI mostly) per second more. Hardly unusual. ok, so no ideas to decrease the response time ? Without knowing the reason of the slowdown no. Probably you have the same problem as this other performance thread (frequent access to pm timer), but we need better trace to confirm that. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Sun, Feb 19, 2012 at 11:13:15AM +0200, Gleb Natapov ecrivait : http://www.roullier.net/report-no-hpet.txt.gz How have you acquired this trace? It does not trace all kvm events only those that have set_irq in them. I run : trace-cmd record -b 2 -e kvm -P process_pid Nothing particularly strange here. ~1745 IRQs are injected into the guest each second. RTC clock is configured to 1kH and other devices contribute ~721 irqs (MSI mostly) per second more. Hardly unusual. ok, so no ideas to decrease the response time ? David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
To complete and because I see another thread on performance trouble with w2008R2 Le Wed, Feb 22, 2012 at 05:33:56PM +0100, David Cure ecrivait : Nothing particularly strange here. ~1745 IRQs are injected into the guest each second. RTC clock is configured to 1kH and other devices contribute ~721 irqs (MSI mostly) per second more. Hardly unusual. ok, so no ideas to decrease the response time ? Before to see the response time problem, we see long network response and download : our app was on one samba server and we see slowly download when the user start the application. To solve, we copy the app directly on the RDS server (with synchro mechanism from samba for new version). And after we see the response time problem. David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Fri, Feb 17, 2012 at 03:01:07PM +0100, David Cure wrote: Le Fri, Feb 17, 2012 at 12:07:14PM +0200, Gleb Natapov ecrivait : Can you do the trace again with -no-hpet on the command line? Is there a way to only trace events on specific VM ? -P pid ok. I take a trace of the VM with 1 user : start trace just before to launch slowly function and stop trace just after the function finish. the report is available here http://www.roullier.net/report-no-hpet.txt.gz How have you acquired this trace? It does not trace all kvm events only those that have set_irq in them. I hope you can see something strange, Nothing particularly strange here. ~1745 IRQs are injected into the guest each second. RTC clock is configured to 1kH and other devices contribute ~721 irqs (MSI mostly) per second more. Hardly unusual. -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Thu, Feb 16, 2012 at 11:01:26AM +0200, Gleb Natapov ecrivait : Can you do the trace again with -no-hpet on the command line? Is there a way to only trace events on specific VM ? David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Fri, Feb 17, 2012 at 09:59:39AM +0100, David Cure wrote: Le Thu, Feb 16, 2012 at 11:01:26AM +0200, Gleb Natapov ecrivait : Can you do the trace again with -no-hpet on the command line? Is there a way to only trace events on specific VM ? -P pid -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Fri, Feb 17, 2012 at 12:07:14PM +0200, Gleb Natapov ecrivait : Can you do the trace again with -no-hpet on the command line? Is there a way to only trace events on specific VM ? -P pid ok. I take a trace of the VM with 1 user : start trace just before to launch slowly function and stop trace just after the function finish. the report is available here http://www.roullier.net/report-no-hpet.txt.gz I hope you can see something strange, David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Tue, Feb 14, 2012 at 08:48:56AM -0500, Vadim Rozenfeld ecrivait : +1 Try Microsoft Windows Performance Toolkit from Windows SDK http://www.microsoft.com/download/en/details.aspx?displaylang=enid=3138 It's really good. I run xperf -on DiagEasy, and as I'm not an windows-er the trace file is available here http://www.roullier.net/srv-rds1.zip I hope someone can see something strange ;) David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
hello, Le Tue, Feb 14, 2012 at 03:40:30PM +0200, Gleb Natapov ecrivait : Try to add -no-hpet to qemu command line and see if it helps. I add this line in my xml definition for libvirt : timer name='hpet' present='no'/ in the clock block. And I see the -no-hpet in the command line : /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 16384 -smp 2,sockets=2,cores=1,threads=1 -name rds1 -uuid 4f72a64e-de1a-b3da-adac-5dffdb9eb6aa -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rds1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -drive file=/dev/drbd1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/Iso/SW_DVD5_Windows_Svr_DC_EE_SE_Web_2008R2_64-bit_French_X15-59758.ISO,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=18,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:36:0c:a1:c4,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k fr -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 But unfortunatly, the result is the same : always 12s to runs this function (to remember 8s on ESXi 4). David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Thu, Feb 16, 2012 at 09:55:53AM +0100, David Cure wrote: hello, Le Tue, Feb 14, 2012 at 03:40:30PM +0200, Gleb Natapov ecrivait : Try to add -no-hpet to qemu command line and see if it helps. I add this line in my xml definition for libvirt : timer name='hpet' present='no'/ in the clock block. And I see the -no-hpet in the command line : /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 16384 -smp 2,sockets=2,cores=1,threads=1 -name rds1 -uuid 4f72a64e-de1a-b3da-adac-5dffdb9eb6aa -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rds1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -drive file=/dev/drbd1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/Iso/SW_DVD5_Windows_Svr_DC_EE_SE_Web_2008R2_64-bit_French_X15-59758.ISO,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=18,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:36:0c:a1:c4,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -k fr -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 But unfortunatly, the result is the same : always 12s to runs this function (to remember 8s on ESXi 4). Can you do the trace again with -no-hpet on the command line? -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Tue, Feb 14, 2012 at 03:32:16PM +0200, Avi Kivity ecrivait : It's reading the HPET like crazy. There are also tons of interrupts. Please use the windows performance tools to see which devices trigger these interrupts. The HPET issue will be fixed by the hyper-V enlightenments, but these will take some time to cook. as you can see, I try -no-hpet and doesn't solve the issue, so perhaps not related to it. Le Tue, Feb 14, 2012 at 08:48:56AM -0500, Vadim Rozenfeld ecrivait : [VR] +1 Try Microsoft Windows Performance Toolkit from Windows SDK http://www.microsoft.com/download/en/details.aspx?displaylang=enid=3138 It's really good. I download and install it. Let you know what we can see. Thanks, David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 02/10/2012 12:09 PM, David Cure wrote: hello, Le Sun, Feb 05, 2012 at 11:38:34AM +0200, Avi Kivity ecrivait : Please post a trace as documented in http://www.linux-kvm.org/page/Tracing. I made the trace : started just before the slow function launch and stoped just after. I start only one VM with 2 vcpus/16G RAM and only one user connected to the VM to launch the test. The trace file is too big to post here, I gzip it and the file is available here : http://www.roullier.net/report.txt.gz I hope you can find something strange. It's reading the HPET like crazy. There are also tons of interrupts. Please use the windows performance tools to see which devices trigger these interrupts. The HPET issue will be fixed by the hyper-V enlightenments, but these will take some time to cook. You can also try vhost-net to improve networking latency. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Tue, Feb 14, 2012 at 03:32:16PM +0200, Avi Kivity wrote: On 02/10/2012 12:09 PM, David Cure wrote: hello, Le Sun, Feb 05, 2012 at 11:38:34AM +0200, Avi Kivity ecrivait : Please post a trace as documented in http://www.linux-kvm.org/page/Tracing. I made the trace : started just before the slow function launch and stoped just after. I start only one VM with 2 vcpus/16G RAM and only one user connected to the VM to launch the test. The trace file is too big to post here, I gzip it and the file is available here : http://www.roullier.net/report.txt.gz I hope you can find something strange. It's reading the HPET like crazy. There are also tons of interrupts. Please use the windows performance tools to see which devices trigger these interrupts. The HPET issue will be fixed by the hyper-V enlightenments, but these will take some time to cook. Try to add -no-hpet to qemu command line and see if it helps. You can also try vhost-net to improve networking latency. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
- Original Message - From: Avi Kivity a...@redhat.com To: David Cure k...@cure.nom.fr Cc: kvm@vger.kernel.org, Vadim Rozenfeld vroze...@redhat.com Sent: Tuesday, February 14, 2012 3:32:16 PM Subject: Re: performance trouble On 02/10/2012 12:09 PM, David Cure wrote: hello, Le Sun, Feb 05, 2012 at 11:38:34AM +0200, Avi Kivity ecrivait : Please post a trace as documented in http://www.linux-kvm.org/page/Tracing. I made the trace : started just before the slow function launch and stoped just after. I start only one VM with 2 vcpus/16G RAM and only one user connected to the VM to launch the test. The trace file is too big to post here, I gzip it and the file is available here : http://www.roullier.net/report.txt.gz I hope you can find something strange. It's reading the HPET like crazy. There are also tons of interrupts. Please use the windows performance tools to see which devices trigger these interrupts. [VR] +1 Try Microsoft Windows Performance Toolkit from Windows SDK http://www.microsoft.com/download/en/details.aspx?displaylang=enid=3138 It's really good. The HPET issue will be fixed by the hyper-V enlightenments, but these will take some time to cook. You can also try vhost-net to improve networking latency. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
hello, Le Sun, Feb 05, 2012 at 11:38:34AM +0200, Avi Kivity ecrivait : Please post a trace as documented in http://www.linux-kvm.org/page/Tracing. I made the trace : started just before the slow function launch and stoped just after. I start only one VM with 2 vcpus/16G RAM and only one user connected to the VM to launch the test. The trace file is too big to post here, I gzip it and the file is available here : http://www.roullier.net/report.txt.gz I hope you can find something strange. David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Hello, Le Thu, Feb 02, 2012 at 11:41:48AM +0100, David Cure ecrivait : For kvm_stats, I mean it's better to have only one VM with one user for the test so I send this evening or tomorrow morning. I attach snapshot of the kvm_stat : the first one just before to run the function and the second one at the end. Here are the vmstat 1 take when the function is running : 4 0 0 32601320 21360 1987760049 1123 14577 21601 8 4 88 0 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 4 0 0 32601196 21360 19877600 0 141 16559 21315 12 3 84 0 2 0 0 32601212 21360 19877600 0 128 17377 24949 10 4 86 0 2 0 0 32601128 21360 19877600 0 120 13064 18986 13 3 85 0 4 0 0 32601216 21360 19877600 0 284 16052 21569 11 4 85 0 3 0 0 32601132 21360 19877600 0 192 15846 23323 10 3 87 0 3 0 0 32601492 21360 19877600 0 564 19116 27478 10 4 85 0 4 0 0 32601492 21360 19877600 0 0 16996 24286 14 5 81 0 2 0 0 32601492 21360 19877600 0 661 15754 23371 9 4 87 0 3 0 0 32601268 21360 1987760036 0 9808 13849 11 1 88 0 2 0 0 32601304 21360 19877600 036 11846 13826 14 1 85 0 3 0 0 32601304 21360 198776006437 9445 12263 18 1 81 0 4 0 0 32601604 21360 19877600 29020 9959 14786 10 2 88 0 I hope you can see something strange ;) David. attachment: kvm_stat1.pngattachment: kvm_stat2.png
Re: performance trouble
Hello, Le Tue, Jan 31, 2012 at 07:21:25PM +0200, Avi Kivity ecrivait : How many vcpus are there in total (over all guests)? For test we run only 2 VM (RDS/TSE) with 2vcpus for each That's really strange. No network traffic/cpu/block I/O? What do 'vmstat 1' and 'top -d 1' say during this? vmstat 1 during the slowly function : procs ---memory-- ---swap-- -io -system-- cpu 0 0 0 16104568 20524 19558800 0 0 9918 19301 1 1 98 0 1 0 0 16104348 20524 19558800 0 0 9868 19312 1 1 98 0 0 0 0 16104224 20524 19558800 012 10598 19956 0 1 99 0 3 0 0 16104224 20524 1955880016 4 15044 25466 6 3 91 0 2 0 0 16104352 20524 19558800 0 0 16126 26861 6 4 90 0 0 0 0 16104476 20524 19558800 0 0 14598 24664 6 3 91 0 0 0 0 16104600 20524 19558800 0 0 15025 24970 7 3 90 0 3 0 0 16104740 20524 19558800 0 0 14859 24942 5 2 93 0 3 0 0 16104748 20524 19558800 012 14898 24633 9 3 87 0 2 0 0 16104872 20524 19558800 0 0 14625 24840 9 5 86 0 3 0 0 16104872 20524 19558800 0 0 14765 24801 4 2 95 0 1 0 0 16104748 20524 19558800 0 146 12356 20437 7 2 91 0 1 0 0 16104616 20524 19558800 0 0 10768 17531 11 1 88 0 2 0 0 16104724 20524 19558800 012 11729 18996 10 2 89 0 top : during the function, top for the KVM where the function runs increase from ~50% to 130% and decrease to ~50% at the end of the function. In Windows, we see that one CPU is used. The CPU in this KVM server runs at 2.3GHz (in the vmware server, cpu runs at 2GHz). For kvm_stats, I mean it's better to have only one VM with one user for the test so I send this evening or tomorrow morning. David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Mon, Jan 30, 2012 at 07:21:20PM +0200, Avi Kivity ecrivait : Start with top and vmstat to see if the cpu or I/O are the bottleneck with top at the kvm layer, the VM take 50% of CPU (the physical box have 16 real cores-not HT- : AMD Opteron 6134). one result of vmstat : procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 5 0 0 9904544 19612 19544000 01110 6 5 88 0 and I run it 1H and the result is always the same (just free result change time to time). (also helpful to run the Windows performance monitor in the guest). If between the start of the function and the response, the are no performance pic or something like that in Windows. What we can see : o we start the function, o we see network communication with the database server during ~3/4s o after nothing during 10s before display the result David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
What's up with the cc list? On 01/31/2012 07:15 PM, David Cure wrote: Le Mon, Jan 30, 2012 at 07:21:20PM +0200, Avi Kivity ecrivait : Start with top and vmstat to see if the cpu or I/O are the bottleneck with top at the kvm layer, the VM take 50% of CPU (the physical box have 16 real cores-not HT- : AMD Opteron 6134). one result of vmstat : procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 5 0 0 9904544 19612 19544000 01110 6 5 88 0 and I run it 1H and the result is always the same (just free result change time to time). Running vmstat with no arguments gives the average for all uptime. It's meaningless. Run 'vmstat 1'. How many vcpus are there in total (over all guests)? What does top show? (also helpful to run the Windows performance monitor in the guest). If between the start of the function and the response, the are no performance pic or something like that in Windows. What we can see : o we start the function, o we see network communication with the database server during ~3/4s o after nothing during 10s before display the result That's really strange. No network traffic/cpu/block I/O? What do 'vmstat 1' and 'top -d 1' say during this? -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Mon, Jan 23, 2012 at 09:28:37AM +0100, David Cure ecrivait : I use several kvm box, and no problem at all except for 1 application that have bad response time. The VM runs Windows 2008R2 and the application is an client-server app develop with progress software and talk to an Oracle databasei (on another server) and we access this app with RDS/TSE. The physical server runs Debian testing to have qemu-kvm 1.0 and linux kernel 3.1 and libvirt 0.9.8. We use virtio for disk and network and use the last driver for Windows (from RH). We have 2 references servers : one physical and one running Vmware. Response time : o physical = 7s o VM under vmware = 8s o VM under KVM = 12s (to complete with qemu-kvm 0.12.5 and kernel 2.6.32 we have 22s ...). I attach the libvirt xml of my vm. How can I see what's append ? Do you have idea to increase performance ? no one interesting to try to track this issue where kvm is so slow ? David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On Monday, January 30, 2012 09:36:55 AM David Cure wrote: Le Mon, Jan 23, 2012 at 09:28:37AM +0100, David Cure ecrivait : I use several kvm box, and no problem at all except for 1 application that have bad response time. The VM runs Windows 2008R2 and the application is an client-server app develop with progress software and talk to an Oracle databasei (on another server) and we access this app with RDS/TSE. The physical server runs Debian testing to have qemu-kvm 1.0 and linux kernel 3.1 and libvirt 0.9.8. We use virtio for disk and network and use the last driver for Windows (from RH). We have 2 references servers : one physical and one running Vmware. Response time : o physical = 7s o VM under vmware = 8s o VM under KVM = 12s (to complete with qemu-kvm 0.12.5 and kernel 2.6.32 we have 22s ...). I attach the libvirt xml of my vm. How can I see what's append ? Do you have idea to increase performance ? no one interesting to try to track this issue where kvm is so slow ? David. Without more info or some way to reproduce the problem, it would be pointless for one of the devs to spend much time on it. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Mon, Jan 30, 2012 at 10:20:34AM -0600, Brian Jackson ecrivait : Without more info or some way to reproduce the problem, it would be pointless for one of the devs to spend much time on it. yes for sure. But I mean there is some way to trace at the KVM level what happens when the program runs in the VM ? And perhaps with these infos some one see the problem ;) (because with the upgrade of KVM/qemu we already cut the response time by 2). It's difficult to have more infos because the software use Progress (an L4G development tool that is not free/open-source) to connect to Oracle database, and I'm not the developper, only the person that install it under KVM. And in the same VM, we have another software (developed only in Java) and it works fine. So we suspect some strange behaviour with progress under KVM. To remember, the user connect with RDS/TSE to the VM to runs the program. David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
On 01/30/2012 06:51 PM, David Cure wrote: Le Mon, Jan 30, 2012 at 10:20:34AM -0600, Brian Jackson ecrivait : Without more info or some way to reproduce the problem, it would be pointless for one of the devs to spend much time on it. yes for sure. But I mean there is some way to trace at the KVM level what happens when the program runs in the VM ? And perhaps with these infos some one see the problem ;) (because with the upgrade of KVM/qemu we already cut the response time by 2). Start with top and vmstat to see if the cpu or I/O are the bottleneck (also helpful to run the Windows performance monitor in the guest). If it's the cpu, run kvm_stat and report its output. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
performance trouble
Hello, I use several kvm box, and no problem at all except for 1 application that have bad response time. The VM runs Windows 2008R2 and the application is an client-server app develop with progress software and talk to an Oracle databasei (on another server) and we access this app with RDS/TSE. The physical server runs Debian testing to have qemu-kvm 1.0 and linux kernel 3.1 and libvirt 0.9.8. We use virtio for disk and network and use the last driver for Windows (from RH). We have 2 references servers : one physical and one running Vmware. Response time : o physical = 7s o VM under vmware = 8s o VM under KVM = 12s (to complete with qemu-kvm 0.12.5 and kernel 2.6.32 we have 22s ...). I attach the libvirt xml of my vm. How can I see what's append ? Do you have idea to increase performance ? David. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: performance trouble
Le Mon, Jan 23, 2012 at 09:28:37AM +0100, David Cure ecrivait : I attach the libvirt xml of my vm. I forget to attach ;) David. rds.xml Description: XML document signature.asc Description: Digital signature
