Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-11-08 Thread haozhong . zhang 
On 11/06/15 21:40, Paolo Bonzini wrote:
> 
> 
> On 06/11/2015 13:42, Haozhong Zhang wrote:
> > On 11/06/15 11:49, Paolo Bonzini wrote:
> >>
> >>
> >> On 20/10/2015 09:39, Haozhong Zhang wrote:
> >>> This patchset adds support for VMX TSC scaling feature which is
> >>> available on Intel Skylake CPU. The specification of VMX TSC scaling
> >>> can be found at
> >>> http://www.intel.com/content/www/us/en/processors/timestamp-counter-scaling-virtualization-white-paper.html
> >>>
> >>> VMX TSC scaling allows guest TSC which is read by guest rdtsc(p)
> >>> instructions increases in a rate that is customized by the hypervisor
> >>> and can be different than the host TSC rate. Basically, VMX TSC
> >>> scaling adds a 64-bit field called TSC multiplier in VMCS so that, if
> >>> VMX TSC scaling is enabled, TSC read by guest rdtsc(p) instructions
> >>> will be calculated by the following formula:
> >>>
> >>>   guest EDX:EAX = (Host TSC * TSC multiplier) >> 48 + VMX TSC Offset
> >>>
> >>> where, Host TSC = Host MSR_IA32_TSC + Host MSR_IA32_TSC_ADJUST.
> >>>
> >>> This patchset, when cooperating with another QEMU patchset (sent in
> >>> another email "target-i386: save/restore vcpu's TSC rate during
> >>> migration"), allows guest programs observe a consistent TSC rate even
> >>> though they are migrated among machines with different host TSC rates.
> >>>
> >>> VMX TSC scaling shares some common logics with SVM TSC ratio which
> >>> is already supported by KVM. Patch 1 ~ 8 move those common logics from
> >>> SVM code to the common code. Upon them, patch 9 ~ 12 add VMX-specific
> >>> support for VMX TSC scaling.
> >>>
> >>> Changes in v2:
> >>>  * Remove the duplicated variable 'kvm_tsc_scaling_ratio_rsvd'.
> >>>  * Remove an unnecessary error check in original patch 2.
> >>>  * Do 64-bit arithmetic by functions recommended by Paolo.
> >>>  * Make kvm_set_tsc_khz() returns an error number so that ioctl
> >>>KVM_SET_TSC_KHZ does not return 0 if errors happen.
> >>>
> >>> Reviewed-by: Eric Northup 
> >>
> >> Thanks for the patches.  There are a couple changes that I can do myself:
> >>
> >> 1) kvm_default_tsc_scaling_ratio can be initialized in
> >> kvm_arch_hardware_setup, since it's just 1ULL <<
> >> kvm_tsc_scaling_ratio_frac_bits
> >>
> > Agree
> > 
> >> 2) things that you are adding to include/linux/kvm_host.h should instead
> >> go in arch/x86/include/linux/kvm_host.h
> >>
> > Agree, because they are x86 specific.
> > 
> >> That's it.  I'll commit it as soon as I test on AMD (today hopefully).
> 
> It tested fine.  I'll give it a shot with the 32-bit mul_u64_u64_shr on
> Monday as well, but I don't expect any issue.
> 
> Thanks, the patches are neat!
> 
> Paolo

Thank you for the test!

Haozhong

> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-11-06 Thread Paolo Bonzini 


On 20/10/2015 09:39, Haozhong Zhang wrote:
> This patchset adds support for VMX TSC scaling feature which is
> available on Intel Skylake CPU. The specification of VMX TSC scaling
> can be found at
> http://www.intel.com/content/www/us/en/processors/timestamp-counter-scaling-virtualization-white-paper.html
> 
> VMX TSC scaling allows guest TSC which is read by guest rdtsc(p)
> instructions increases in a rate that is customized by the hypervisor
> and can be different than the host TSC rate. Basically, VMX TSC
> scaling adds a 64-bit field called TSC multiplier in VMCS so that, if
> VMX TSC scaling is enabled, TSC read by guest rdtsc(p) instructions
> will be calculated by the following formula:
> 
>   guest EDX:EAX = (Host TSC * TSC multiplier) >> 48 + VMX TSC Offset
> 
> where, Host TSC = Host MSR_IA32_TSC + Host MSR_IA32_TSC_ADJUST.
> 
> This patchset, when cooperating with another QEMU patchset (sent in
> another email "target-i386: save/restore vcpu's TSC rate during
> migration"), allows guest programs observe a consistent TSC rate even
> though they are migrated among machines with different host TSC rates.
> 
> VMX TSC scaling shares some common logics with SVM TSC ratio which
> is already supported by KVM. Patch 1 ~ 8 move those common logics from
> SVM code to the common code. Upon them, patch 9 ~ 12 add VMX-specific
> support for VMX TSC scaling.
> 
> Changes in v2:
>  * Remove the duplicated variable 'kvm_tsc_scaling_ratio_rsvd'.
>  * Remove an unnecessary error check in original patch 2.
>  * Do 64-bit arithmetic by functions recommended by Paolo.
>  * Make kvm_set_tsc_khz() returns an error number so that ioctl
>KVM_SET_TSC_KHZ does not return 0 if errors happen.
> 
> Reviewed-by: Eric Northup 

Thanks for the patches.  There are a couple changes that I can do myself:

1) kvm_default_tsc_scaling_ratio can be initialized in
kvm_arch_hardware_setup, since it's just 1ULL <<
kvm_tsc_scaling_ratio_frac_bits

2) things that you are adding to include/linux/kvm_host.h should instead
go in arch/x86/include/linux/kvm_host.h

That's it.  I'll commit it as soon as I test on AMD (today hopefully).

Paolo

> Haozhong Zhang (12):
>   KVM: x86: Collect information for setting TSC scaling ratio
>   KVM: x86: Add a common TSC scaling ratio field in kvm_vcpu_arch
>   KVM: x86: Add a common TSC scaling function
>   KVM: x86: Replace call-back set_tsc_khz() with a common function
>   KVM: x86: Replace call-back compute_tsc_offset() with a common function
>   KVM: x86: Move TSC scaling logic out of call-back adjust_tsc_offset()
>   KVM: x86: Move TSC scaling logic out of call-back read_l1_tsc()
>   KVM: x86: Use the correct vcpu's TSC rate to compute time scale
>   KVM: VMX: Enable and initialize VMX TSC scaling
>   KVM: VMX: Setup TSC scaling ratio when a vcpu is loaded
>   KVM: VMX: Use a scaled host TSC for guest readings of MSR_IA32_TSC
>   KVM: VMX: Dump TSC multiplier in dump_vmcs()
> 
>  arch/x86/include/asm/kvm_host.h |  24 +++
>  arch/x86/include/asm/vmx.h  |   3 +
>  arch/x86/kvm/lapic.c|   4 +-
>  arch/x86/kvm/svm.c  | 116 --
>  arch/x86/kvm/vmx.c  |  64 ++-
>  arch/x86/kvm/x86.c  | 134 
> +++-
>  include/linux/kvm_host.h|  20 ++
>  include/linux/math64.h  |  99 +
>  8 files changed, 297 insertions(+), 167 deletions(-)
> 
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-11-06 Thread Haozhong Zhang 
On 11/06/15 11:49, Paolo Bonzini wrote:
> 
> 
> On 20/10/2015 09:39, Haozhong Zhang wrote:
> > This patchset adds support for VMX TSC scaling feature which is
> > available on Intel Skylake CPU. The specification of VMX TSC scaling
> > can be found at
> > http://www.intel.com/content/www/us/en/processors/timestamp-counter-scaling-virtualization-white-paper.html
> > 
> > VMX TSC scaling allows guest TSC which is read by guest rdtsc(p)
> > instructions increases in a rate that is customized by the hypervisor
> > and can be different than the host TSC rate. Basically, VMX TSC
> > scaling adds a 64-bit field called TSC multiplier in VMCS so that, if
> > VMX TSC scaling is enabled, TSC read by guest rdtsc(p) instructions
> > will be calculated by the following formula:
> > 
> >   guest EDX:EAX = (Host TSC * TSC multiplier) >> 48 + VMX TSC Offset
> > 
> > where, Host TSC = Host MSR_IA32_TSC + Host MSR_IA32_TSC_ADJUST.
> > 
> > This patchset, when cooperating with another QEMU patchset (sent in
> > another email "target-i386: save/restore vcpu's TSC rate during
> > migration"), allows guest programs observe a consistent TSC rate even
> > though they are migrated among machines with different host TSC rates.
> > 
> > VMX TSC scaling shares some common logics with SVM TSC ratio which
> > is already supported by KVM. Patch 1 ~ 8 move those common logics from
> > SVM code to the common code. Upon them, patch 9 ~ 12 add VMX-specific
> > support for VMX TSC scaling.
> > 
> > Changes in v2:
> >  * Remove the duplicated variable 'kvm_tsc_scaling_ratio_rsvd'.
> >  * Remove an unnecessary error check in original patch 2.
> >  * Do 64-bit arithmetic by functions recommended by Paolo.
> >  * Make kvm_set_tsc_khz() returns an error number so that ioctl
> >KVM_SET_TSC_KHZ does not return 0 if errors happen.
> > 
> > Reviewed-by: Eric Northup 
> 
> Thanks for the patches.  There are a couple changes that I can do myself:
> 
> 1) kvm_default_tsc_scaling_ratio can be initialized in
> kvm_arch_hardware_setup, since it's just 1ULL <<
> kvm_tsc_scaling_ratio_frac_bits
>
Agree

> 2) things that you are adding to include/linux/kvm_host.h should instead
> go in arch/x86/include/linux/kvm_host.h
>
Agree, because they are x86 specific.

> That's it.  I'll commit it as soon as I test on AMD (today hopefully).
>

Thanks,
Haozhong

> Paolo
> 
> > Haozhong Zhang (12):
> >   KVM: x86: Collect information for setting TSC scaling ratio
> >   KVM: x86: Add a common TSC scaling ratio field in kvm_vcpu_arch
> >   KVM: x86: Add a common TSC scaling function
> >   KVM: x86: Replace call-back set_tsc_khz() with a common function
> >   KVM: x86: Replace call-back compute_tsc_offset() with a common function
> >   KVM: x86: Move TSC scaling logic out of call-back adjust_tsc_offset()
> >   KVM: x86: Move TSC scaling logic out of call-back read_l1_tsc()
> >   KVM: x86: Use the correct vcpu's TSC rate to compute time scale
> >   KVM: VMX: Enable and initialize VMX TSC scaling
> >   KVM: VMX: Setup TSC scaling ratio when a vcpu is loaded
> >   KVM: VMX: Use a scaled host TSC for guest readings of MSR_IA32_TSC
> >   KVM: VMX: Dump TSC multiplier in dump_vmcs()
> > 
> >  arch/x86/include/asm/kvm_host.h |  24 +++
> >  arch/x86/include/asm/vmx.h  |   3 +
> >  arch/x86/kvm/lapic.c|   4 +-
> >  arch/x86/kvm/svm.c  | 116 --
> >  arch/x86/kvm/vmx.c  |  64 ++-
> >  arch/x86/kvm/x86.c  | 134 
> > +++-
> >  include/linux/kvm_host.h|  20 ++
> >  include/linux/math64.h  |  99 +
> >  8 files changed, 297 insertions(+), 167 deletions(-)
> > 
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-11-06 Thread Paolo Bonzini 


On 06/11/2015 13:42, Haozhong Zhang wrote:
> On 11/06/15 11:49, Paolo Bonzini wrote:
>>
>>
>> On 20/10/2015 09:39, Haozhong Zhang wrote:
>>> This patchset adds support for VMX TSC scaling feature which is
>>> available on Intel Skylake CPU. The specification of VMX TSC scaling
>>> can be found at
>>> http://www.intel.com/content/www/us/en/processors/timestamp-counter-scaling-virtualization-white-paper.html
>>>
>>> VMX TSC scaling allows guest TSC which is read by guest rdtsc(p)
>>> instructions increases in a rate that is customized by the hypervisor
>>> and can be different than the host TSC rate. Basically, VMX TSC
>>> scaling adds a 64-bit field called TSC multiplier in VMCS so that, if
>>> VMX TSC scaling is enabled, TSC read by guest rdtsc(p) instructions
>>> will be calculated by the following formula:
>>>
>>>   guest EDX:EAX = (Host TSC * TSC multiplier) >> 48 + VMX TSC Offset
>>>
>>> where, Host TSC = Host MSR_IA32_TSC + Host MSR_IA32_TSC_ADJUST.
>>>
>>> This patchset, when cooperating with another QEMU patchset (sent in
>>> another email "target-i386: save/restore vcpu's TSC rate during
>>> migration"), allows guest programs observe a consistent TSC rate even
>>> though they are migrated among machines with different host TSC rates.
>>>
>>> VMX TSC scaling shares some common logics with SVM TSC ratio which
>>> is already supported by KVM. Patch 1 ~ 8 move those common logics from
>>> SVM code to the common code. Upon them, patch 9 ~ 12 add VMX-specific
>>> support for VMX TSC scaling.
>>>
>>> Changes in v2:
>>>  * Remove the duplicated variable 'kvm_tsc_scaling_ratio_rsvd'.
>>>  * Remove an unnecessary error check in original patch 2.
>>>  * Do 64-bit arithmetic by functions recommended by Paolo.
>>>  * Make kvm_set_tsc_khz() returns an error number so that ioctl
>>>KVM_SET_TSC_KHZ does not return 0 if errors happen.
>>>
>>> Reviewed-by: Eric Northup 
>>
>> Thanks for the patches.  There are a couple changes that I can do myself:
>>
>> 1) kvm_default_tsc_scaling_ratio can be initialized in
>> kvm_arch_hardware_setup, since it's just 1ULL <<
>> kvm_tsc_scaling_ratio_frac_bits
>>
> Agree
> 
>> 2) things that you are adding to include/linux/kvm_host.h should instead
>> go in arch/x86/include/linux/kvm_host.h
>>
> Agree, because they are x86 specific.
> 
>> That's it.  I'll commit it as soon as I test on AMD (today hopefully).

It tested fine.  I'll give it a shot with the 32-bit mul_u64_u64_shr on
Monday as well, but I don't expect any issue.

Thanks, the patches are neat!

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-10-23 Thread Joerg Roedel 
On Tue, Oct 20, 2015 at 03:39:00PM +0800, Haozhong Zhang wrote:
> VMX TSC scaling shares some common logics with SVM TSC ratio which
> is already supported by KVM. Patch 1 ~ 8 move those common logics from
> SVM code to the common code. Upon them, patch 9 ~ 12 add VMX-specific
> support for VMX TSC scaling.

Have you tested your changes on an AMD machine too?


Joerg

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-10-23 Thread Joerg Roedel 
On Fri, Oct 23, 2015 at 08:32:28PM +0800, Haozhong Zhang wrote:
> No, since I don't have AMD machines at hand. The modifications to SVM
> code are mostly lifting common code with VMX TSC scaling code, so it
> should still work on AMD machines.

Well, I think it would be good if you can provide a Tested-by on AMD
machines from someone who has one. Or get one yourself when changing AMD
specific code, they are not that expensive :)
I can do some testing when I am back from my travels, but that will not
be before early November.

Joerg
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-10-23 Thread Paolo Bonzini 


On 23/10/2015 14:46, Joerg Roedel wrote:
>> > No, since I don't have AMD machines at hand. The modifications to SVM
>> > code are mostly lifting common code with VMX TSC scaling code, so it
>> > should still work on AMD machines.
> Well, I think it would be good if you can provide a Tested-by on AMD
> machines from someone who has one. Or get one yourself when changing AMD
> specific code, they are not that expensive :)
> I can do some testing when I am back from my travels, but that will not
> be before early November.

I have one now (mine, not just Red Hat's). :D

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-10-23 Thread Haozhong Zhang 
On Fri, Oct 23, 2015 at 02:46:19PM +0200, Joerg Roedel wrote:
> On Fri, Oct 23, 2015 at 08:32:28PM +0800, Haozhong Zhang wrote:
> > No, since I don't have AMD machines at hand. The modifications to SVM
> > code are mostly lifting common code with VMX TSC scaling code, so it
> > should still work on AMD machines.
> 
> Well, I think it would be good if you can provide a Tested-by on AMD
> machines from someone who has one. Or get one yourself when changing AMD
> specific code, they are not that expensive :)
> I can do some testing when I am back from my travels, but that will not
> be before early November.
> 
>   Joerg

I'll try to get a test result. And it would be very appreciated if you
could test as well.

Thanks!
Haozhong

> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-10-23 Thread Haozhong Zhang 
On Fri, Oct 23, 2015 at 12:06:50PM +0200, Joerg Roedel wrote:
> On Tue, Oct 20, 2015 at 03:39:00PM +0800, Haozhong Zhang wrote:
> > VMX TSC scaling shares some common logics with SVM TSC ratio which
> > is already supported by KVM. Patch 1 ~ 8 move those common logics from
> > SVM code to the common code. Upon them, patch 9 ~ 12 add VMX-specific
> > support for VMX TSC scaling.
> 
> Have you tested your changes on an AMD machine too?
> 
> 
>   Joerg
> 

No, since I don't have AMD machines at hand. The modifications to SVM
code are mostly lifting common code with VMX TSC scaling code, so it
should still work on AMD machines.

Haozhong
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-10-23 Thread Haozhong Zhang 
On Fri, Oct 23, 2015 at 02:51:06PM +0200, Paolo Bonzini wrote:
> 
> 
> On 23/10/2015 14:46, Joerg Roedel wrote:
> >> > No, since I don't have AMD machines at hand. The modifications to SVM
> >> > code are mostly lifting common code with VMX TSC scaling code, so it
> >> > should still work on AMD machines.
> > Well, I think it would be good if you can provide a Tested-by on AMD
> > machines from someone who has one. Or get one yourself when changing AMD
> > specific code, they are not that expensive :)
> > I can do some testing when I am back from my travels, but that will not
> > be before early November.
> 
> I have one now (mine, not just Red Hat's). :D
> 
> Paolo

Hi Paolo,

I just posted the test instructions. It would be very appreciated if
you can help to test this patchset on AMD machines (two are required).

Thanks,
Haozhong

> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 00/12] KVM: x86: add support for VMX TSC scaling

2015-10-23 Thread Haozhong Zhang 
Following is how I test this patchset. It should also apply to AMD
machines by replacing Intel with AMD and VMX TSC scaling with SVM TSC
ratio.

* Hardware Requirements
  1) Two machines with Intel CPUs, called M_A and M_B below.
  2) TSC frequency of CPUs on M_A is different from CPUs on M_B.
 Suppose TSC frequency on M_A is f_a KHz.
  3) At least CPUs on M_B support VMX TSC scaling.

* Software Requirements
  1) Apply this patchset to KVM on both machines.
  2) Apply QEMU patches[1] to QEMU commit 40fe17b on both machines

* Test Process
  1) Start a linux guest on M_A
   qemu-system-x86_64 -enable-kvm -smp 4 -cpu qemu66 -m 512 -hda linux.img
   
  2) In guest linux, check the TSC frequency detected by Linux kernel.
 e.g. search in dmeg for messages like
   "tsc: Detected XYZ.ABC MHz processor" or
   "tsc: Refined TSC clocksource calibration: XYZ.ABC MHz"
  
  3) Start QEMU waiting for migration on M_B:
   qemu-system-x86_64 -enable-kvm -smp 4 -cpu qemu64,load-tsc-freq -m 512 
-hda linux.img -incoming tcp:0:1234
   
  4) Migrate above VM to M_B as normal in QEMU monitor:
   migrate tcp::1234
   
  5) After the migration, if VMX TSC scaling and this patchset work on
 M_B, no messages like
   "Clocksource tsc unstable (delta = x ns)"
 should appear in dmesg of guest linux

  6) Furthermore, users can also check whether guest TSC after the
 migration increases in the same rate as before by running the
 attached program test_tsc in VM:
   ./test_tsc N f_a
 It measures the number of TSC ticks passed in N seconds, and
 divides it by the expected TSC frequency f_a to get the output
 result. If this patchset works, the output should be very closed
 to N
  
[1] http://www.spinics.net/lists/kvm/msg122421.html

Thanks,
Haozhong
#include 
#include 
#include 
#include 

static inline uint64_t rdtsc(void)
{
uint32_t lo, hi;
asm volatile("lfence; rdtsc" : "=a" (lo), "=d" (hi));
return (uint64_t)hi << 32 | lo;
}

int main(int argc, char **argv)
{
uint64_t tsc0, tsc1;
int ns, tsc_khz;
double delta;

if (argc < 2) {
printf("Usage: %s  \n", argv[0]);
return -1;
}

if ((ns = atoi(argv[1])) <= 0)
return -1;
if ((tsc_khz = atoi(argv[2])) <= 0)
return -1;

tsc0 = rdtsc();
sleep(ns);
tsc1 = rdtsc();

delta = tsc1 - tsc0;
printf("Passed %lf s\n", delta / (tsc_khz * 1000.0));

return 0;
}