Re: [PATCH v7 0/2] KVM: LAPIC: Implement Exitless Timer
On Sat, 6 Jul 2019 at 09:26, Wanpeng Li wrote:
>
> Dedicated instances are currently disturbed by unnecessary jitter due
> to the emulated lapic timers fire on the same pCPUs which vCPUs resident.
> There is no hardware virtual timer on Intel for guest like ARM. Both
> programming timer in guest and the emulated timer fires incur vmexits.
> This patchset tries to avoid vmexit which is incurred by the emulated
> timer fires in dedicated instance scenario.
>
> When nohz_full is enabled in dedicated instances scenario, the unpinned
> timer will be moved to the nearest busy housekeepers after commit
> 9642d18eee2cd (nohz: Affine unpinned timers to housekeepers) and commit
> 444969223c8 ("sched/nohz: Fix affine unpinned timers mess"). However,
> KVM always makes lapic timer pinned to the pCPU which vCPU residents, the
> reason is explained by commit 61abdbe0 (kvm: x86: make lapic hrtimer
> pinned). Actually, these emulated timers can be offload to the housekeeping
> cpus since APICv is really common in recent years. The guest timer interrupt
> is injected by posted-interrupt which is delivered by housekeeping cpu
> once the emulated timer fires.
>
> The host admin should fine tuned, e.g. dedicated instances scenario w/
> nohz_full cover the pCPUs which vCPUs resident, several pCPUs surplus
> for busy housekeeping, disable mwait/hlt/pause vmexits to keep in non-root
> mode, ~3% redis performance benefit can be observed on Skylake server.
>
> w/o patchset:
>
> VM-EXIT Samples Samples% Time% Min Time Max Time Avg time
>
> EXTERNAL_INTERRUPT4291649.43% 39.30% 0.47us 106.09us 0.71us (
> +- 1.09% )
>
> w/ patchset:
>
> VM-EXIT Samples Samples% Time% Min Time Max Time
> Avg time
>
> EXTERNAL_INTERRUPT6871 9.29% 2.96% 0.44us57.88us 0.72us (
> +- 4.02% )
>
> Cc: Paolo Bonzini
> Cc: Radim Krčmář
> Cc: Marcelo Tosatti
>
> v6 -> v7:
> * remove bool argument
>
> v5 -> v6:
> * don't overwrites whatever the user specified
> * introduce kvm_can_post_timer_interrupt and kvm_use_posted_timer_interrupt
> * remove kvm_hlt_in_guest() condition
> * squash all of 2/3/4 together
>
> v4 -> v5:
> * update patch description in patch 1/4
> * feed latest apic->lapic_timer.expired_tscdeadline to
> kvm_wait_lapic_expire()
> * squash advance timer handling to patch 2/4
>
> v3 -> v4:
> * drop the HRTIMER_MODE_ABS_PINNED, add kick after set pending timer
> * don't posted inject already-expired timer
>
> v2 -> v3:
> * disarming the vmx preemption timer when
> posted_interrupt_inject_timer_enabled()
> * check kvm_hlt_in_guest instead
>
> v1 -> v2:
> * check vcpu_halt_in_guest
> * move module parameter from kvm-intel to kvm
> * add housekeeping_enabled
> * rename apic_timer_expired_pi to kvm_apic_inject_pending_timer_irqs
>
>
> Wanpeng Li (2):
> KVM: LAPIC: Make lapic timer unpinned
> KVM: LAPIC: Inject timer interrupt via posted interrupt
There is a further optimization for this feature in houseeking/hrtimer
subsystem.
[1] https://lkml.org/lkml/2019/7/25/963
[2] https://lkml.org/lkml/2019/6/28/231
The [2] patch tries to optimize the worst case, however, it will not
be merged by maintainers and get offline confirm, Thomas will refactor
this to avoid to predict the future on every timer enqueue. Anyway, it
still should be considered to be backported to product environment as
long as get_nohz_timer_target() is using.
Regards,
Wanpeng Li
Re: [PATCH v7 0/2] KVM: LAPIC: Implement Exitless Timer
On Thu, 11 Jul 2019 at 22:05, Paolo Bonzini wrote: > > On 11/07/19 15:50, Wanpeng Li wrote: > > kindly ping， > > Sorry, I need more time to review this. It's basically the only > remaining item for the 5.3 merge window, even though it won't be part of > the first pull request to Linus. Thank you! Hope finally we will not miss it. :) Regards, Wanpeng Li
Re: [PATCH v7 0/2] KVM: LAPIC: Implement Exitless Timer
On 11/07/19 15:50, Wanpeng Li wrote:
> kindly ping，
Sorry, I need more time to review this. It's basically the only
remaining item for the 5.3 merge window, even though it won't be part of
the first pull request to Linus.
Paolo
>> Dedicated instances are currently disturbed by unnecessary jitter due
>> to the emulated lapic timers fire on the same pCPUs which vCPUs resident.
>> There is no hardware virtual timer on Intel for guest like ARM. Both
>> programming timer in guest and the emulated timer fires incur vmexits.
>> This patchset tries to avoid vmexit which is incurred by the emulated
>> timer fires in dedicated instance scenario.
>>
>> When nohz_full is enabled in dedicated instances scenario, the unpinned
>> timer will be moved to the nearest busy housekeepers after commit
>> 9642d18eee2cd (nohz: Affine unpinned timers to housekeepers) and commit
>> 444969223c8 ("sched/nohz: Fix affine unpinned timers mess"). However,
>> KVM always makes lapic timer pinned to the pCPU which vCPU residents, the
>> reason is explained by commit 61abdbe0 (kvm: x86: make lapic hrtimer
>> pinned). Actually, these emulated timers can be offload to the housekeeping
>> cpus since APICv is really common in recent years. The guest timer interrupt
>> is injected by posted-interrupt which is delivered by housekeeping cpu
>> once the emulated timer fires.
>>
>> The host admin should fine tuned, e.g. dedicated instances scenario w/
>> nohz_full cover the pCPUs which vCPUs resident, several pCPUs surplus
>> for busy housekeeping, disable mwait/hlt/pause vmexits to keep in non-root
>> mode, ~3% redis performance benefit can be observed on Skylake server.
>>
>> w/o patchset:
>>
>>VM-EXIT Samples Samples% Time% Min Time Max Time Avg time
>>
>> EXTERNAL_INTERRUPT4291649.43% 39.30% 0.47us 106.09us 0.71us
>> ( +- 1.09% )
>>
>> w/ patchset:
>>
>>VM-EXIT Samples Samples% Time% Min Time Max Time
>> Avg time
>>
>> EXTERNAL_INTERRUPT6871 9.29% 2.96% 0.44us57.88us 0.72us
>> ( +- 4.02% )
>>
>> Cc: Paolo Bonzini
>> Cc: Radim Krčmář
>> Cc: Marcelo Tosatti
>>
>> v6 -> v7:
>> * remove bool argument
>>
>> v5 -> v6:
>> * don't overwrites whatever the user specified
>> * introduce kvm_can_post_timer_interrupt and kvm_use_posted_timer_interrupt
>> * remove kvm_hlt_in_guest() condition
>> * squash all of 2/3/4 together
>>
>> v4 -> v5:
>> * update patch description in patch 1/4
>> * feed latest apic->lapic_timer.expired_tscdeadline to
>> kvm_wait_lapic_expire()
>> * squash advance timer handling to patch 2/4
>>
>> v3 -> v4:
>> * drop the HRTIMER_MODE_ABS_PINNED, add kick after set pending timer
>> * don't posted inject already-expired timer
>>
>> v2 -> v3:
>> * disarming the vmx preemption timer when
>> posted_interrupt_inject_timer_enabled()
>> * check kvm_hlt_in_guest instead
>>
>> v1 -> v2:
>> * check vcpu_halt_in_guest
>> * move module parameter from kvm-intel to kvm
>> * add housekeeping_enabled
>> * rename apic_timer_expired_pi to kvm_apic_inject_pending_timer_irqs
>>
>>
>> Wanpeng Li (2):
>> KVM: LAPIC: Make lapic timer unpinned
>> KVM: LAPIC: Inject timer interrupt via posted interrupt
>>
>> arch/x86/kvm/lapic.c| 109
>> ++--
>> arch/x86/kvm/lapic.h| 1 +
>> arch/x86/kvm/vmx/vmx.c | 3 +-
>> arch/x86/kvm/x86.c | 12 +++--
>> arch/x86/kvm/x86.h | 2 +
>> include/linux/sched/isolation.h | 2 +
>> kernel/sched/isolation.c| 6 +++
>> 7 files changed, 90 insertions(+), 45 deletions(-)
>>
>> --
>> 1.8.3.1
>>
[PATCH v7 0/2] KVM: LAPIC: Implement Exitless Timer
Dedicated instances are currently disturbed by unnecessary jitter due
to the emulated lapic timers fire on the same pCPUs which vCPUs resident.
There is no hardware virtual timer on Intel for guest like ARM. Both
programming timer in guest and the emulated timer fires incur vmexits.
This patchset tries to avoid vmexit which is incurred by the emulated
timer fires in dedicated instance scenario.
When nohz_full is enabled in dedicated instances scenario, the unpinned
timer will be moved to the nearest busy housekeepers after commit
9642d18eee2cd (nohz: Affine unpinned timers to housekeepers) and commit
444969223c8 ("sched/nohz: Fix affine unpinned timers mess"). However,
KVM always makes lapic timer pinned to the pCPU which vCPU residents, the
reason is explained by commit 61abdbe0 (kvm: x86: make lapic hrtimer
pinned). Actually, these emulated timers can be offload to the housekeeping
cpus since APICv is really common in recent years. The guest timer interrupt
is injected by posted-interrupt which is delivered by housekeeping cpu
once the emulated timer fires.
The host admin should fine tuned, e.g. dedicated instances scenario w/
nohz_full cover the pCPUs which vCPUs resident, several pCPUs surplus
for busy housekeeping, disable mwait/hlt/pause vmexits to keep in non-root
mode, ~3% redis performance benefit can be observed on Skylake server.
w/o patchset:
VM-EXIT Samples Samples% Time% Min Time Max Time Avg time
EXTERNAL_INTERRUPT4291649.43% 39.30% 0.47us 106.09us 0.71us (
+- 1.09% )
w/ patchset:
VM-EXIT Samples Samples% Time% Min Time Max Time Avg
time
EXTERNAL_INTERRUPT6871 9.29% 2.96% 0.44us57.88us 0.72us (
+- 4.02% )
Cc: Paolo Bonzini
Cc: Radim Krčmář
Cc: Marcelo Tosatti
v6 -> v7:
* remove bool argument
v5 -> v6:
* don't overwrites whatever the user specified
* introduce kvm_can_post_timer_interrupt and kvm_use_posted_timer_interrupt
* remove kvm_hlt_in_guest() condition
* squash all of 2/3/4 together
v4 -> v5:
* update patch description in patch 1/4
* feed latest apic->lapic_timer.expired_tscdeadline to kvm_wait_lapic_expire()
* squash advance timer handling to patch 2/4
v3 -> v4:
* drop the HRTIMER_MODE_ABS_PINNED, add kick after set pending timer
* don't posted inject already-expired timer
v2 -> v3:
* disarming the vmx preemption timer when
posted_interrupt_inject_timer_enabled()
* check kvm_hlt_in_guest instead
v1 -> v2:
* check vcpu_halt_in_guest
* move module parameter from kvm-intel to kvm
* add housekeeping_enabled
* rename apic_timer_expired_pi to kvm_apic_inject_pending_timer_irqs
Wanpeng Li (2):
KVM: LAPIC: Make lapic timer unpinned
KVM: LAPIC: Inject timer interrupt via posted interrupt
arch/x86/kvm/lapic.c| 109 ++--
arch/x86/kvm/lapic.h| 1 +
arch/x86/kvm/vmx/vmx.c | 3 +-
arch/x86/kvm/x86.c | 12 +++--
arch/x86/kvm/x86.h | 2 +
include/linux/sched/isolation.h | 2 +
kernel/sched/isolation.c| 6 +++
7 files changed, 90 insertions(+), 45 deletions(-)
--
1.8.3.1
