RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-10-15 Thread Wu, Feng 


> -Original Message-
> From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo
> Bonzini
> Sent: Friday, October 16, 2015 2:13 AM
> To: David Matlack <dmatl...@google.com>; Wu, Feng <feng...@intel.com>
> Cc: alex.william...@redhat.com; Joerg Roedel <j...@8bytes.org>; Marcelo
> Tosatti <mtosa...@redhat.com>; eric.au...@linaro.org; kvm list
> <kvm@vger.kernel.org>; io...@lists.linux-foundation.org; linux-
> ker...@vger.kernel.org
> Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when
> vCPU is blocked
> 
> 
> 
> On 15/10/2015 19:39, David Matlack wrote:
> > But after spending more time reading the source code this morning I
> > found that kvm_vcpu_check_block() eventually calls into
> > vmx_sync_pir_to_irr(), which copies PIR to IRR and clears ON. And then
> > apic_find_highest_irr() detects the pending posted interrupt.
> 
> Right.  And related to this, Feng, can you check if this is still
> necessary on kvm/queue:
> 
> @@ -6518,6 +6523,20 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>   kvm_vcpu_reload_apic_access_page(vcpu);
>   }
> 
> + /*
> +  * KVM_REQ_EVENT is not set when posted interrupts are set by
> +  * VT-d hardware, so we have to update RVI unconditionally.
> +  */
> + if (kvm_lapic_enabled(vcpu)) {
> + /*
> +  * Update architecture specific hints for APIC
> +  * virtual interrupt delivery.
> +  */
> + if (kvm_x86_ops->hwapic_irr_update)
> + kvm_x86_ops->hwapic_irr_update(vcpu,
> + kvm_lapic_find_highest_irr(vcpu));
> + }
> +
>   if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win) {
>   kvm_apic_accept_events(vcpu);
>   if (vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) {
> @@ -6534,13 +6553,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>   kvm_x86_ops->enable_irq_window(vcpu);
> 
>   if (kvm_lapic_enabled(vcpu)) {
> - /*
> -  * Update architecture specific hints for APIC
> -  * virtual interrupt delivery.
> -  */
> - if (kvm_x86_ops->hwapic_irr_update)
> - kvm_x86_ops->hwapic_irr_update(vcpu,
> - kvm_lapic_find_highest_irr(vcpu));
>   update_cr8_intercept(vcpu);
>   kvm_lapic_sync_to_vapic(vcpu);
>   }
> 

I think the above code is needed, before the place where 'KVM_REQ_EVENT'
got checked in vcpu_enter_guest(), VT-d hardware can issue notification
event at any time. Consider the following scenario:

vcpu_run()
{
..  

for(;;) {
point #1
vcpu_enter_guest()
}   

point #2
}

For example, if we receive notification events issued by VT-d hardware at
point #1 and point#2, then enter vcpu_enter_guest() with 'KVM_REQ_EVENT'
not set, the interrupts cannot be delivered to guest during _this_ VM-Entry.

The point is that VT-d hardware can issue notification event at any time,
but it cannot set 'KVM_REQ_EVENT' like software does.

Maybe one thing we can do is only executing the following code when
vt-d pi is enabled,

 +  /*
 +   * KVM_REQ_EVENT is not set when posted interrupts are set by
 +   * VT-d hardware, so we have to update RVI unconditionally.
 +   */
 +  if (kvm_lapic_enabled(vcpu)) {
 +  /*
 +   * Update architecture specific hints for APIC
 +   * virtual interrupt delivery.
 +   */
 +  if (kvm_x86_ops->hwapic_irr_update)
 +  kvm_x86_ops->hwapic_irr_update(vcpu,
 +  kvm_lapic_find_highest_irr(vcpu));
 +  }
 +

And do this inside the KVM_REQ_EVENT check when VT-d PI is not enabled.

Thanks,
Feng

> 
> It may be obsolete now that we have the patch from Radim to set
> KVM_REQ_EVENT
> in vmx_sync_pir_to_irr
> (http://permalink.gmane.org/gmane.linux.kernel/2057138).
> 
> Thanks,
> 
> Paolo

Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-10-15 Thread David Matlack 
On Wed, Oct 14, 2015 at 6:33 PM, Wu, Feng <feng...@intel.com> wrote:
>
>> -Original Message-
>> From: David Matlack [mailto:dmatl...@google.com]
>> Sent: Thursday, October 15, 2015 7:41 AM
>> To: Wu, Feng <feng...@intel.com>
>> Cc: Paolo Bonzini <pbonz...@redhat.com>; alex.william...@redhat.com; Joerg
>> Roedel <j...@8bytes.org>; Marcelo Tosatti <mtosa...@redhat.com>;
>> eric.au...@linaro.org; kvm list <kvm@vger.kernel.org>; iommu@lists.linux-
>> foundation.org; linux-ker...@vger.kernel.org
>> Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when
>> vCPU is blocked
>>
>> Hi Feng.
>>
>> On Fri, Sep 18, 2015 at 7:29 AM, Feng Wu <feng...@intel.com> wrote:
>> > This patch updates the Posted-Interrupts Descriptor when vCPU
>> > is blocked.
>> >
>> > pre-block:
>> > - Add the vCPU to the blocked per-CPU list
>> > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR
>> >
>> > post-block:
>> > - Remove the vCPU from the per-CPU list
>>
>> I'm wondering what happens if a posted interrupt arrives at the IOMMU
>> after pre-block and before post-block.
>>
>> In pre_block, NV is set to POSTED_INTR_WAKEUP_VECTOR. IIUC, this means
>> future posted interrupts will not trigger "Posted-Interrupt Processing"
>> (PIR will not get copied to VIRR). Instead, the IOMMU will do ON := 1,
>> PIR |= (1 << vector), and send POSTED_INTR_WAKEUP_VECTOR. PIWV calls
>> wakeup_handler which does kvm_vcpu_kick. kvm_vcpu_kick does a wait-queue
>> wakeup and possibly a scheduler ipi.
>>
>> But the VCPU is sitting in kvm_vcpu_block. It spins and/or schedules
>> (wait queue) until it has a reason to wake up. I couldn't find a code
>> path from kvm_vcpu_block that lead to checking ON or PIR. How does the
>> blocked VCPU "receive" the posted interrupt? (And when does Posted-
>> Interrupt Processing get triggered?)
>
> In the pre_block, it also change the 'NDST' filed to the pCPU, on which the 
> vCPU
> is put to the per-CPU list 'blocked_vcpu_on_cpu', so when posted-interrupts
> come it, it will sent the wakeup notification event to the pCPU above, then in
> the wakeup_handler, it can find the vCPU from the per-CPU list, hence
> kvm_vcpu_kick can wake up it.

Thank you for your response. I was actually confused about something
else. After wakeup_handler->kvm_vcpu_kick causes the vcpu to wake up,
that vcpu calls kvm_vcpu_check_block() to check if there are pending
events, otherwise the vcpu goes back to sleep. I had trouble yesterday
finding the code path from kvm_vcpu_check_block() which checks PIR/ON.

But after spending more time reading the source code this morning I
found that kvm_vcpu_check_block() eventually calls into
vmx_sync_pir_to_irr(), which copies PIR to IRR and clears ON. And then
apic_find_highest_irr() detects the pending posted interrupt.

>
> Thanks,
> Feng
>
>>
>> Thanks!
>>
>> >
>> > Signed-off-by: Feng Wu <feng...@intel.com>
>> > ---
>> > v9:
>> > - Add description for blocked_vcpu_on_cpu_lock in
>> Documentation/virtual/kvm/locking.txt
>> > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then
>> >   !irq_remapping_cap(IRQ_POSTING_CAP)
>> >
>> > v8:
>> > - Rename 'pi_pre_block' to 'pre_block'
>> > - Rename 'pi_post_block' to 'post_block'
>> > - Change some comments
>> > - Only add the vCPU to the blocking list when the VM has assigned devices.
>> >
>> >  Documentation/virtual/kvm/locking.txt |  12 +++
>> >  arch/x86/include/asm/kvm_host.h   |  13 +++
>> >  arch/x86/kvm/vmx.c| 153
>> ++
>> >  arch/x86/kvm/x86.c|  53 +---
>> >  include/linux/kvm_host.h  |   3 +
>> >  virt/kvm/kvm_main.c   |   3 +
>> >  6 files changed, 227 insertions(+), 10 deletions(-)
>> >
>> > diff --git a/Documentation/virtual/kvm/locking.txt
>> b/Documentation/virtual/kvm/locking.txt
>> > index d68af4d..19f94a6 100644
>> > --- a/Documentation/virtual/kvm/locking.txt
>> > +++ b/Documentation/virtual/kvm/locking.txt
>> > @@ -166,3 +166,15 @@ Comment:   The srcu read lock must be held while
>> accessing memslots (e.g.
>> > MMIO/PIO address->device structure mapping (kvm->buses).
>> > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu
>> > if it is needed by multiple func

Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-10-15 Thread Paolo Bonzini 


On 15/10/2015 19:39, David Matlack wrote:
> But after spending more time reading the source code this morning I
> found that kvm_vcpu_check_block() eventually calls into
> vmx_sync_pir_to_irr(), which copies PIR to IRR and clears ON. And then
> apic_find_highest_irr() detects the pending posted interrupt.

Right.  And related to this, Feng, can you check if this is still
necessary on kvm/queue:

@@ -6518,6 +6523,20 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
kvm_vcpu_reload_apic_access_page(vcpu);
}
 
+   /*
+* KVM_REQ_EVENT is not set when posted interrupts are set by
+* VT-d hardware, so we have to update RVI unconditionally.
+*/
+   if (kvm_lapic_enabled(vcpu)) {
+   /*
+* Update architecture specific hints for APIC
+* virtual interrupt delivery.
+*/
+   if (kvm_x86_ops->hwapic_irr_update)
+   kvm_x86_ops->hwapic_irr_update(vcpu,
+   kvm_lapic_find_highest_irr(vcpu));
+   }
+
if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win) {
kvm_apic_accept_events(vcpu);
if (vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) {
@@ -6534,13 +6553,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
kvm_x86_ops->enable_irq_window(vcpu);
 
if (kvm_lapic_enabled(vcpu)) {
-   /*
-* Update architecture specific hints for APIC
-* virtual interrupt delivery.
-*/
-   if (kvm_x86_ops->hwapic_irr_update)
-   kvm_x86_ops->hwapic_irr_update(vcpu,
-   kvm_lapic_find_highest_irr(vcpu));
update_cr8_intercept(vcpu);
kvm_lapic_sync_to_vapic(vcpu);
}


It may be obsolete now that we have the patch from Radim to set KVM_REQ_EVENT
in vmx_sync_pir_to_irr (http://permalink.gmane.org/gmane.linux.kernel/2057138).

Thanks,

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-10-14 Thread Wu, Feng 


> -Original Message-
> From: David Matlack [mailto:dmatl...@google.com]
> Sent: Thursday, October 15, 2015 7:41 AM
> To: Wu, Feng <feng...@intel.com>
> Cc: Paolo Bonzini <pbonz...@redhat.com>; alex.william...@redhat.com; Joerg
> Roedel <j...@8bytes.org>; Marcelo Tosatti <mtosa...@redhat.com>;
> eric.au...@linaro.org; kvm list <kvm@vger.kernel.org>; iommu@lists.linux-
> foundation.org; linux-ker...@vger.kernel.org
> Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when
> vCPU is blocked
> 
> Hi Feng.
> 
> On Fri, Sep 18, 2015 at 7:29 AM, Feng Wu <feng...@intel.com> wrote:
> > This patch updates the Posted-Interrupts Descriptor when vCPU
> > is blocked.
> >
> > pre-block:
> > - Add the vCPU to the blocked per-CPU list
> > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR
> >
> > post-block:
> > - Remove the vCPU from the per-CPU list
> 
> I'm wondering what happens if a posted interrupt arrives at the IOMMU
> after pre-block and before post-block.
> 
> In pre_block, NV is set to POSTED_INTR_WAKEUP_VECTOR. IIUC, this means
> future posted interrupts will not trigger "Posted-Interrupt Processing"
> (PIR will not get copied to VIRR). Instead, the IOMMU will do ON := 1,
> PIR |= (1 << vector), and send POSTED_INTR_WAKEUP_VECTOR. PIWV calls
> wakeup_handler which does kvm_vcpu_kick. kvm_vcpu_kick does a wait-queue
> wakeup and possibly a scheduler ipi.
> 
> But the VCPU is sitting in kvm_vcpu_block. It spins and/or schedules
> (wait queue) until it has a reason to wake up. I couldn't find a code
> path from kvm_vcpu_block that lead to checking ON or PIR. How does the
> blocked VCPU "receive" the posted interrupt? (And when does Posted-
> Interrupt Processing get triggered?)

In the pre_block, it also change the 'NDST' filed to the pCPU, on which the vCPU
is put to the per-CPU list 'blocked_vcpu_on_cpu', so when posted-interrupts
come it, it will sent the wakeup notification event to the pCPU above, then in
the wakeup_handler, it can find the vCPU from the per-CPU list, hence
kvm_vcpu_kick can wake up it.

Thanks,
Feng

> 
> Thanks!
> 
> >
> > Signed-off-by: Feng Wu <feng...@intel.com>
> > ---
> > v9:
> > - Add description for blocked_vcpu_on_cpu_lock in
> Documentation/virtual/kvm/locking.txt
> > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then
> >   !irq_remapping_cap(IRQ_POSTING_CAP)
> >
> > v8:
> > - Rename 'pi_pre_block' to 'pre_block'
> > - Rename 'pi_post_block' to 'post_block'
> > - Change some comments
> > - Only add the vCPU to the blocking list when the VM has assigned devices.
> >
> >  Documentation/virtual/kvm/locking.txt |  12 +++
> >  arch/x86/include/asm/kvm_host.h   |  13 +++
> >  arch/x86/kvm/vmx.c| 153
> ++
> >  arch/x86/kvm/x86.c|  53 +---
> >  include/linux/kvm_host.h  |   3 +
> >  virt/kvm/kvm_main.c   |   3 +
> >  6 files changed, 227 insertions(+), 10 deletions(-)
> >
> > diff --git a/Documentation/virtual/kvm/locking.txt
> b/Documentation/virtual/kvm/locking.txt
> > index d68af4d..19f94a6 100644
> > --- a/Documentation/virtual/kvm/locking.txt
> > +++ b/Documentation/virtual/kvm/locking.txt
> > @@ -166,3 +166,15 @@ Comment:   The srcu read lock must be held while
> accessing memslots (e.g.
> > MMIO/PIO address->device structure mapping (kvm->buses).
> > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu
> > if it is needed by multiple functions.
> > +
> > +Name:  blocked_vcpu_on_cpu_lock
> > +Type:  spinlock_t
> > +Arch:  x86
> > +Protects:  blocked_vcpu_on_cpu
> > +Comment:   This is a per-CPU lock and it is used for VT-d 
> > posted-interrupts.
> > +   When VT-d posted-interrupts is supported and the VM has 
> > assigned
> > +   devices, we put the blocked vCPU on the list 
> > blocked_vcpu_on_cpu
> > +   protected by blocked_vcpu_on_cpu_lock, when VT-d hardware
> issues
> > +   wakeup notification event since external interrupts from the
> > +   assigned devices happens, we will find the vCPU on the list 
> > to
> > +   wakeup.
> > diff --git a/arch/x86/include/asm/kvm_host.h
> b/arch/x86/include/asm/kvm_host.h
> > index 0ddd353..304fbb5 100644
> > --- a/arch/x86/include/asm/kvm_host.h
> > +++ b/arch/x86/include/asm/kvm_host.h
> > @@ -552

Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-10-14 Thread David Matlack 
Hi Feng.

On Fri, Sep 18, 2015 at 7:29 AM, Feng Wu  wrote:
> This patch updates the Posted-Interrupts Descriptor when vCPU
> is blocked.
>
> pre-block:
> - Add the vCPU to the blocked per-CPU list
> - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR
>
> post-block:
> - Remove the vCPU from the per-CPU list

I'm wondering what happens if a posted interrupt arrives at the IOMMU
after pre-block and before post-block.

In pre_block, NV is set to POSTED_INTR_WAKEUP_VECTOR. IIUC, this means
future posted interrupts will not trigger "Posted-Interrupt Processing"
(PIR will not get copied to VIRR). Instead, the IOMMU will do ON := 1,
PIR |= (1 << vector), and send POSTED_INTR_WAKEUP_VECTOR. PIWV calls
wakeup_handler which does kvm_vcpu_kick. kvm_vcpu_kick does a wait-queue
wakeup and possibly a scheduler ipi.

But the VCPU is sitting in kvm_vcpu_block. It spins and/or schedules
(wait queue) until it has a reason to wake up. I couldn't find a code
path from kvm_vcpu_block that lead to checking ON or PIR. How does the
blocked VCPU "receive" the posted interrupt? (And when does Posted-
Interrupt Processing get triggered?)

Thanks!

>
> Signed-off-by: Feng Wu 
> ---
> v9:
> - Add description for blocked_vcpu_on_cpu_lock in 
> Documentation/virtual/kvm/locking.txt
> - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then
>   !irq_remapping_cap(IRQ_POSTING_CAP)
>
> v8:
> - Rename 'pi_pre_block' to 'pre_block'
> - Rename 'pi_post_block' to 'post_block'
> - Change some comments
> - Only add the vCPU to the blocking list when the VM has assigned devices.
>
>  Documentation/virtual/kvm/locking.txt |  12 +++
>  arch/x86/include/asm/kvm_host.h   |  13 +++
>  arch/x86/kvm/vmx.c| 153 
> ++
>  arch/x86/kvm/x86.c|  53 +---
>  include/linux/kvm_host.h  |   3 +
>  virt/kvm/kvm_main.c   |   3 +
>  6 files changed, 227 insertions(+), 10 deletions(-)
>
> diff --git a/Documentation/virtual/kvm/locking.txt 
> b/Documentation/virtual/kvm/locking.txt
> index d68af4d..19f94a6 100644
> --- a/Documentation/virtual/kvm/locking.txt
> +++ b/Documentation/virtual/kvm/locking.txt
> @@ -166,3 +166,15 @@ Comment:   The srcu read lock must be held while 
> accessing memslots (e.g.
> MMIO/PIO address->device structure mapping (kvm->buses).
> The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu
> if it is needed by multiple functions.
> +
> +Name:  blocked_vcpu_on_cpu_lock
> +Type:  spinlock_t
> +Arch:  x86
> +Protects:  blocked_vcpu_on_cpu
> +Comment:   This is a per-CPU lock and it is used for VT-d 
> posted-interrupts.
> +   When VT-d posted-interrupts is supported and the VM has 
> assigned
> +   devices, we put the blocked vCPU on the list 
> blocked_vcpu_on_cpu
> +   protected by blocked_vcpu_on_cpu_lock, when VT-d hardware 
> issues
> +   wakeup notification event since external interrupts from the
> +   assigned devices happens, we will find the vCPU on the list to
> +   wakeup.
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 0ddd353..304fbb5 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -552,6 +552,8 @@ struct kvm_vcpu_arch {
>  */
> bool write_fault_to_shadow_pgtable;
>
> +   bool halted;
> +
> /* set at EPT violation at this point */
> unsigned long exit_qualification;
>
> @@ -864,6 +866,17 @@ struct kvm_x86_ops {
> /* pmu operations of sub-arch */
> const struct kvm_pmu_ops *pmu_ops;
>
> +   /*
> +* Architecture specific hooks for vCPU blocking due to
> +* HLT instruction.
> +* Returns for .pre_block():
> +*- 0 means continue to block the vCPU.
> +*- 1 means we cannot block the vCPU since some event
> +*happens during this period, such as, 'ON' bit in
> +*posted-interrupts descriptor is set.
> +*/
> +   int (*pre_block)(struct kvm_vcpu *vcpu);
> +   void (*post_block)(struct kvm_vcpu *vcpu);
> int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq,
>   uint32_t guest_irq, bool set);
>  };
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index 902a67d..9968896 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -879,6 +879,13 @@ static DEFINE_PER_CPU(struct vmcs *, current_vmcs);
>  static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
>  static DEFINE_PER_CPU(struct desc_ptr, host_gdt);
>
> +/*
> + * We maintian a per-CPU linked-list of vCPU, so in wakeup_handler() we
> + * can find which vCPU should be waken up.
> + */
> +static DEFINE_PER_CPU(struct list_head, blocked_vcpu_on_cpu);
> +static DEFINE_PER_CPU(spinlock_t,

RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-09-20 Thread Wu, Feng 


> -Original Message-
> From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo
> Bonzini
> Sent: Saturday, September 19, 2015 12:07 AM
> To: Wu, Feng; Alex Williamson; j...@8bytes.org; Marcelo Tosatti
> Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list;
> Eric Auger
> Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when
> vCPU is blocked
> 
> 
> 
> On 18/09/2015 16:29, Feng Wu wrote:
> > This patch updates the Posted-Interrupts Descriptor when vCPU
> > is blocked.
> >
> > pre-block:
> > - Add the vCPU to the blocked per-CPU list
> > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR
> >
> > post-block:
> > - Remove the vCPU from the per-CPU list
> >
> > Signed-off-by: Feng Wu
> <feng.wu-ral2jqcrhueavxtiumw...@public.gmane.org>
> > ---
> > v9:
> > - Add description for blocked_vcpu_on_cpu_lock in
> Documentation/virtual/kvm/locking.txt
> > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then
> >   !irq_remapping_cap(IRQ_POSTING_CAP)
> >
> > v8:
> > - Rename 'pi_pre_block' to 'pre_block'
> > - Rename 'pi_post_block' to 'post_block'
> > - Change some comments
> > - Only add the vCPU to the blocking list when the VM has assigned devices.
> >
> >  Documentation/virtual/kvm/locking.txt |  12 +++
> >  arch/x86/include/asm/kvm_host.h   |  13 +++
> >  arch/x86/kvm/vmx.c| 153
> ++
> >  arch/x86/kvm/x86.c|  53 +---
> >  include/linux/kvm_host.h  |   3 +
> >  virt/kvm/kvm_main.c   |   3 +
> >  6 files changed, 227 insertions(+), 10 deletions(-)
> >
> > diff --git a/Documentation/virtual/kvm/locking.txt
> b/Documentation/virtual/kvm/locking.txt
> > index d68af4d..19f94a6 100644
> > --- a/Documentation/virtual/kvm/locking.txt
> > +++ b/Documentation/virtual/kvm/locking.txt
> > @@ -166,3 +166,15 @@ Comment:   The srcu read lock must be held while
> accessing memslots (e.g.
> > MMIO/PIO address->device structure mapping (kvm->buses).
> > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu
> > if it is needed by multiple functions.
> > +
> > +Name:  blocked_vcpu_on_cpu_lock
> > +Type:  spinlock_t
> > +Arch:  x86
> > +Protects:  blocked_vcpu_on_cpu
> > +Comment:   This is a per-CPU lock and it is used for VT-d 
> > posted-interrupts.
> > +   When VT-d posted-interrupts is supported and the VM has assigned
> > +   devices, we put the blocked vCPU on the list blocked_vcpu_on_cpu
> > +   protected by blocked_vcpu_on_cpu_lock, when VT-d hardware issues
> > +   wakeup notification event since external interrupts from the
> > +   assigned devices happens, we will find the vCPU on the list to
> > +   wakeup.
> > diff --git a/arch/x86/include/asm/kvm_host.h
> b/arch/x86/include/asm/kvm_host.h
> > index 0ddd353..304fbb5 100644
> > --- a/arch/x86/include/asm/kvm_host.h
> > +++ b/arch/x86/include/asm/kvm_host.h
> > @@ -552,6 +552,8 @@ struct kvm_vcpu_arch {
> >  */
> > bool write_fault_to_shadow_pgtable;
> >
> > +   bool halted;
> > +
> > /* set at EPT violation at this point */
> > unsigned long exit_qualification;
> >
> > @@ -864,6 +866,17 @@ struct kvm_x86_ops {
> > /* pmu operations of sub-arch */
> > const struct kvm_pmu_ops *pmu_ops;
> >
> > +   /*
> > +* Architecture specific hooks for vCPU blocking due to
> > +* HLT instruction.
> > +* Returns for .pre_block():
> > +*- 0 means continue to block the vCPU.
> > +*- 1 means we cannot block the vCPU since some event
> > +*happens during this period, such as, 'ON' bit in
> > +*posted-interrupts descriptor is set.
> > +*/
> > +   int (*pre_block)(struct kvm_vcpu *vcpu);
> > +   void (*post_block)(struct kvm_vcpu *vcpu);
> > int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq,
> >   uint32_t guest_irq, bool set);
> >  };
> > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> > index 902a67d..9968896 100644
> > --- a/arch/x86/kvm/vmx.c
> > +++ b/arch/x86/kvm/vmx.c
> > @@ -879,6 +879,13 @@ static DEFINE_PER_CPU(struct vmcs *,
> current_vmcs);
> >  static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
> >  static DEFINE_PER_CPU(struct des

RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-09-20 Thread Wu, Feng 


> -Original Message-
> From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo
> Bonzini
> Sent: Monday, September 21, 2015 1:33 PM
> To: Wu, Feng; Alex Williamson; j...@8bytes.org; Marcelo Tosatti
> Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list;
> Eric Auger
> Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when
> vCPU is blocked
> 
> 
> 
> On 21/09/2015 04:16, Wu, Feng wrote:
> > I tested the above patch you suggested, it works fine. Thank you! So
> > do I need to resend a new version or you can handle it in your tree?
> 
> I will handle it.

Thanks a lot for your review on this series!

Thanks,
Feng

> 
> Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-09-20 Thread Paolo Bonzini 


On 21/09/2015 04:16, Wu, Feng wrote:
> I tested the above patch you suggested, it works fine. Thank you! So
> do I need to resend a new version or you can handle it in your tree?

I will handle it.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-09-19 Thread Wu, Feng 


> -Original Message-
> From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo
> Bonzini
> Sent: Saturday, September 19, 2015 12:07 AM
> To: Wu, Feng; Alex Williamson; j...@8bytes.org; Marcelo Tosatti
> Cc: io...@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list;
> Eric Auger
> Subject: Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when
> vCPU is blocked
> 
> 
> 
> On 18/09/2015 16:29, Feng Wu wrote:
> > This patch updates the Posted-Interrupts Descriptor when vCPU
> > is blocked.
> >
> > pre-block:
> > - Add the vCPU to the blocked per-CPU list
> > - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR
> >
> > post-block:
> > - Remove the vCPU from the per-CPU list
> >
> > Signed-off-by: Feng Wu
> <feng.wu-ral2jqcrhueavxtiumw...@public.gmane.org>
> > ---
> > v9:
> > - Add description for blocked_vcpu_on_cpu_lock in
> Documentation/virtual/kvm/locking.txt
> > - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then
> >   !irq_remapping_cap(IRQ_POSTING_CAP)
> >
> > v8:
> > - Rename 'pi_pre_block' to 'pre_block'
> > - Rename 'pi_post_block' to 'post_block'
> > - Change some comments
> > - Only add the vCPU to the blocking list when the VM has assigned devices.
> >
> >  Documentation/virtual/kvm/locking.txt |  12 +++
> >  arch/x86/include/asm/kvm_host.h   |  13 +++
> >  arch/x86/kvm/vmx.c| 153
> ++
> >  arch/x86/kvm/x86.c|  53 +---
> >  include/linux/kvm_host.h  |   3 +
> >  virt/kvm/kvm_main.c   |   3 +
> >  6 files changed, 227 insertions(+), 10 deletions(-)
> >
> > diff --git a/Documentation/virtual/kvm/locking.txt
> b/Documentation/virtual/kvm/locking.txt
> > index d68af4d..19f94a6 100644
> > --- a/Documentation/virtual/kvm/locking.txt
> > +++ b/Documentation/virtual/kvm/locking.txt
> > @@ -166,3 +166,15 @@ Comment:   The srcu read lock must be held while
> accessing memslots (e.g.
> > MMIO/PIO address->device structure mapping (kvm->buses).
> > The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu
> > if it is needed by multiple functions.
> > +
> > +Name:  blocked_vcpu_on_cpu_lock
> > +Type:  spinlock_t
> > +Arch:  x86
> > +Protects:  blocked_vcpu_on_cpu
> > +Comment:   This is a per-CPU lock and it is used for VT-d 
> > posted-interrupts.
> > +   When VT-d posted-interrupts is supported and the VM has assigned
> > +   devices, we put the blocked vCPU on the list blocked_vcpu_on_cpu
> > +   protected by blocked_vcpu_on_cpu_lock, when VT-d hardware issues
> > +   wakeup notification event since external interrupts from the
> > +   assigned devices happens, we will find the vCPU on the list to
> > +   wakeup.
> > diff --git a/arch/x86/include/asm/kvm_host.h
> b/arch/x86/include/asm/kvm_host.h
> > index 0ddd353..304fbb5 100644
> > --- a/arch/x86/include/asm/kvm_host.h
> > +++ b/arch/x86/include/asm/kvm_host.h
> > @@ -552,6 +552,8 @@ struct kvm_vcpu_arch {
> >  */
> > bool write_fault_to_shadow_pgtable;
> >
> > +   bool halted;
> > +
> > /* set at EPT violation at this point */
> > unsigned long exit_qualification;
> >
> > @@ -864,6 +866,17 @@ struct kvm_x86_ops {
> > /* pmu operations of sub-arch */
> > const struct kvm_pmu_ops *pmu_ops;
> >
> > +   /*
> > +* Architecture specific hooks for vCPU blocking due to
> > +* HLT instruction.
> > +* Returns for .pre_block():
> > +*- 0 means continue to block the vCPU.
> > +*- 1 means we cannot block the vCPU since some event
> > +*happens during this period, such as, 'ON' bit in
> > +*posted-interrupts descriptor is set.
> > +*/
> > +   int (*pre_block)(struct kvm_vcpu *vcpu);
> > +   void (*post_block)(struct kvm_vcpu *vcpu);
> > int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq,
> >   uint32_t guest_irq, bool set);
> >  };
> > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> > index 902a67d..9968896 100644
> > --- a/arch/x86/kvm/vmx.c
> > +++ b/arch/x86/kvm/vmx.c
> > @@ -879,6 +879,13 @@ static DEFINE_PER_CPU(struct vmcs *,
> current_vmcs);
> >  static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
> >  static DEFINE_PER_CPU(struct des

[PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-09-18 Thread Feng Wu 
This patch updates the Posted-Interrupts Descriptor when vCPU
is blocked.

pre-block:
- Add the vCPU to the blocked per-CPU list
- Set 'NV' to POSTED_INTR_WAKEUP_VECTOR

post-block:
- Remove the vCPU from the per-CPU list

Signed-off-by: Feng Wu 
---
v9:
- Add description for blocked_vcpu_on_cpu_lock in 
Documentation/virtual/kvm/locking.txt
- Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then
  !irq_remapping_cap(IRQ_POSTING_CAP)

v8:
- Rename 'pi_pre_block' to 'pre_block'
- Rename 'pi_post_block' to 'post_block'
- Change some comments
- Only add the vCPU to the blocking list when the VM has assigned devices.

 Documentation/virtual/kvm/locking.txt |  12 +++
 arch/x86/include/asm/kvm_host.h   |  13 +++
 arch/x86/kvm/vmx.c| 153 ++
 arch/x86/kvm/x86.c|  53 +---
 include/linux/kvm_host.h  |   3 +
 virt/kvm/kvm_main.c   |   3 +
 6 files changed, 227 insertions(+), 10 deletions(-)

diff --git a/Documentation/virtual/kvm/locking.txt 
b/Documentation/virtual/kvm/locking.txt
index d68af4d..19f94a6 100644
--- a/Documentation/virtual/kvm/locking.txt
+++ b/Documentation/virtual/kvm/locking.txt
@@ -166,3 +166,15 @@ Comment:   The srcu read lock must be held while accessing 
memslots (e.g.
MMIO/PIO address->device structure mapping (kvm->buses).
The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu
if it is needed by multiple functions.
+
+Name:  blocked_vcpu_on_cpu_lock
+Type:  spinlock_t
+Arch:  x86
+Protects:  blocked_vcpu_on_cpu
+Comment:   This is a per-CPU lock and it is used for VT-d 
posted-interrupts.
+   When VT-d posted-interrupts is supported and the VM has assigned
+   devices, we put the blocked vCPU on the list blocked_vcpu_on_cpu
+   protected by blocked_vcpu_on_cpu_lock, when VT-d hardware issues
+   wakeup notification event since external interrupts from the
+   assigned devices happens, we will find the vCPU on the list to
+   wakeup.
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 0ddd353..304fbb5 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -552,6 +552,8 @@ struct kvm_vcpu_arch {
 */
bool write_fault_to_shadow_pgtable;
 
+   bool halted;
+
/* set at EPT violation at this point */
unsigned long exit_qualification;
 
@@ -864,6 +866,17 @@ struct kvm_x86_ops {
/* pmu operations of sub-arch */
const struct kvm_pmu_ops *pmu_ops;
 
+   /*
+* Architecture specific hooks for vCPU blocking due to
+* HLT instruction.
+* Returns for .pre_block():
+*- 0 means continue to block the vCPU.
+*- 1 means we cannot block the vCPU since some event
+*happens during this period, such as, 'ON' bit in
+*posted-interrupts descriptor is set.
+*/
+   int (*pre_block)(struct kvm_vcpu *vcpu);
+   void (*post_block)(struct kvm_vcpu *vcpu);
int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq,
  uint32_t guest_irq, bool set);
 };
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index 902a67d..9968896 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -879,6 +879,13 @@ static DEFINE_PER_CPU(struct vmcs *, current_vmcs);
 static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
 static DEFINE_PER_CPU(struct desc_ptr, host_gdt);
 
+/*
+ * We maintian a per-CPU linked-list of vCPU, so in wakeup_handler() we
+ * can find which vCPU should be waken up.
+ */
+static DEFINE_PER_CPU(struct list_head, blocked_vcpu_on_cpu);
+static DEFINE_PER_CPU(spinlock_t, blocked_vcpu_on_cpu_lock);
+
 static unsigned long *vmx_io_bitmap_a;
 static unsigned long *vmx_io_bitmap_b;
 static unsigned long *vmx_msr_bitmap_legacy;
@@ -2985,6 +2992,8 @@ static int hardware_enable(void)
return -EBUSY;
 
INIT_LIST_HEAD(_cpu(loaded_vmcss_on_cpu, cpu));
+   INIT_LIST_HEAD(_cpu(blocked_vcpu_on_cpu, cpu));
+   spin_lock_init(_cpu(blocked_vcpu_on_cpu_lock, cpu));
 
/*
 * Now we can enable the vmclear operation in kdump
@@ -6121,6 +6130,25 @@ static void update_ple_window_actual_max(void)
ple_window_grow, INT_MIN);
 }
 
+/*
+ * Handler for POSTED_INTERRUPT_WAKEUP_VECTOR.
+ */
+static void wakeup_handler(void)
+{
+   struct kvm_vcpu *vcpu;
+   int cpu = smp_processor_id();
+
+   spin_lock(_cpu(blocked_vcpu_on_cpu_lock, cpu));
+   list_for_each_entry(vcpu, _cpu(blocked_vcpu_on_cpu, cpu),
+   blocked_vcpu_list) {
+   struct pi_desc *pi_desc = vcpu_to_pi_desc(vcpu);
+
+   if (pi_test_on(pi_desc) == 1)
+

Re: [PATCH v9 17/18] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked

2015-09-18 Thread Paolo Bonzini 


On 18/09/2015 16:29, Feng Wu wrote:
> This patch updates the Posted-Interrupts Descriptor when vCPU
> is blocked.
> 
> pre-block:
> - Add the vCPU to the blocked per-CPU list
> - Set 'NV' to POSTED_INTR_WAKEUP_VECTOR
> 
> post-block:
> - Remove the vCPU from the per-CPU list
> 
> Signed-off-by: Feng Wu 
> ---
> v9:
> - Add description for blocked_vcpu_on_cpu_lock in 
> Documentation/virtual/kvm/locking.txt
> - Check !kvm_arch_has_assigned_device(vcpu->kvm) first, then
>   !irq_remapping_cap(IRQ_POSTING_CAP)
> 
> v8:
> - Rename 'pi_pre_block' to 'pre_block'
> - Rename 'pi_post_block' to 'post_block'
> - Change some comments
> - Only add the vCPU to the blocking list when the VM has assigned devices.
> 
>  Documentation/virtual/kvm/locking.txt |  12 +++
>  arch/x86/include/asm/kvm_host.h   |  13 +++
>  arch/x86/kvm/vmx.c| 153 
> ++
>  arch/x86/kvm/x86.c|  53 +---
>  include/linux/kvm_host.h  |   3 +
>  virt/kvm/kvm_main.c   |   3 +
>  6 files changed, 227 insertions(+), 10 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/locking.txt 
> b/Documentation/virtual/kvm/locking.txt
> index d68af4d..19f94a6 100644
> --- a/Documentation/virtual/kvm/locking.txt
> +++ b/Documentation/virtual/kvm/locking.txt
> @@ -166,3 +166,15 @@ Comment: The srcu read lock must be held while accessing 
> memslots (e.g.
>   MMIO/PIO address->device structure mapping (kvm->buses).
>   The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu
>   if it is needed by multiple functions.
> +
> +Name:blocked_vcpu_on_cpu_lock
> +Type:spinlock_t
> +Arch:x86
> +Protects:blocked_vcpu_on_cpu
> +Comment: This is a per-CPU lock and it is used for VT-d 
> posted-interrupts.
> + When VT-d posted-interrupts is supported and the VM has assigned
> + devices, we put the blocked vCPU on the list blocked_vcpu_on_cpu
> + protected by blocked_vcpu_on_cpu_lock, when VT-d hardware issues
> + wakeup notification event since external interrupts from the
> + assigned devices happens, we will find the vCPU on the list to
> + wakeup.
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 0ddd353..304fbb5 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -552,6 +552,8 @@ struct kvm_vcpu_arch {
>*/
>   bool write_fault_to_shadow_pgtable;
>  
> + bool halted;
> +
>   /* set at EPT violation at this point */
>   unsigned long exit_qualification;
>  
> @@ -864,6 +866,17 @@ struct kvm_x86_ops {
>   /* pmu operations of sub-arch */
>   const struct kvm_pmu_ops *pmu_ops;
>  
> + /*
> +  * Architecture specific hooks for vCPU blocking due to
> +  * HLT instruction.
> +  * Returns for .pre_block():
> +  *- 0 means continue to block the vCPU.
> +  *- 1 means we cannot block the vCPU since some event
> +  *happens during this period, such as, 'ON' bit in
> +  *posted-interrupts descriptor is set.
> +  */
> + int (*pre_block)(struct kvm_vcpu *vcpu);
> + void (*post_block)(struct kvm_vcpu *vcpu);
>   int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq,
> uint32_t guest_irq, bool set);
>  };
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index 902a67d..9968896 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -879,6 +879,13 @@ static DEFINE_PER_CPU(struct vmcs *, current_vmcs);
>  static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
>  static DEFINE_PER_CPU(struct desc_ptr, host_gdt);
>  
> +/*
> + * We maintian a per-CPU linked-list of vCPU, so in wakeup_handler() we
> + * can find which vCPU should be waken up.
> + */
> +static DEFINE_PER_CPU(struct list_head, blocked_vcpu_on_cpu);
> +static DEFINE_PER_CPU(spinlock_t, blocked_vcpu_on_cpu_lock);
> +
>  static unsigned long *vmx_io_bitmap_a;
>  static unsigned long *vmx_io_bitmap_b;
>  static unsigned long *vmx_msr_bitmap_legacy;
> @@ -2985,6 +2992,8 @@ static int hardware_enable(void)
>   return -EBUSY;
>  
>   INIT_LIST_HEAD(_cpu(loaded_vmcss_on_cpu, cpu));
> + INIT_LIST_HEAD(_cpu(blocked_vcpu_on_cpu, cpu));
> + spin_lock_init(_cpu(blocked_vcpu_on_cpu_lock, cpu));
>  
>   /*
>* Now we can enable the vmclear operation in kdump
> @@ -6121,6 +6130,25 @@ static void update_ple_window_actual_max(void)
>   ple_window_grow, INT_MIN);
>  }
>  
> +/*
> + * Handler for POSTED_INTERRUPT_WAKEUP_VECTOR.
> + */
> +static void wakeup_handler(void)
> +{
> + struct kvm_vcpu *vcpu;
> + int cpu = smp_processor_id();
> +
> + spin_lock(_cpu(blocked_vcpu_on_cpu_lock, cpu));
> +