Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-21 Thread Radim Kr?má? 
2015-01-20 23:04+0200, Nadav Amit:
 Radim Kr?má? rkrc...@redhat.com wrote:
  2015-01-14 01:27+, Wu, Feng:
  the new
  hardware even doesn't consider the TPR for lowest priority interrupts
  delivery.
  
  A bold move ... what hardware was the first to do so?
  
  I think it was starting with Nehalem.
  
  Thanks,  (Could be that QPI can't inform about TPR changes anymore ...)
  
  I played with Linux's TPR on Haswell and found that is has no effect.
 
 Sorry for jumping into the discussion, but doesn’t it depend on
 IA32_MISC_ENABLE[23]? This bit disables xTPR messages. On my machine it is
 set (probably by the BIOS), but since there is no IA32_MISC_ENABLE is not
 locked for changes, the OS can control it.

Thanks, I didn't know about it.
On Ivy Bridge EP (the only modern machine at hand), the bit was set by
default.  After clearing it, TPR still had no effect.

The most relevant mention of xTPR I found is related to FSB [1].
[2] isn't enlightening, so there might be more from QPI-era ...


---
1: Intel® E7320 Memory Controller Hub (MCH) Datasheet
   
http://www.intel.com/content/dam/doc/datasheet/e7320-memory-controller-hub-datasheet.pdf
   5.2.2 System Bus Interrupts
2: Intel® Xeon® Processor E5 v2 Family: Datasheet, Vol. 2
   
http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v2-datasheet-vol-2.pdf
   6.1.2 IntControl
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-20 Thread Nadav Amit 
Radim Kr?má? rkrc...@redhat.com wrote:

 2015-01-14 01:27+, Wu, Feng:
 the new
 hardware even doesn't consider the TPR for lowest priority interrupts
 delivery.
 
 A bold move ... what hardware was the first to do so?
 
 I think it was starting with Nehalem.
 
 Thanks,  (Could be that QPI can't inform about TPR changes anymore ...)
 
 I played with Linux's TPR on Haswell and found that is has no effect.

Sorry for jumping into the discussion, but doesn’t it depend on
IA32_MISC_ENABLE[23]? This bit disables xTPR messages. On my machine it is
set (probably by the BIOS), but since there is no IA32_MISC_ENABLE is not
locked for changes, the OS can control it.

Nadav
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-14 Thread Radim Kr?má? 
2015-01-14 01:27+, Wu, Feng:
  the new
   hardware even doesn't consider the TPR for lowest priority interrupts
  delivery.
  
  A bold move ... what hardware was the first to do so?
 
 I think it was starting with Nehalem.

Thanks,  (Could be that QPI can't inform about TPR changes anymore ...)

I played with Linux's TPR on Haswell and found that is has no effect.

   As discussed with Paolo before, I will submit a patch to support lowest
  priority for PI
   after this series is merged.
  
  Sure, I see only two good solutions though
   1) don't optimize lowest priority with PI
   2) don't balance lowest priority
 
 As discussed with Paolo before, as the first stage, we only support single-CPU
 lowest priority for PI, since this is a new hardware feature enabling, Paolo 
 trends
 to do simple things in the beginning.

I agree, that is the best we can do without changing lowest priority.

I wanted to avoid a future solution that would introduce two behaviors
for lowest priority (round robin and something).
Round robin (anything dynamic) can't be done with PI, hence the question
if we can remove it.

   Then we will support full lowest 
 priority for
 it, such as, using vector hashing (this is one method of what hardware do for
 lowest priority today), I need to get some detailed information about this 
 from
 hardware guys before enabling it.

I wasn't able to confirm hashing, is it a recent addition?
I'm not sure we want it then:  OS still has to take care of proper
distribution, being predictable is better than uncertain gains, and
we'll save code.

The same should apply to hardware though ... do you know the reasons
behind vector hashing?

Thank you.
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-14 Thread Paolo Bonzini 


On 14/01/2015 02:27, Wu, Feng wrote:
 As discussed with Paolo before, as the first stage, we only support single-CPU
 lowest priority for PI, since this is a new hardware feature enabling, Paolo 
 trends
 to do simple things in the beginning.

:)

Nice way to sum it up!

Paolo

 Then we will support full lowest priority for
 it, such as, using vector hashing (this is one method of what hardware do for
 lowest priority today), I need to get some detailed information about this 
 from
 hardware guys before enabling it.
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-13 Thread Wu, Feng 


 -Original Message-
 From: Radim Kr?má? [mailto:rkrc...@redhat.com]
 Sent: Wednesday, January 14, 2015 12:17 AM
 To: Wu, Feng
 Cc: Paolo Bonzini; t...@linutronix.de; mi...@redhat.com; h...@zytor.com;
 x...@kernel.org; g...@kernel.org; dw...@infradead.org; j...@8bytes.org;
 alex.william...@redhat.com; jiang@linux.intel.com; eric.au...@linaro.org;
 linux-ker...@vger.kernel.org; iommu@lists.linux-foundation.org;
 k...@vger.kernel.org
 Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
 VT-d PI
 
 2015-01-13 00:27+, Wu, Feng:
   On 09/01/2015 15:54, Radim Krčmář wrote:
There are two points relevant to this patch in new KVM's implementation,
(KVM: x86: amend APIC lowest priority arbitration,
 https://lkml.org/lkml/2015/1/9/362)
   
1) lowest priority depends on TPR
2) there is no need for balancing
   
(1) has to be considered with PI as well.
  
   The chipset doesn't support it. :(
  
I kept (2) to avoid whining from people building on that behaviour, but
lowest priority backed by PI could be transparent without it.
   
Patch below removes the balancing, but I am not sure this is a price we
allowed ourselves to pay ... what are your opinions?
  
   I wouldn't mind, but it requires a lot of benchmarking.
 
  In fact, the real hardware may do lowest priority in round robin way,
 
 Yes, but we won't emulate round robin with PI and I think it is wrong to
 have backends with significantly different guest-visible behaviors.
 
 
 the new
  hardware even doesn't consider the TPR for lowest priority interrupts
 delivery.
 
 A bold move ... what hardware was the first to do so?

I think it was starting with Nehalem.

 
  As discussed with Paolo before, I will submit a patch to support lowest
 priority for PI
  after this series is merged.
 
 Sure, I see only two good solutions though
  1) don't optimize lowest priority with PI
  2) don't balance lowest priority

As discussed with Paolo before, as the first stage, we only support single-CPU
lowest priority for PI, since this is a new hardware feature enabling, Paolo 
trends
to do simple things in the beginning. Then we will support full lowest priority 
for
it, such as, using vector hashing (this is one method of what hardware do for
lowest priority today), I need to get some detailed information about this from
hardware guys before enabling it.

Thanks,
Feng
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-13 Thread Radim Kr?má? 
2015-01-13 00:27+, Wu, Feng:
  On 09/01/2015 15:54, Radim Krčmář wrote:
   There are two points relevant to this patch in new KVM's implementation,
   (KVM: x86: amend APIC lowest priority arbitration,
https://lkml.org/lkml/2015/1/9/362)
  
   1) lowest priority depends on TPR
   2) there is no need for balancing
  
   (1) has to be considered with PI as well.
  
  The chipset doesn't support it. :(
  
   I kept (2) to avoid whining from people building on that behaviour, but
   lowest priority backed by PI could be transparent without it.
  
   Patch below removes the balancing, but I am not sure this is a price we
   allowed ourselves to pay ... what are your opinions?
  
  I wouldn't mind, but it requires a lot of benchmarking.
 
 In fact, the real hardware may do lowest priority in round robin way,

Yes, but we won't emulate round robin with PI and I think it is wrong to
have backends with significantly different guest-visible behaviors.

   the new
 hardware even doesn't consider the TPR for lowest priority interrupts 
 delivery.

A bold move ... what hardware was the first to do so?

 As discussed with Paolo before, I will submit a patch to support lowest 
 priority for PI
 after this series is merged.

Sure, I see only two good solutions though
 1) don't optimize lowest priority with PI
 2) don't balance lowest priority
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-12 Thread Wu, Feng 


 -Original Message-
 From: Paolo Bonzini [mailto:pbonz...@redhat.com]
 Sent: Friday, January 09, 2015 10:56 PM
 To: Radim Krčmář; Wu, Feng
 Cc: t...@linutronix.de; mi...@redhat.com; h...@zytor.com; x...@kernel.org;
 g...@kernel.org; dw...@infradead.org; j...@8bytes.org;
 alex.william...@redhat.com; jiang@linux.intel.com; eric.au...@linaro.org;
 linux-ker...@vger.kernel.org; iommu@lists.linux-foundation.org;
 k...@vger.kernel.org
 Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
 VT-d PI
 
 
 
 On 09/01/2015 15:54, Radim Krčmář wrote:
  There are two points relevant to this patch in new KVM's implementation,
  (KVM: x86: amend APIC lowest priority arbitration,
   https://lkml.org/lkml/2015/1/9/362)
 
  1) lowest priority depends on TPR
  2) there is no need for balancing
 
  (1) has to be considered with PI as well.
 
 The chipset doesn't support it. :(
 
  I kept (2) to avoid whining from people building on that behaviour, but
  lowest priority backed by PI could be transparent without it.
 
  Patch below removes the balancing, but I am not sure this is a price we
  allowed ourselves to pay ... what are your opinions?
 
 I wouldn't mind, but it requires a lot of benchmarking.

In fact, the real hardware may do lowest priority in round robin way, the new
hardware even doesn't consider the TPR for lowest priority interrupts delivery.

As discussed with Paolo before, I will submit a patch to support lowest 
priority for PI
after this series is merged.

Thanks,
Feng

 
 Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-09 Thread Paolo Bonzini 


On 09/01/2015 16:12, Radim Krčmář wrote:
  The chipset doesn't support it. :(
 
 I meant that we need to recompute PI entries for lowest priority
 interrupts every time guest's TPR changes.
 
 Luckily, Linux doesn't use TPR, but other OS might be a reason to drop
 lowest priority from PI optimizations.  (Or make it more complicated.)

Doing vector hashing is a possibility as well.  I would like to know
what existing chipsets do in practice, then we can mimic it.

Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-09 Thread Radim Krčmář 
2015-01-09 16:18+0100, Paolo Bonzini:
 On 09/01/2015 16:12, Radim Krčmář wrote:
   The chipset doesn't support it. :(
  
  I meant that we need to recompute PI entries for lowest priority
  interrupts every time guest's TPR changes.
  
  Luckily, Linux doesn't use TPR, but other OS might be a reason to drop
  lowest priority from PI optimizations.  (Or make it more complicated.)
 
 Doing vector hashing is a possibility as well.  I would like to know
 what existing chipsets do in practice, then we can mimic it.

When looking at /proc/interrupts from time to time, I have only seen
interrupts landing on the first CPU of the set.

We could also distinguish between AMD and Intel ...
AMD should deliver to the highest APIC ID.
(If we still need to decide after focus processor and APR checks.)

I'll try to check using a more trustworthy approach.
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-09 Thread Radim Krčmář 
2014-12-12 23:14+0800, Feng Wu:
 This patch defines a new interface kvm_find_dest_vcpu for
 VT-d PI, which can returns the destination vCPU of the
 interrupt for guests.
 
 Since VT-d PI cannot handle broadcast/multicast interrupt,
 Here we only handle Fixed and Lowest priority interrupts.
 
 The current method of handling guest lowest priority interrtups
 is to use a counter 'apic_arb_prio' for each vCPU, we choose the
 vCPU with smallest 'apic_arb_prio' and then increase it by 1.
 However, for VT-d PI, we cannot re-use this, since we no longer
 have control to 'apic_arb_prio' with posted interrupt direct
 delivery by Hardware.
 
 Here, we introduce a similar way with 'apic_arb_prio' to handle
 guest lowest priority interrtups when VT-d PI is used. Here is the
 ideas:
 - Each vCPU has a counter 'round_robin_counter'.
 - When guests sets an interrupts to lowest priority, we choose
 the vCPU with smallest 'round_robin_counter' as the destination,
 then increase it.

There are two points relevant to this patch in new KVM's implementation,
(KVM: x86: amend APIC lowest priority arbitration,
 https://lkml.org/lkml/2015/1/9/362)

1) lowest priority depends on TPR
2) there is no need for balancing

(1) has to be considered with PI as well.
I kept (2) to avoid whining from people building on that behaviour, but
lowest priority backed by PI could be transparent without it.

Patch below removes the balancing, but I am not sure this is a price we
allowed ourselves to pay ... what are your opinions?


---8---
KVM: x86: don't balance lowest priority interrupts

Balancing is not mandated by specification and real hardware most likely
doesn't do it.  We break backward compatibility to allow optimizations.
(Posted interrupts can deliver to only one fixed destination.)

Signed-off-by: Radim Krčmář rkrc...@redhat.com
---
 arch/x86/include/asm/kvm_host.h | 1 -
 arch/x86/kvm/lapic.c| 8 ++--
 2 files changed, 2 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 97a5dd0222c8..aa4bd8286232 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -370,7 +370,6 @@ struct kvm_vcpu_arch {
u64 apic_base;
struct kvm_lapic *apic;/* kernel irqchip context */
unsigned long apic_attention;
-   int32_t apic_arb_prio;
int mp_state;
u64 ia32_misc_enable_msr;
bool tpr_access_reporting;
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 5b9d8c589bba..eb85af8e8fc0 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -749,7 +749,6 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int 
delivery_mode,
  trig_mode, vector);
switch (delivery_mode) {
case APIC_DM_LOWEST:
-   vcpu-arch.apic_arb_prio++;
case APIC_DM_FIXED:
/* FIXME add logic for vcpu on reset */
if (unlikely(!apic_enabled(apic)))
@@ -837,11 +836,9 @@ int kvm_apic_compare_prio(struct kvm_vcpu *vcpu1, struct 
kvm_vcpu *vcpu2)
 *  - uses the APR register (which also considers ISR and IRR),
 *  - chooses the highest APIC ID when APRs are identical,
 *  - and allows a focus processor.
-* XXX: pseudo-balancing with apic_arb_prio is a KVM-specific feature
 */
-   int tpr = kvm_apic_get_reg(vcpu1-arch.apic, APIC_TASKPRI) -
- kvm_apic_get_reg(vcpu2-arch.apic, APIC_TASKPRI);
-   return tpr ? : vcpu1-arch.apic_arb_prio - vcpu2-arch.apic_arb_prio;
+   return kvm_apic_get_reg(vcpu1-arch.apic, APIC_TASKPRI) -
+  kvm_apic_get_reg(vcpu2-arch.apic, APIC_TASKPRI);
 }
 
 static void kvm_ioapic_send_eoi(struct kvm_lapic *apic, int vector)
@@ -1595,7 +1592,6 @@ void kvm_lapic_reset(struct kvm_vcpu *vcpu)
vcpu-arch.pv_eoi.msr_val = 0;
apic_update_ppr(apic);
 
-   vcpu-arch.apic_arb_prio = 0;
vcpu-arch.apic_attention = 0;
 
apic_debug(%s: vcpu=%p, id=%d, base_msr=
-- 
2.2.0

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2015-01-09 Thread Paolo Bonzini 


On 09/01/2015 15:54, Radim Krčmář wrote:
 There are two points relevant to this patch in new KVM's implementation,
 (KVM: x86: amend APIC lowest priority arbitration,
  https://lkml.org/lkml/2015/1/9/362)
 
 1) lowest priority depends on TPR
 2) there is no need for balancing
 
 (1) has to be considered with PI as well.

The chipset doesn't support it. :(

 I kept (2) to avoid whining from people building on that behaviour, but
 lowest priority backed by PI could be transparent without it.
 
 Patch below removes the balancing, but I am not sure this is a price we
 allowed ourselves to pay ... what are your opinions?

I wouldn't mind, but it requires a lot of benchmarking.

Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-22 Thread Paolo Bonzini 


On 22/12/2014 05:48, Wu, Feng wrote:
 Do you mean we don't support Lowest priority interrupts? As I mentioned 
 before,
 Lowest priority interrupts is widely used in Linux, so I think supporting 
 lowest priority
 interrupts is very important for Linux guest OS. Do you have any 
 ideas/suggestions about
 how to support Lowest priority interrupts for PI? Thanks a lot!

Can you support them only if the destination is a single CPU?

Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-22 Thread Wu, Feng 


 -Original Message-
 From: Paolo Bonzini [mailto:pbonz...@redhat.com]
 Sent: Monday, December 22, 2014 5:28 PM
 To: Wu, Feng; Zhang, Yang Z; Thomas Gleixner; Ingo Molnar; H. Peter Anvin;
 x...@kernel.org; Gleb Natapov; dw...@infradead.org; j...@8bytes.org;
 Alex Williamson; Jiang Liu
 Cc: iommu@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list;
 Eric Auger
 Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
 VT-d PI
 
 
 
 On 22/12/2014 05:48, Wu, Feng wrote:
  Do you mean we don't support Lowest priority interrupts? As I mentioned
 before,
  Lowest priority interrupts is widely used in Linux, so I think supporting 
  lowest
 priority
  interrupts is very important for Linux guest OS. Do you have any
 ideas/suggestions about
  how to support Lowest priority interrupts for PI? Thanks a lot!
 
 Can you support them only if the destination is a single CPU?

Sorry, I am not quite understand this. I still don't understand the single 
CPU here.
Lowest priority interrupts always have a cpumask which contains multiple CPU.

Thanks,
Feng

 
 Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-22 Thread Paolo Bonzini 


On 22/12/2014 12:04, Wu, Feng wrote:
  Can you support them only if the destination is a single CPU?

 Sorry, I am not quite understand this. I still don't understand the single 
 CPU here.
 Lowest priority interrupts always have a cpumask which contains multiple CPU.

Yes, and those need not be accelerated.  But what if you set affinity to
a single CPU?

Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-22 Thread Wu, Feng 


 -Original Message-
 From: Paolo Bonzini [mailto:pbonz...@redhat.com]
 Sent: Monday, December 22, 2014 7:07 PM
 To: Wu, Feng; Zhang, Yang Z; Thomas Gleixner; Ingo Molnar; H. Peter Anvin;
 x...@kernel.org; Gleb Natapov; dw...@infradead.org; j...@8bytes.org;
 Alex Williamson; Jiang Liu
 Cc: iommu@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list;
 Eric Auger
 Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
 VT-d PI
 
 
 
 On 22/12/2014 12:04, Wu, Feng wrote:
   Can you support them only if the destination is a single CPU?
 
  Sorry, I am not quite understand this. I still don't understand the single 
  CPU
 here.
  Lowest priority interrupts always have a cpumask which contains multiple
 CPU.
 
 Yes, and those need not be accelerated.  But what if you set affinity to
 a single CPU?

How do I set affinity to a single CPU if guest configure a lowest priority 
interrupt? Thanks a lot!

Thanks,
Feng

 
 Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-22 Thread Paolo Bonzini 
 The problem is we still need to support PI with lowest priority delivery mode 
 even if guest does not configure irq affinity via /proc/irq/. Don't we?

Yes, but we can get the basic support working first.

I and Feng talked on irc and agreed to start with a simple implementation. Then 
he can add support for vector hashing for all interrupts, both vt-d pi and 
normal LAPIC interrupts.

Paolo 


___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-22 Thread Yong Wang 
On Mon, Dec 22, 2014 at 12:23:36PM +0100, Paolo Bonzini wrote:
 
 
 On 22/12/2014 12:17, Wu, Feng wrote:
  Yes, and those need not be accelerated.  But what if you set
  affinity to a single CPU?
 
  How do I set affinity to a single CPU if guest configure a lowest
  priority interrupt? Thanks a lot!
 
 I mean if the guest (via irqbalance and /proc/irq/) configures affinity
 to a single vCPU.  In that case, you can use PI.
 

The problem is we still need to support PI with lowest priority delivery mode
even if guest does not configure irq affinity via /proc/irq/. Don't we?

Thanks
-Yong

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-22 Thread Zhang, Yang Z 
Paolo Bonzini wrote on 2014-12-23:
 The problem is we still need to support PI with lowest priority
 delivery mode
 even if guest does not configure irq affinity via /proc/irq/. Don't we?
 
 Yes, but we can get the basic support working first.
 
 I and Feng talked on irc and agreed to start with a simple
 implementation. Then he can add support for vector hashing for all
 interrupts, both vt-d pi and normal LAPIC interrupts.

Agree. Obviously, current PI has some limitations to achieve highest 
performance. We can start with a simple implementation but must ensure we don't 
change hardware's behavior(From guest's point). Feng's current implementation 
or use the way I suggested are both ok since both of them cannot solve the 
problem well.

 
 Paolo



Best regards,
Yang


___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-21 Thread Wu, Feng 


 -Original Message-
 From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo
 Bonzini
 Sent: Saturday, December 20, 2014 9:17 PM
 To: Wu, Feng; Paolo Bonzini; Zhang, Yang Z; Thomas Gleixner; Ingo Molnar; H.
 Peter Anvin; x...@kernel.org; Gleb Natapov; dw...@infradead.org;
 j...@8bytes.org; Alex Williamson; Jiang Liu
 Cc: iommu@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list;
 Eric Auger
 Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
 VT-d PI
 
 
 
 On 20/12/2014 00:48, Wu, Feng wrote:
  In my understanding, lowest priority interrupts are always delivered to a
  Single CPU, we need to find the right destination CPU from the cpumask.
 
 Yes, but which CPU however differs every time the interrupt is
 delivered.  So the emulation here is a bit poor.  For now, please limit
 PI to fixed interrupts.

Do you mean we don't support Lowest priority interrupts? As I mentioned before,
Lowest priority interrupts is widely used in Linux, so I think supporting 
lowest priority
interrupts is very important for Linux guest OS. Do you have any 
ideas/suggestions about
how to support Lowest priority interrupts for PI? Thanks a lot!

Thanks,
Feng

 
  Actually, we don't support posting broadcast/multicast interrupts, because
  the interrupt is associated with one Posted-interrupts descriptor, then one
  vCPU.
 
 Understood.
 
 Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-20 Thread Paolo Bonzini 


On 20/12/2014 00:48, Wu, Feng wrote:
 In my understanding, lowest priority interrupts are always delivered to a
 Single CPU, we need to find the right destination CPU from the cpumask.

Yes, but which CPU however differs every time the interrupt is
delivered.  So the emulation here is a bit poor.  For now, please limit
PI to fixed interrupts.

 Actually, we don't support posting broadcast/multicast interrupts, because
 the interrupt is associated with one Posted-interrupts descriptor, then one
 vCPU.

Understood.

Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-19 Thread Paolo Bonzini 


On 19/12/2014 02:30, Wu, Feng wrote:
 How this can work well? All subsequent interrupts are delivered to
 one vCPU? It shouldn't be the best solution, need more consideration.

 Well, it's a hardware limitation.  The alternative (which is easy to
 implement) is to only do PI for single-CPU interrupts.  This should work
 well for multiqueue NICs (and of course for UP guests :)), so perhaps
 it's a good idea to only support that as a first attempt.

 Paolo
 
 Paolo, what do you mean by single-CPU interrupts? Do you mean we don't
 support lowest priority interrupts for PI? But Linux OS uses lowest priority
 for most of the case? If so, we can hardly get benefit from this feature for
 Linux guest OS.

You can post lowest priority interrupts if they are delivered to a
single CPU, in which case they are effectively fixed priority.

If they are broadcast to multiple CPUs, do not post them.

Paolo
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-19 Thread Wu, Feng 


 -Original Message-
 From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo
 Bonzini
 Sent: Friday, December 19, 2014 7:59 PM
 To: Wu, Feng; Paolo Bonzini; Zhang, Yang Z; Thomas Gleixner; Ingo Molnar; H.
 Peter Anvin; x...@kernel.org; Gleb Natapov; dw...@infradead.org;
 j...@8bytes.org; Alex Williamson; Jiang Liu
 Cc: iommu@lists.linux-foundation.org; linux-ker...@vger.kernel.org; KVM list;
 Eric Auger
 Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
 VT-d PI
 
 
 
 On 19/12/2014 02:30, Wu, Feng wrote:
  How this can work well? All subsequent interrupts are delivered to
  one vCPU? It shouldn't be the best solution, need more consideration.
 
  Well, it's a hardware limitation.  The alternative (which is easy to
  implement) is to only do PI for single-CPU interrupts.  This should work
  well for multiqueue NICs (and of course for UP guests :)), so perhaps
  it's a good idea to only support that as a first attempt.
 
  Paolo
 
  Paolo, what do you mean by single-CPU interrupts? Do you mean we don't
  support lowest priority interrupts for PI? But Linux OS uses lowest priority
  for most of the case? If so, we can hardly get benefit from this feature for
  Linux guest OS.
 
 You can post lowest priority interrupts if they are delivered to a
 single CPU, in which case they are effectively fixed priority.
 
 If they are broadcast to multiple CPUs, do not post them.
 
 Paolo

In my understanding, lowest priority interrupts are always delivered to a
Single CPU, we need to find the right destination CPU from the cpumask.
This is what I do in this patch. Did I misunderstanding something in your
Comments? Thanks a lot!

Actually, we don't support posting broadcast/multicast interrupts, because
the interrupt is associated with one Posted-interrupts descriptor, then one
vCPU.

Thanks,
Feng

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-18 Thread Zhang, Yang Z 
Feng Wu wrote on 2014-12-12:
 This patch defines a new interface kvm_find_dest_vcpu for
 VT-d PI, which can returns the destination vCPU of the
 interrupt for guests.
 
 Since VT-d PI cannot handle broadcast/multicast interrupt,
 Here we only handle Fixed and Lowest priority interrupts.
 
 The current method of handling guest lowest priority interrtups
 is to use a counter 'apic_arb_prio' for each vCPU, we choose the
 vCPU with smallest 'apic_arb_prio' and then increase it by 1.
 However, for VT-d PI, we cannot re-use this, since we no longer
 have control to 'apic_arb_prio' with posted interrupt direct
 delivery by Hardware.
 
 Here, we introduce a similar way with 'apic_arb_prio' to handle guest
 lowest priority interrtups when VT-d PI is used. Here is the ideas: -
 Each vCPU has a counter 'round_robin_counter'. - When guests sets an
 interrupts to lowest priority, we choose the vCPU with smallest
 'round_robin_counter' as the destination, then increase it.
 
How this can work well? All subsequent interrupts are delivered to one vCPU? It 
shouldn't be the best solution, need more consideration. Also, I think you 
should take the apic_arb_prio into consider since the priority is for the whole 
vCPU not for one interrupt.

Best regards,
Yang


___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-18 Thread Wu, Feng 


 -Original Message-
 From: Paolo Bonzini [mailto:pbonz...@redhat.com]
 Sent: Friday, December 19, 2014 12:58 AM
 To: Zhang, Yang Z; Wu, Feng; t...@linutronix.de; mi...@redhat.com;
 h...@zytor.com; x...@kernel.org; g...@kernel.org; dw...@infradead.org;
 j...@8bytes.org; alex.william...@redhat.com; jiang@linux.intel.com
 Cc: eric.au...@linaro.org; linux-ker...@vger.kernel.org;
 iommu@lists.linux-foundation.org; k...@vger.kernel.org
 Subject: Re: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
 VT-d PI
 
 
 
 On 18/12/2014 15:49, Zhang, Yang Z wrote:
  Here, we introduce a similar way with 'apic_arb_prio' to handle
  guest lowest priority interrtups when VT-d PI is used. Here is
  the ideas: - Each vCPU has a counter 'round_robin_counter'. -
  When guests sets an interrupts to lowest priority, we choose the
  vCPU with smallest 'round_robin_counter' as the destination, then
  increase it.
 
  How this can work well? All subsequent interrupts are delivered to
  one vCPU? It shouldn't be the best solution, need more consideration.
 
 Well, it's a hardware limitation.  The alternative (which is easy to
 implement) is to only do PI for single-CPU interrupts.  This should work
 well for multiqueue NICs (and of course for UP guests :)), so perhaps
 it's a good idea to only support that as a first attempt.
 
 Paolo

Paolo, what do you mean by single-CPU interrupts? Do you mean we don't
support lowest priority interrupts for PI? But Linux OS uses lowest priority
for most of the case? If so, we can hardly get benefit from this feature for
Linux guest OS.

Thanks,
Feng

 
  Also, I think you should take the apic_arb_prio into consider since
  the priority is for the whole vCPU not for one interrupt.
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI

2014-12-18 Thread Wu, Feng 


 -Original Message-
 From: Zhang, Yang Z
 Sent: Friday, December 19, 2014 9:14 AM
 To: Paolo Bonzini; Wu, Feng; t...@linutronix.de; mi...@redhat.com;
 h...@zytor.com; x...@kernel.org; g...@kernel.org; dw...@infradead.org;
 j...@8bytes.org; alex.william...@redhat.com; jiang@linux.intel.com
 Cc: eric.au...@linaro.org; linux-ker...@vger.kernel.org;
 iommu@lists.linux-foundation.org; k...@vger.kernel.org
 Subject: RE: [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for
 VT-d PI
 
 Paolo Bonzini wrote on 2014-12-19:
 
 
  On 18/12/2014 15:49, Zhang, Yang Z wrote:
  Here, we introduce a similar way with 'apic_arb_prio' to handle
  guest lowest priority interrtups when VT-d PI is used. Here is the
  ideas: - Each vCPU has a counter 'round_robin_counter'. - When
  guests sets an interrupts to lowest priority, we choose the vCPU
  with smallest 'round_robin_counter' as the destination, then
  increase it.
 
  How this can work well? All subsequent interrupts are delivered to
  one vCPU? It shouldn't be the best solution, need more consideration.
 
  Well, it's a hardware limitation.  The alternative (which is easy to
 
 Agree, it is limited by hardware. But lowest priority distributes the 
 interrupt
 more efficient than fixed mode. And current implementation more likes to
 switch the lowest priority mode to fixed mode. In case of interrupt intensive
 environment, this may be a bottleneck and VM may not benefit greatly from
 VT-d PI. But agree again, it is really a hardware limitation.
 
  implement) is to only do PI for single-CPU interrupts.  This should
  work well for multiqueue NICs (and of course for UP guests :)), so
  perhaps it's a good idea to only support that as a first attempt.
 
 The more easy way is to deliver the interrupt to the first matched VCPU we 
 find.
 The round_robin_counter really helps nothing here since the interrupt is
 delivered by hardware directly.
 
 
  Paolo
 
  Also, I think you should take the apic_arb_prio into consider since
  the priority is for the whole vCPU not for one interrupt.
 
 
 Best regards,
 Yang

In fact, the current solution was discussed with Rajesh in the cc List, here is 
Rajesh's original words:

When you see a guest requesting a lowest priority interrupts (by programming 
the virtual IOAPIC, or by programming the virtual MSI/MSI-X registers), have 
KVM associate it to a vCPU.  Or, put another way, use the 'apic_arb_prio' 
method you describe below, but instead of using it at time of interrupt (which 
you no longer have control with posted interrupt direct delivery), do it at 
time of initializing the interrupt resource.  This way, if the guest asks for 4 
lowest priority interrupts, and say you a guest with two vCPUs, the first 
interrupt request will be serviced by KVM by assigning it through posting to 
vCPU0, the next one goes to vCPU1, the next one would go back to vCPU0, and so 
forth..  You could also choose to do this based on vector hashing instead of 
round-robin.

Thanks,
Feng

 

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu