RE: [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS emulation

[Pavel Fedin]

 Hello!

> -Original Message-
> From: kvm-ow...@vger.kernel.org [mailto:kvm-ow...@vger.kernel.org] On Behalf 
> Of Andre Przywara
> Sent: Wednesday, October 07, 2015 5:55 PM
> To: marc.zyng...@arm.com; christoffer.d...@linaro.org
> Cc: eric.au...@linaro.org; p.fe...@samsung.com; kvm...@lists.cs.columbia.edu; 
> linux-arm-
> ker...@lists.infradead.org; kvm@vger.kernel.org
> Subject: [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS 
> emulation
> 
> As the actual LPI number in a guest can be quite high, but is mostly
> assigned using a very sparse allocation scheme, bitmaps and arrays
> for storing the virtual interrupt status are a waste of memory.
> We use our equivalent of the "Interrupt Translation Table Entry"
> (ITTE) to hold this extra status information for a virtual LPI.
> As the normal VGIC code cannot use its fancy bitmaps to manage
> pending interrupts, we provide a hook in the VGIC code to let the
> ITS emulation handle the list register queueing itself.
> LPIs are located in a separate number range (>=8192), so
> distinguishing them is easy. With LPIs being only edge-triggered, we
> get away with a less complex IRQ handling.
> We extend the number of bits for storing the IRQ number in our
> LR struct to 16 to cover the LPI numbers we support as well.
> 
> Signed-off-by: Andre Przywara <andre.przyw...@arm.com>
> ---
> Changelog v2..v3:
> - extend LR data structure to hold 16-bit wide IRQ IDs
> - only clear pending bit if IRQ could be queued
> - adapt __kvm_vgic_sync_hwstate() to upstream changes
> 
>  include/kvm/arm_vgic.h  |  4 +-
>  virt/kvm/arm/its-emul.c | 75 
>  virt/kvm/arm/its-emul.h |  3 ++
>  virt/kvm/arm/vgic-v3-emul.c |  2 +
>  virt/kvm/arm/vgic.c | 93 
> +++--
>  5 files changed, 148 insertions(+), 29 deletions(-)
> 
> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> index c3eb414..035911f 100644
> --- a/include/kvm/arm_vgic.h
> +++ b/include/kvm/arm_vgic.h
> @@ -95,7 +95,7 @@ enum vgic_type {
>  #define LR_HW(1 << 3)
> 
>  struct vgic_lr {
> - unsigned irq:10;
> + unsigned irq:16;
>   union {
>   unsigned hwirq:10;
>   unsigned source:3;
> @@ -147,6 +147,8 @@ struct vgic_vm_ops {
>   int (*init_model)(struct kvm *);
>   void(*destroy_model)(struct kvm *);
>   int (*map_resources)(struct kvm *, const struct vgic_params *);
> + bool(*queue_lpis)(struct kvm_vcpu *);
> + void(*unqueue_lpi)(struct kvm_vcpu *, int irq);
>  };
> 
>  struct vgic_io_device {
> diff --git a/virt/kvm/arm/its-emul.c b/virt/kvm/arm/its-emul.c
> index bab8033..8349970 100644
> --- a/virt/kvm/arm/its-emul.c
> +++ b/virt/kvm/arm/its-emul.c
> @@ -59,8 +59,27 @@ struct its_itte {
>   struct its_collection *collection;
>   u32 lpi;
>   u32 event_id;
> + bool enabled;
> + unsigned long *pending;
>  };
> 
> +/* To be used as an iterator this macro misses the enclosing parentheses */
> +#define for_each_lpi(dev, itte, kvm) \
> + list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
> + list_for_each_entry(itte, &(dev)->itt, itte_list)
> +
> +static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
> +{
> + struct its_device *device;
> + struct its_itte *itte;
> +
> + for_each_lpi(device, itte, kvm) {
> + if (itte->lpi == lpi)
> + return itte;
> + }
> + return NULL;
> +}
> +
>  #define BASER_BASE_ADDRESS(x) ((x) & 0xf000ULL)
> 
>  /* The distributor lock is held by the VGIC MMIO handler. */
> @@ -154,9 +173,65 @@ static bool handle_mmio_gits_idregs(struct kvm_vcpu 
> *vcpu,
>   return false;
>  }
> 
> +/*
> + * Find all enabled and pending LPIs and queue them into the list
> + * registers.
> + * The dist lock is held by the caller.
> + */
> +bool vits_queue_lpis(struct kvm_vcpu *vcpu)
> +{
> + struct vgic_its *its = >kvm->arch.vgic.its;
> + struct its_device *device;
> + struct its_itte *itte;
> + bool ret = true;
> +
> + if (!vgic_has_its(vcpu->kvm))
> + return true;
> + if (!its->enabled || !vcpu->kvm->arch.vgic.lpis_enabled)
> + return true;
> +
> + spin_lock(>lock);
> + for_each_lpi(device, itte, vcpu->kvm) {
> + if (!itte->enabled || !test_bit(vcpu->vcpu_id, itte->pending))
> + continue;
> +
> + if (!itte->collection)
>

RE: [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS emulation

[Pavel Fedin]

 Hello!

>  Shouldn't we also have 'break' here? If vgic_queue_irq() returns false, this 
> means we have no
> more
> LRs to use, therefore it makes no sense to keep iterating.

 No, don't listen to me. :) Because of piggyback, we indeed have to recheck all 
the interrupts.

 P.S. I still sometimes lose LPIs, and this is not related to spurious 
injection fix, because i
tried to omit resetting irq_pending_on_cpu bit, and still lost some LPIs. Will 
try to compare with
v2, because with v2 i don't remember this problem.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia


--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS emulation

[Andre Przywara]

Hi Pavel,

On 12/10/15 08:40, Pavel Fedin wrote:
>  Hello!
> 
>> -Original Message-
>> From: kvm-ow...@vger.kernel.org [mailto:kvm-ow...@vger.kernel.org] On Behalf 
>> Of Andre Przywara
>> Sent: Wednesday, October 07, 2015 5:55 PM
>> To: marc.zyng...@arm.com; christoffer.d...@linaro.org
>> Cc: eric.au...@linaro.org; p.fe...@samsung.com; 
>> kvm...@lists.cs.columbia.edu; linux-arm-
>> ker...@lists.infradead.org; kvm@vger.kernel.org
>> Subject: [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS 
>> emulation
>>
>> As the actual LPI number in a guest can be quite high, but is mostly
>> assigned using a very sparse allocation scheme, bitmaps and arrays
>> for storing the virtual interrupt status are a waste of memory.
>> We use our equivalent of the "Interrupt Translation Table Entry"
>> (ITTE) to hold this extra status information for a virtual LPI.
>> As the normal VGIC code cannot use its fancy bitmaps to manage
>> pending interrupts, we provide a hook in the VGIC code to let the
>> ITS emulation handle the list register queueing itself.
>> LPIs are located in a separate number range (>=8192), so
>> distinguishing them is easy. With LPIs being only edge-triggered, we
>> get away with a less complex IRQ handling.
>> We extend the number of bits for storing the IRQ number in our
>> LR struct to 16 to cover the LPI numbers we support as well.
>>
>> Signed-off-by: Andre Przywara <andre.przyw...@arm.com>
>> ---
>> Changelog v2..v3:
>> - extend LR data structure to hold 16-bit wide IRQ IDs
>> - only clear pending bit if IRQ could be queued
>> - adapt __kvm_vgic_sync_hwstate() to upstream changes
>>
>>  include/kvm/arm_vgic.h  |  4 +-
>>  virt/kvm/arm/its-emul.c | 75 
>>  virt/kvm/arm/its-emul.h |  3 ++
>>  virt/kvm/arm/vgic-v3-emul.c |  2 +
>>  virt/kvm/arm/vgic.c | 93 
>> +++--
>>  5 files changed, 148 insertions(+), 29 deletions(-)
>>
>> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
>> index c3eb414..035911f 100644
>> --- a/include/kvm/arm_vgic.h
>> +++ b/include/kvm/arm_vgic.h
>> @@ -95,7 +95,7 @@ enum vgic_type {
>>  #define LR_HW   (1 << 3)
>>
>>  struct vgic_lr {
>> -unsigned irq:10;
>> +unsigned irq:16;
>>  union {
>>  unsigned hwirq:10;
>>  unsigned source:3;
>> @@ -147,6 +147,8 @@ struct vgic_vm_ops {
>>  int (*init_model)(struct kvm *);
>>  void(*destroy_model)(struct kvm *);
>>  int (*map_resources)(struct kvm *, const struct vgic_params *);
>> +bool(*queue_lpis)(struct kvm_vcpu *);
>> +void(*unqueue_lpi)(struct kvm_vcpu *, int irq);
>>  };
>>
>>  struct vgic_io_device {
>> diff --git a/virt/kvm/arm/its-emul.c b/virt/kvm/arm/its-emul.c
>> index bab8033..8349970 100644
>> --- a/virt/kvm/arm/its-emul.c
>> +++ b/virt/kvm/arm/its-emul.c
>> @@ -59,8 +59,27 @@ struct its_itte {
>>  struct its_collection *collection;
>>  u32 lpi;
>>  u32 event_id;
>> +bool enabled;
>> +unsigned long *pending;
>>  };
>>
>> +/* To be used as an iterator this macro misses the enclosing parentheses */
>> +#define for_each_lpi(dev, itte, kvm) \
>> +list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
>> +list_for_each_entry(itte, &(dev)->itt, itte_list)
>> +
>> +static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
>> +{
>> +struct its_device *device;
>> +struct its_itte *itte;
>> +
>> +for_each_lpi(device, itte, kvm) {
>> +if (itte->lpi == lpi)
>> +return itte;
>> +}
>> +return NULL;
>> +}
>> +
>>  #define BASER_BASE_ADDRESS(x) ((x) & 0xf000ULL)
>>
>>  /* The distributor lock is held by the VGIC MMIO handler. */
>> @@ -154,9 +173,65 @@ static bool handle_mmio_gits_idregs(struct kvm_vcpu 
>> *vcpu,
>>  return false;
>>  }
>>
>> +/*
>> + * Find all enabled and pending LPIs and queue them into the list
>> + * registers.
>> + * The dist lock is held by the caller.
>> + */
>> +bool vits_queue_lpis(struct kvm_vcpu *vcpu)
>> +{
>> +struct vgic_its *its = >kvm->arch.vgic.its;
>> +struct its_device *device;
>> +struct its_itte *itte;
>> +bool ret = true;
>&g

Re: [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS emulation

[Marc Zyngier]

On 07/10/15 16:10, Pavel Fedin wrote:
>  Hello!
> 
>> As the actual LPI number in a guest can be quite high, but is mostly
>> assigned using a very sparse allocation scheme, bitmaps and arrays
>> for storing the virtual interrupt status are a waste of memory.
>> We use our equivalent of the "Interrupt Translation Table Entry"
>> (ITTE) to hold this extra status information for a virtual LPI.
> 
> You know, not that i'm strongly against current approach and want you
> to redo everything once again, but... Is it architecturally correct
> to intertwine LPIs and ITS so much? As far as i

Yes it is.

> understand arch manual, it is possible to have LPIs without ITS
> (triggered by something else?). Shouldn't we do the same, and just
> add LPI support to our redistributors, and then proceed with the 
> ITS?

No. We're implementing a monolithic GICv3 that doesn't offer writing to
the redistributors directly from a device. And frankly, that's good enough.

> As to memory consumption, do we really need to store own copy of
> tables? After all, it's just a memory. What if we map a pointer
> directly into guest's memory (which it writes to 
> PROPBASER/PENDBASER), and just keep it? There will be no issues with
> caching and synchronization at all.

Sure. And you then have to parse and validate all the tables each and
every time you're going to inject an interrupt (because the guest can
change the table content behind your back). You are quickly going to
notice that your performance is abysmal.

At that point, you're going to start being clever, and add a cache. And
guess what, that's what the HW does too. And then you'll make your cache
a convenient structure to be able to quickly inject interrupts. And
that's what the HW does too. And finally, you're going to realize that
populating a cache sucks, and you're going to keep all the state where
it is convenient, when it is convenient (and that's basically always).

The HW can't do that, but we can.

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS emulation

[Andre Przywara]

As the actual LPI number in a guest can be quite high, but is mostly
assigned using a very sparse allocation scheme, bitmaps and arrays
for storing the virtual interrupt status are a waste of memory.
We use our equivalent of the "Interrupt Translation Table Entry"
(ITTE) to hold this extra status information for a virtual LPI.
As the normal VGIC code cannot use its fancy bitmaps to manage
pending interrupts, we provide a hook in the VGIC code to let the
ITS emulation handle the list register queueing itself.
LPIs are located in a separate number range (>=8192), so
distinguishing them is easy. With LPIs being only edge-triggered, we
get away with a less complex IRQ handling.
We extend the number of bits for storing the IRQ number in our
LR struct to 16 to cover the LPI numbers we support as well.

Signed-off-by: Andre Przywara 
---
Changelog v2..v3:
- extend LR data structure to hold 16-bit wide IRQ IDs
- only clear pending bit if IRQ could be queued
- adapt __kvm_vgic_sync_hwstate() to upstream changes

 include/kvm/arm_vgic.h  |  4 +-
 virt/kvm/arm/its-emul.c | 75 
 virt/kvm/arm/its-emul.h |  3 ++
 virt/kvm/arm/vgic-v3-emul.c |  2 +
 virt/kvm/arm/vgic.c | 93 +++--
 5 files changed, 148 insertions(+), 29 deletions(-)

diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index c3eb414..035911f 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -95,7 +95,7 @@ enum vgic_type {
 #define LR_HW  (1 << 3)
 
 struct vgic_lr {
-   unsigned irq:10;
+   unsigned irq:16;
union {
unsigned hwirq:10;
unsigned source:3;
@@ -147,6 +147,8 @@ struct vgic_vm_ops {
int (*init_model)(struct kvm *);
void(*destroy_model)(struct kvm *);
int (*map_resources)(struct kvm *, const struct vgic_params *);
+   bool(*queue_lpis)(struct kvm_vcpu *);
+   void(*unqueue_lpi)(struct kvm_vcpu *, int irq);
 };
 
 struct vgic_io_device {
diff --git a/virt/kvm/arm/its-emul.c b/virt/kvm/arm/its-emul.c
index bab8033..8349970 100644
--- a/virt/kvm/arm/its-emul.c
+++ b/virt/kvm/arm/its-emul.c
@@ -59,8 +59,27 @@ struct its_itte {
struct its_collection *collection;
u32 lpi;
u32 event_id;
+   bool enabled;
+   unsigned long *pending;
 };
 
+/* To be used as an iterator this macro misses the enclosing parentheses */
+#define for_each_lpi(dev, itte, kvm) \
+   list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
+   list_for_each_entry(itte, &(dev)->itt, itte_list)
+
+static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
+{
+   struct its_device *device;
+   struct its_itte *itte;
+
+   for_each_lpi(device, itte, kvm) {
+   if (itte->lpi == lpi)
+   return itte;
+   }
+   return NULL;
+}
+
 #define BASER_BASE_ADDRESS(x) ((x) & 0xf000ULL)
 
 /* The distributor lock is held by the VGIC MMIO handler. */
@@ -154,9 +173,65 @@ static bool handle_mmio_gits_idregs(struct kvm_vcpu *vcpu,
return false;
 }
 
+/*
+ * Find all enabled and pending LPIs and queue them into the list
+ * registers.
+ * The dist lock is held by the caller.
+ */
+bool vits_queue_lpis(struct kvm_vcpu *vcpu)
+{
+   struct vgic_its *its = >kvm->arch.vgic.its;
+   struct its_device *device;
+   struct its_itte *itte;
+   bool ret = true;
+
+   if (!vgic_has_its(vcpu->kvm))
+   return true;
+   if (!its->enabled || !vcpu->kvm->arch.vgic.lpis_enabled)
+   return true;
+
+   spin_lock(>lock);
+   for_each_lpi(device, itte, vcpu->kvm) {
+   if (!itte->enabled || !test_bit(vcpu->vcpu_id, itte->pending))
+   continue;
+
+   if (!itte->collection)
+   continue;
+
+   if (itte->collection->target_addr != vcpu->vcpu_id)
+   continue;
+
+
+   if (vgic_queue_irq(vcpu, 0, itte->lpi))
+   __clear_bit(vcpu->vcpu_id, itte->pending);
+   else
+   ret = false;
+   }
+
+   spin_unlock(>lock);
+   return ret;
+}
+
+/* Called with the distributor lock held by the caller. */
+void vits_unqueue_lpi(struct kvm_vcpu *vcpu, int lpi)
+{
+   struct vgic_its *its = >kvm->arch.vgic.its;
+   struct its_itte *itte;
+
+   spin_lock(>lock);
+
+   /* Find the right ITTE and put the pending state back in there */
+   itte = find_itte_by_lpi(vcpu->kvm, lpi);
+   if (itte)
+   __set_bit(vcpu->vcpu_id, itte->pending);
+
+   spin_unlock(>lock);
+}
+
 static void its_free_itte(struct its_itte *itte)
 {
list_del(>itte_list);
+   kfree(itte->pending);
kfree(itte);
 }
 
diff --git a/virt/kvm/arm/its-emul.h b/virt/kvm/arm/its-emul.h
index 

RE: [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS emulation

[Pavel Fedin]

 Hello!

> As the actual LPI number in a guest can be quite high, but is mostly
> assigned using a very sparse allocation scheme, bitmaps and arrays
> for storing the virtual interrupt status are a waste of memory.
> We use our equivalent of the "Interrupt Translation Table Entry"
> (ITTE) to hold this extra status information for a virtual LPI.

 You know, not that i'm strongly against current approach and want you to redo 
everything once
again, but... Is it architecturally correct to intertwine LPIs and ITS so much? 
As far as i
understand arch manual, it is possible to have LPIs without ITS (triggered by 
something else?).
Shouldn't we do the same, and just add LPI support to our redistributors, and 
then proceed with the
ITS?
 As to memory consumption, do we really need to store own copy of tables? After 
all, it's just a
memory. What if we map a pointer directly into guest's memory (which it writes 
to
PROPBASER/PENDBASER), and just keep it? There will be no issues with caching 
and synchronization at
all.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia


--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS emulation

[Marc Zyngier]

On 07/10/15 16:46, Pavel Fedin wrote:
>  Hello!
> 
>> Sure. And you then have to parse and validate all the tables each and
>> every time you're going to inject an interrupt (because the guest can
>> change the table content behind your back). You are quickly going to
>> notice that your performance is abysmal.
> 
>  I don't see any real problems, at least with LPI tables. If the guest 
> changes something, it will be
> immediately available to us. I don't see any need to seriously validate 
> something, at least here.
> Pending bit is just pending bit, and configuration is just priority value 
> plus enable bit.
>  But, well, if we think a bit better, in case of pending bit modification, 
> the operations on both
> guest and host side have to be atomic, otherwise we can clobber our table if, 
> for example, both host
> and guest modify adjacent bits. And there's no way to interlock with the 
> guest. So, OK, i accept
> your point.

The pending table is the least of our concerns. Device table, ITTs,
collections. That's the real problem.

M.
-- 
Jazz is not dead. It just smells funny...
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v3 12/16] KVM: arm64: handle pending bit for LPIs in ITS emulation

[Pavel Fedin]

 Hello!

> Sure. And you then have to parse and validate all the tables each and
> every time you're going to inject an interrupt (because the guest can
> change the table content behind your back). You are quickly going to
> notice that your performance is abysmal.

 I don't see any real problems, at least with LPI tables. If the guest changes 
something, it will be
immediately available to us. I don't see any need to seriously validate 
something, at least here.
Pending bit is just pending bit, and configuration is just priority value plus 
enable bit.
 But, well, if we think a bit better, in case of pending bit modification, the 
operations on both
guest and host side have to be atomic, otherwise we can clobber our table if, 
for example, both host
and guest modify adjacent bits. And there's no way to interlock with the guest. 
So, OK, i accept
your point.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia


--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html