Re: [Xen-devel] [PATCH v3 06/26] ARM: GICv3 ITS: introduce device mapping

2017-04-03 Thread Julien Grall 

Hi Andre,

Mostly repeating my comments from the previous version.

On 31/03/17 19:05, Andre Przywara wrote:

[...]


+static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
+ uint8_t size_bits, paddr_t itt_addr, bool valid)
+{
+uint64_t cmd[4];
+
+if ( valid )
+{
+ASSERT(size_bits < 32);


Again, it would be better if you do the check against the real number in 
hardware (i.e GITS_TYPER.ID_bits).



+ASSERT(!(itt_addr & ~GENMASK_ULL(51, 8)));
+}
+cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
+cmd[1] = size_bits;


I would have expected to see size_bits - 1 to accommodate all the 
helpers rather than relying on them.


[...]


+static int remove_mapped_guest_device(struct its_devices *dev)
+{
+int ret;
+
+if ( dev->hw_its )
+{
+/* MAPD also discards all events with this device ID. */
+int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, false);


You are re-defining ret. Why?

[...]


+static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
+{
+struct host_its *hw_its;
+
+list_for_each_entry(hw_its, _its_list, entry)
+{
+if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )


Again, why not storing the ITS address rather than the doorbell to avoid 
"+ ITS_DOORBELL_OFFSET" ?


[...]


+/*
+ * Map a hardware device, identified by a certain host ITS and its device ID
+ * to domain d, a guest ITS (identified by its doorbell address) and device ID.
+ * Also provide the number of events (MSIs) needed for that device.
+ * This does not check if this particular hardware device is already mapped
+ * at another domain, it is expected that this would be done by the caller.
+ */
+int gicv3_its_map_guest_device(struct domain *d,
+   paddr_t host_doorbell, uint32_t host_devid,
+   paddr_t guest_doorbell, uint32_t guest_devid,
+   uint32_t nr_events, bool valid)


I am sure I said it somewhere in this series, nr_events likely needs to 
be sanitized against the hardware value. Same for host_devid.


[...]


+parent = *new;
+cmp = compare_its_guest_devices(temp, guest_doorbell, guest_devid);
+if ( !cmp )
+{
+if ( !valid )
+rb_erase(>rbnode, >arch.vgic.its_devices);
+
+spin_unlock(>arch.vgic.its_devices_lock);
+
+if ( valid )


Again, a printk(XENLOG_GUEST...) here would be useful to know which host 
DeviceID was associated to the guest DeviceID.



+return -EBUSY;
+
+return remove_mapped_guest_device(temp);


Again, just above you removed the device from the RB-tree but this 
function may fail and never free the memory. This means that memory will 
be leaked leading to a potential denial of service.



+}
+
+if ( cmp > 0 )
+new = &((*new)->rb_left);
+else
+new = &((*new)->rb_right);
+}
+
+if ( !valid )
+goto out_unlock;
+
+ret = -ENOMEM;
+
+/* An Interrupt Translation Table needs to be 256-byte aligned. */
+itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);


See Vijay's comment. But why don't you round up nr_events at the 
beginning once for all rather than doing it in the middle?


[...]


+out_unlock:
+spin_unlock(>arch.vgic.its_devices_lock);
+if ( dev )
+{
+xfree(dev->pend_irqs);
+xfree(dev->host_lpi_blocks);


Where is host_lpi_blocks allocated? Why is it freed here?


+}
+xfree(itt_addr);
+xfree(dev);
+return ret;
+}
+
+/* Removing any connections a domain had to any ITS in the system. */
+void gicv3_its_unmap_all_devices(struct domain *d)
+{
+struct rb_node *victim;
+struct its_devices *dev;
+
+/*
+ * This is an easily readable, but suboptimal implementation.
+ * It uses the provided iteration wrapper and erases each node, which
+ * possibly triggers rebalancing.
+ * This seems overkill since we are going to abolish the whole tree, but
+ * avoids an open-coded re-implementation of the traversal functions with
+ * some recursive function calls.
+ */


Well, you updated the comment but it does not make the performance 
problem going away... Xen cannot be preempted, so if it takes too long, 
you will have an impact on the overall system.


As said previously, I think it would be fair to assume that all devices 
will be deassigned before the ITS is destroyed. So I would just drop 
this function. Not that we have the same assumption in the SMMU driver.


If you disagree please say why. But ignoring comments will not help here.


+restart:
+spin_lock(>arch.vgic.its_devices_lock);
+if ( (victim = rb_first(>arch.vgic.its_devices)) )
+{
+dev = rb_entry(victim, struct its_devices, rbnode);
+rb_erase(victim, >arch.vgic.its_devices);
+
+

Re: [Xen-devel] [PATCH v3 06/26] ARM: GICv3 ITS: introduce device mapping

2017-04-03 Thread Julien Grall 

On 01/04/17 09:01, Vijay Kilari wrote:

Hi Andre,


Hi Vijay,


On Fri, Mar 31, 2017 at 11:35 PM, Andre Przywara  wrote:

+/* An Interrupt Translation Table needs to be 256-byte aligned. */
+itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);


  As I mentioned, in previous version, if itt_addr is not enough size,
ITS would overwrite and corrupt memory.
Similar to size passed in MAPD cmd, itt_addr should also be allocated of size
ROUNDUP(nr_events, LPI_BLOCK).


ROUNDUP(nr_events, LPI_BLOCK) would still be wrong as the MAPD command 
works in term of bits. You have to round up to the next bit.


Cheers,

--
Julien Grall

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 06/26] ARM: GICv3 ITS: introduce device mapping

2017-04-01 Thread Vijay Kilari 
Hi Andre,

On Fri, Mar 31, 2017 at 11:35 PM, Andre Przywara  wrote:
> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
> those IDs, which we directly pass on to the host.
> For this we have to map each device that Dom0 may request to a host
> ITS device with the same identifier.
> Allocate the respective memory and enter each device into an rbtree to
> later be able to iterate over it or to easily teardown guests.
>
> Signed-off-by: Andre Przywara 
> ---
>  xen/arch/arm/gic-v3-its.c| 227 
> +++
>  xen/arch/arm/vgic-v3.c   |   4 +
>  xen/include/asm-arm/domain.h |   3 +
>  xen/include/asm-arm/gic_v3_its.h |  23 
>  4 files changed, 257 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 1ac598f..295f7dc 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -21,6 +21,8 @@
>  #include 
>  #include 
>  #include 
> +#include 
> +#include 
>  #include 
>  #include 
>  #include 
> @@ -32,6 +34,18 @@
>
>  LIST_HEAD(host_its_list);
>
> +struct its_devices {
> +struct rb_node rbnode;
> +struct host_its *hw_its;
> +void *itt_addr;
> +paddr_t guest_doorbell; /* Identifies the virtual ITS */
> +uint32_t host_devid;
> +uint32_t guest_devid;
> +uint32_t eventids;  /* Number of event IDs (MSIs) */
> +uint32_t *host_lpi_blocks;  /* Which LPIs are used on the host */
> +struct pending_irq *pend_irqs;  /* One struct per event */
> +};
> +
>  bool gicv3_its_host_has_its(void)
>  {
>  return !list_empty(_its_list);
> @@ -151,6 +165,26 @@ static int its_send_cmd_mapc(struct host_its *its, 
> uint32_t collection_id,
>  return its_send_command(its, cmd);
>  }
>
> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
> + uint8_t size_bits, paddr_t itt_addr, bool valid)
> +{
> +uint64_t cmd[4];
> +
> +if ( valid )
> +{
> +ASSERT(size_bits < 32);
> +ASSERT(!(itt_addr & ~GENMASK_ULL(51, 8)));
> +}
> +cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
> +cmd[1] = size_bits;
> +cmd[2] = itt_addr;
> +if ( valid )
> +cmd[2] |= GITS_VALID_BIT;
> +cmd[3] = 0x00;
> +
> +return its_send_command(its, cmd);
> +}
> +
>  /* Set up the (1:1) collection mapping for the given host CPU. */
>  int gicv3_its_setup_collection(unsigned int cpu)
>  {
> @@ -376,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its 
> *hw_its)
>  hw_its->devid_bits = min(hw_its->devid_bits, max_its_device_bits);
>  if ( reg & GITS_TYPER_PTA )
>  hw_its->flags |= HOST_ITS_USES_PTA;
> +hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
>
>  for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
>  {
> @@ -432,6 +467,197 @@ int gicv3_its_init(void)
>  return 0;
>  }
>
> +static int remove_mapped_guest_device(struct its_devices *dev)
> +{
> +int ret;
> +
> +if ( dev->hw_its )
> +{
> +/* MAPD also discards all events with this device ID. */
> +int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, 
> false);
> +if ( ret )
> +return ret;
> +}
> +
> +ret = gicv3_its_wait_commands(dev->hw_its);
> +if ( ret )
> +return ret;
> +
> +xfree(dev->itt_addr);
> +xfree(dev->pend_irqs);
> +xfree(dev);
> +
> +return 0;
> +}
> +
> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
> +{
> +struct host_its *hw_its;
> +
> +list_for_each_entry(hw_its, _its_list, entry)
> +{
> +if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
> +return hw_its;
> +}
> +
> +return NULL;
> +}
> +
> +static int compare_its_guest_devices(struct its_devices *dev,
> + paddr_t doorbell, uint32_t devid)
> +{
> +if ( dev->guest_doorbell < doorbell )
> +return -1;
> +
> +if ( dev->guest_doorbell > doorbell )
> +return 1;
> +
> +if ( dev->guest_devid < devid )
> +return -1;
> +
> +if ( dev->guest_devid > devid )
> +return 1;
> +
> +return 0;
> +}
> +
> +/*
> + * Map a hardware device, identified by a certain host ITS and its device ID
> + * to domain d, a guest ITS (identified by its doorbell address) and device 
> ID.
> + * Also provide the number of events (MSIs) needed for that device.
> + * This does not check if this particular hardware device is already mapped
> + * at another domain, it is expected that this would be done by the caller.
> + */
> +int gicv3_its_map_guest_device(struct domain *d,
> +   paddr_t host_doorbell, uint32_t host_devid,
> +   paddr_t guest_doorbell, uint32_t guest_devid,
> +   uint32_t nr_events, bool valid)
> +{
> +void *itt_addr = NULL;
> +

Re: [Xen-devel] [PATCH v3 06/26] ARM: GICv3 ITS: introduce device mapping

2017-03-31 Thread Stefano Stabellini 
On Fri, 31 Mar 2017, Andre Przywara wrote:
> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
> those IDs, which we directly pass on to the host.
> For this we have to map each device that Dom0 may request to a host
> ITS device with the same identifier.
> Allocate the respective memory and enter each device into an rbtree to
> later be able to iterate over it or to easily teardown guests.
> 
> Signed-off-by: Andre Przywara 

See 3a044833-4b61-d807-600c-cf88d6e19...@arm.com and
alpine.DEB.2.10.1703211718190.11679@sstabellini-ThinkPad-X260 


> ---
>  xen/arch/arm/gic-v3-its.c| 227 
> +++
>  xen/arch/arm/vgic-v3.c   |   4 +
>  xen/include/asm-arm/domain.h |   3 +
>  xen/include/asm-arm/gic_v3_its.h |  23 
>  4 files changed, 257 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 1ac598f..295f7dc 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -21,6 +21,8 @@
>  #include 
>  #include 
>  #include 
> +#include 
> +#include 
>  #include 
>  #include 
>  #include 
> @@ -32,6 +34,18 @@
>  
>  LIST_HEAD(host_its_list);
>  
> +struct its_devices {
> +struct rb_node rbnode;
> +struct host_its *hw_its;
> +void *itt_addr;
> +paddr_t guest_doorbell; /* Identifies the virtual ITS */
> +uint32_t host_devid;
> +uint32_t guest_devid;
> +uint32_t eventids;  /* Number of event IDs (MSIs) */
> +uint32_t *host_lpi_blocks;  /* Which LPIs are used on the host */
> +struct pending_irq *pend_irqs;  /* One struct per event */
> +};
> +
>  bool gicv3_its_host_has_its(void)
>  {
>  return !list_empty(_its_list);
> @@ -151,6 +165,26 @@ static int its_send_cmd_mapc(struct host_its *its, 
> uint32_t collection_id,
>  return its_send_command(its, cmd);
>  }
>  
> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
> + uint8_t size_bits, paddr_t itt_addr, bool valid)
> +{
> +uint64_t cmd[4];
> +
> +if ( valid )
> +{
> +ASSERT(size_bits < 32);
> +ASSERT(!(itt_addr & ~GENMASK_ULL(51, 8)));
> +}
> +cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
> +cmd[1] = size_bits;
> +cmd[2] = itt_addr;
> +if ( valid )
> +cmd[2] |= GITS_VALID_BIT;
> +cmd[3] = 0x00;
> +
> +return its_send_command(its, cmd);
> +}
> +
>  /* Set up the (1:1) collection mapping for the given host CPU. */
>  int gicv3_its_setup_collection(unsigned int cpu)
>  {
> @@ -376,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its 
> *hw_its)
>  hw_its->devid_bits = min(hw_its->devid_bits, max_its_device_bits);
>  if ( reg & GITS_TYPER_PTA )
>  hw_its->flags |= HOST_ITS_USES_PTA;
> +hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
>  
>  for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
>  {
> @@ -432,6 +467,197 @@ int gicv3_its_init(void)
>  return 0;
>  }
>  
> +static int remove_mapped_guest_device(struct its_devices *dev)
> +{
> +int ret;
> +
> +if ( dev->hw_its )
> +{
> +/* MAPD also discards all events with this device ID. */
> +int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, 
> false);
> +if ( ret )
> +return ret;
> +}
> +
> +ret = gicv3_its_wait_commands(dev->hw_its);
> +if ( ret )
> +return ret;
> +
> +xfree(dev->itt_addr);
> +xfree(dev->pend_irqs);
> +xfree(dev);
> +
> +return 0;
> +}
> +
> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
> +{
> +struct host_its *hw_its;
> +
> +list_for_each_entry(hw_its, _its_list, entry)
> +{
> +if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
> +return hw_its;
> +}
> +
> +return NULL;
> +}
> +
> +static int compare_its_guest_devices(struct its_devices *dev,
> + paddr_t doorbell, uint32_t devid)
> +{
> +if ( dev->guest_doorbell < doorbell )
> +return -1;
> +
> +if ( dev->guest_doorbell > doorbell )
> +return 1;
> +
> +if ( dev->guest_devid < devid )
> +return -1;
> +
> +if ( dev->guest_devid > devid )
> +return 1;
> +
> +return 0;
> +}
> +
> +/*
> + * Map a hardware device, identified by a certain host ITS and its device ID
> + * to domain d, a guest ITS (identified by its doorbell address) and device 
> ID.
> + * Also provide the number of events (MSIs) needed for that device.
> + * This does not check if this particular hardware device is already mapped
> + * at another domain, it is expected that this would be done by the caller.
> + */
> +int gicv3_its_map_guest_device(struct domain *d,
> +   paddr_t host_doorbell, uint32_t host_devid,
> +   paddr_t guest_doorbell, uint32_t guest_devid,
> +

[Xen-devel] [PATCH v3 06/26] ARM: GICv3 ITS: introduce device mapping

2017-03-31 Thread Andre Przywara 
The ITS uses device IDs to map LPIs to a device. Dom0 will later use
those IDs, which we directly pass on to the host.
For this we have to map each device that Dom0 may request to a host
ITS device with the same identifier.
Allocate the respective memory and enter each device into an rbtree to
later be able to iterate over it or to easily teardown guests.

Signed-off-by: Andre Przywara 
---
 xen/arch/arm/gic-v3-its.c| 227 +++
 xen/arch/arm/vgic-v3.c   |   4 +
 xen/include/asm-arm/domain.h |   3 +
 xen/include/asm-arm/gic_v3_its.h |  23 
 4 files changed, 257 insertions(+)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 1ac598f..295f7dc 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -21,6 +21,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -32,6 +34,18 @@
 
 LIST_HEAD(host_its_list);
 
+struct its_devices {
+struct rb_node rbnode;
+struct host_its *hw_its;
+void *itt_addr;
+paddr_t guest_doorbell; /* Identifies the virtual ITS */
+uint32_t host_devid;
+uint32_t guest_devid;
+uint32_t eventids;  /* Number of event IDs (MSIs) */
+uint32_t *host_lpi_blocks;  /* Which LPIs are used on the host */
+struct pending_irq *pend_irqs;  /* One struct per event */
+};
+
 bool gicv3_its_host_has_its(void)
 {
 return !list_empty(_its_list);
@@ -151,6 +165,26 @@ static int its_send_cmd_mapc(struct host_its *its, 
uint32_t collection_id,
 return its_send_command(its, cmd);
 }
 
+static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
+ uint8_t size_bits, paddr_t itt_addr, bool valid)
+{
+uint64_t cmd[4];
+
+if ( valid )
+{
+ASSERT(size_bits < 32);
+ASSERT(!(itt_addr & ~GENMASK_ULL(51, 8)));
+}
+cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
+cmd[1] = size_bits;
+cmd[2] = itt_addr;
+if ( valid )
+cmd[2] |= GITS_VALID_BIT;
+cmd[3] = 0x00;
+
+return its_send_command(its, cmd);
+}
+
 /* Set up the (1:1) collection mapping for the given host CPU. */
 int gicv3_its_setup_collection(unsigned int cpu)
 {
@@ -376,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its 
*hw_its)
 hw_its->devid_bits = min(hw_its->devid_bits, max_its_device_bits);
 if ( reg & GITS_TYPER_PTA )
 hw_its->flags |= HOST_ITS_USES_PTA;
+hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
 
 for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
 {
@@ -432,6 +467,197 @@ int gicv3_its_init(void)
 return 0;
 }
 
+static int remove_mapped_guest_device(struct its_devices *dev)
+{
+int ret;
+
+if ( dev->hw_its )
+{
+/* MAPD also discards all events with this device ID. */
+int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, false);
+if ( ret )
+return ret;
+}
+
+ret = gicv3_its_wait_commands(dev->hw_its);
+if ( ret )
+return ret;
+
+xfree(dev->itt_addr);
+xfree(dev->pend_irqs);
+xfree(dev);
+
+return 0;
+}
+
+static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
+{
+struct host_its *hw_its;
+
+list_for_each_entry(hw_its, _its_list, entry)
+{
+if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
+return hw_its;
+}
+
+return NULL;
+}
+
+static int compare_its_guest_devices(struct its_devices *dev,
+ paddr_t doorbell, uint32_t devid)
+{
+if ( dev->guest_doorbell < doorbell )
+return -1;
+
+if ( dev->guest_doorbell > doorbell )
+return 1;
+
+if ( dev->guest_devid < devid )
+return -1;
+
+if ( dev->guest_devid > devid )
+return 1;
+
+return 0;
+}
+
+/*
+ * Map a hardware device, identified by a certain host ITS and its device ID
+ * to domain d, a guest ITS (identified by its doorbell address) and device ID.
+ * Also provide the number of events (MSIs) needed for that device.
+ * This does not check if this particular hardware device is already mapped
+ * at another domain, it is expected that this would be done by the caller.
+ */
+int gicv3_its_map_guest_device(struct domain *d,
+   paddr_t host_doorbell, uint32_t host_devid,
+   paddr_t guest_doorbell, uint32_t guest_devid,
+   uint32_t nr_events, bool valid)
+{
+void *itt_addr = NULL;
+struct host_its *hw_its;
+struct its_devices *dev = NULL;
+struct rb_node **new = >arch.vgic.its_devices.rb_node, *parent = NULL;
+int ret = -ENOENT;
+
+hw_its = gicv3_its_find_by_doorbell(host_doorbell);
+if ( !hw_its )
+return ret;
+
+/* check for already existing mappings */
+spin_lock(>arch.vgic.its_devices_lock);
+while ( *new )
+{
+struct