Re: [Xen-devel] [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping

[Andre Przywara]

Hi,

On 03/04/17 21:41, Julien Grall wrote:
> 
> 
> On 04/03/2017 09:08 PM, Andre Przywara wrote:
>> Hi,
> 
> Hi Andre,
> 
>> On 22/03/17 17:29, Julien Grall wrote:
 +int gicv3_its_map_guest_device(struct domain *d,
 +   paddr_t host_doorbell, uint32_t
 host_devid,
 +   paddr_t guest_doorbell, uint32_t
 guest_devid,
 +   uint32_t nr_events, bool valid)
 +{
 +void *itt_addr = NULL;
 +struct host_its *hw_its;
 +struct its_devices *dev = NULL, *temp;
 +struct rb_node **new = >arch.vgic.its_devices.rb_node, *parent
 = NULL;
 +int ret = -ENOENT;
 +
 +hw_its = gicv3_its_find_by_doorbell(host_doorbell);
 +if ( !hw_its )
 +return ret;
 +
 +/* check for already existing mappings */
 +spin_lock(>arch.vgic.its_devices_lock);
 +while ( *new )
 +{
 +temp = rb_entry(*new, struct its_devices, rbnode);
 +
 +parent = *new;
 +if ( !compare_its_guest_devices(temp, guest_doorbell,
 guest_devid) )
 +{
 +if ( !valid )
 +rb_erase(>rbnode, >arch.vgic.its_devices);
 +
 +spin_unlock(>arch.vgic.its_devices_lock);
 +
 +if ( valid )
>>>
>>> Again, a printk(XENLOG_GUEST...) here would be useful to know which host
>>> DeviceID was associated to the guest DeviceID.
>>
>> I added a gprintk(XENLOG_DEBUG, ), which I think is more appropriate (as
>> it may spam the console when some stupid guest is running). Let me know
>> if you want to have a different loglevel.
> 
> I don't think this is more appropriate. gprintk will print the domain ID
> of the current domain, whilst this function will be called by the
> toolstack in the future.
> 
> Furthemore, if you look at the implementation of gprintk you will notice
> that it is basically turning into printk(XENLOG_GUEST...) and adding
> information of the current vCPU.
> 
> What matters for ratelimiting is XENLOG_GUEST.
> 
>>
 +return -EBUSY;
 +
 +return remove_mapped_guest_device(temp);
>>>
>>> Just above you removed the device from the RB-tree but this function may
>>> fail and never free the memory. This means that memory will be leaked
>>> leading to a potential denial of service.
>>
>> So I fixed this case in v4, though there is still a tiny chance of a
>> memleak: if the MAPD(V=0) command fails. We can't free the ITT table
>> then, really, because it still belongs to the ITS. I don't think we can
>> do much about it, though.
> 
> This is a leak and even tiny is quite worrying. How do you plan to
> address this in the future? What is the best thing to do?

In an ideal world we would probably have a spillover queue for ITS
commands. Whenever a command can't be queued to the hardware ITS queue
immediately, the guest should be put to wait somehow and the commands
recorded, to be executed whenever the hardware command queue gets free
slots again.
However I think we can't really do this today, because we don't have a
good way of putting a guest to sleep when it trapped on an MMIO access.
When in the future device assignment / de-assignment is handled via a
hypercall, we can probably address this more easily.

As for the likeliness: I think this is extremely rare. With our large
command queue and the ITS running at multiple hundred MHz I don't think
we will ever run into this situation, really, especially not with just Dom0.

So for now are we OK with this comment noting the corner case? Maybe a TODO?

Cheers,
Andre.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping

[Julien Grall]

On 04/03/2017 09:08 PM, Andre Przywara wrote:

Hi,


Hi Andre,


On 22/03/17 17:29, Julien Grall wrote:

+int gicv3_its_map_guest_device(struct domain *d,
+   paddr_t host_doorbell, uint32_t
host_devid,
+   paddr_t guest_doorbell, uint32_t
guest_devid,
+   uint32_t nr_events, bool valid)
+{
+void *itt_addr = NULL;
+struct host_its *hw_its;
+struct its_devices *dev = NULL, *temp;
+struct rb_node **new = >arch.vgic.its_devices.rb_node, *parent
= NULL;
+int ret = -ENOENT;
+
+hw_its = gicv3_its_find_by_doorbell(host_doorbell);
+if ( !hw_its )
+return ret;
+
+/* check for already existing mappings */
+spin_lock(>arch.vgic.its_devices_lock);
+while ( *new )
+{
+temp = rb_entry(*new, struct its_devices, rbnode);
+
+parent = *new;
+if ( !compare_its_guest_devices(temp, guest_doorbell,
guest_devid) )
+{
+if ( !valid )
+rb_erase(>rbnode, >arch.vgic.its_devices);
+
+spin_unlock(>arch.vgic.its_devices_lock);
+
+if ( valid )


Again, a printk(XENLOG_GUEST...) here would be useful to know which host
DeviceID was associated to the guest DeviceID.


I added a gprintk(XENLOG_DEBUG, ), which I think is more appropriate (as
it may spam the console when some stupid guest is running). Let me know
if you want to have a different loglevel.


I don't think this is more appropriate. gprintk will print the domain ID 
of the current domain, whilst this function will be called by the 
toolstack in the future.


Furthemore, if you look at the implementation of gprintk you will notice 
that it is basically turning into printk(XENLOG_GUEST...) and adding 
information of the current vCPU.


What matters for ratelimiting is XENLOG_GUEST.




+return -EBUSY;
+
+return remove_mapped_guest_device(temp);


Just above you removed the device from the RB-tree but this function may
fail and never free the memory. This means that memory will be leaked
leading to a potential denial of service.


So I fixed this case in v4, though there is still a tiny chance of a
memleak: if the MAPD(V=0) command fails. We can't free the ITT table
then, really, because it still belongs to the ITS. I don't think we can
do much about it, though.


This is a leak and even tiny is quite worrying. How do you plan to 
address this in the future? What is the best thing to do?



I free the other allocations of the memory now, anyway.


+}
+
+if ( compare_its_guest_devices(temp, guest_doorbell,
guest_devid) > 0 )
+new = &((*new)->rb_left);
+else
+new = &((*new)->rb_right);
+}
+
+if ( !valid )
+goto out_unlock;
+
+ret = -ENOMEM;
+
+/* An Interrupt Translation Table needs to be 256-byte aligned. */
+itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);
+if ( !itt_addr )
+goto out_unlock;
+
+dev = xzalloc(struct its_devices);
+if ( !dev )
+goto out_unlock;
+
+ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,


I don't understand why nr_events - 1. Can you explain?


Xen lacks an ilog2, so "fls" is the closest I could find. "fls" has this
slightly weird semantic (from the Linux source):
"Note fls(0) = 0, fls(1) = 1, fls(0x8000) = 32."
I think this translates into: "How many bits do I need to express this
number?". For our case the highest event number we need to encode is
"nr_events - 1", hence the subtraction.
So is it worth to introduce a:
static inline int ilog2_64(uint64_t n) ...
in xen/include/asm-arm/bitops.h to document this?


This might make easier to read the code.



[...]


+/* Removing any connections a domain had to any ITS in the system. */
+void gicv3_its_unmap_all_devices(struct domain *d)
+{
+struct rb_node *victim;
+struct its_devices *dev;
+
+/*
+ * This is an easily readable, yet inefficient implementation.
+ * It uses the provided iteration wrapper and erases each node,
which
+ * possibly triggers rebalancing.
+ * This seems overkill since we are going to abolish the whole
tree, but
+ * avoids an open-coded re-implementation of the traversal
functions with
+ * some recursive function calls.
+ * Performance does not matter here, since we are destroying a
domain.


Again, this is slightly untrue. Performance matter when destroying a
domain as Xen cannot be preempted. So if it takes too long, you will
have an impact on the overall system.


I reworded this sentence in v3, since you apparently misunderstood me.
By inefficient I meant sub-optimal, but this is not a _critical_ path,
so we don't care too much. The execution time is clearly bounded by the
number of devices. We simply shouldn't allow gazillion of devices on a
DomU if we care about those things.


This is a very naive way of thinking how domain destruction is working 
on 

Re: [Xen-devel] [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping

[Andre Przywara]

Hi,

On 22/03/17 17:29, Julien Grall wrote:
> Hi Andre,
> 
> On 16/03/17 11:20, Andre Przywara wrote:
>> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
>> those IDs, which we directly pass on to the host.
>> For this we have to map each device that Dom0 may request to a host
>> ITS device with the same identifier.
>> Allocate the respective memory and enter each device into an rbtree to
>> later be able to iterate over it or to easily teardown guests.
>>
>> Signed-off-by: Andre Przywara 
>> ---
>>  xen/arch/arm/gic-v3-its.c| 207
>> +++
>>  xen/arch/arm/vgic-v3.c   |   3 +
>>  xen/include/asm-arm/domain.h |   3 +
>>  xen/include/asm-arm/gic_v3_its.h |  18 
>>  4 files changed, 231 insertions(+)
>>
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index 5c11b0d..60b15b5 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -21,6 +21,8 @@
>>  #include 
>>  #include 
>>  #include 
>> +#include 
>> +#include 
>>  #include 
>>  #include 
>>  #include 
>> @@ -32,6 +34,17 @@
>>
>>  LIST_HEAD(host_its_list);
>>
>> +struct its_devices {
>> +struct rb_node rbnode;
>> +struct host_its *hw_its;
>> +void *itt_addr;
>> +paddr_t guest_doorbell;
> 
> I think it would be worth to explain in the commit message why you need
> the guest_doorbell in the struct its_devices and how you plan to use it.

In v4 I now also elaborated on the reason in a comment (before the
struct), which I deem more useful than something in the commit message.

>> +uint32_t host_devid;
>> +uint32_t guest_devid;
>> +uint32_t eventids;
>> +uint32_t *host_lpis;
>> +};
>> +
>>  bool gicv3_its_host_has_its(void)
>>  {
>>  return !list_empty(_its_list);
>> @@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its
>> *its, uint32_t collection_id,
>>  return its_send_command(its, cmd);
>>  }
>>
>> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
>> + uint8_t size_bits, paddr_t itt_addr,
>> bool valid)
>> +{
>> +uint64_t cmd[4];
>> +
>> +if ( valid )
>> +{
>> +ASSERT(size_bits < 32);
> 
> It would be better if you do the check against the real number in
> hardware (i.e GITS_TYPER.ID_bits).

Added in v4.

> 
>> +ASSERT(!(itt_addr & ~GENMASK(51, 8)));
>> +}
>> +cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
>> +cmd[1] = valid ? size_bits : 0x00;
> 
> This is really confusing. The check was not on the previous version. So
> why do you need that?

Admittedly I was taken away be some intention to check this here
properly. But since itt_addr and size are only valid with V=1, I removed
this in v3.

> Also, it would have been better to hide the "size - 1" in the helper
> avoiding to really on the caller to do the right thing.

I tend to agree, but then we have the awkward case where an unmap passes
0 in size, which then gets decremented by one. But you are right that
it's still saner this way, so I pass 1 now in the unmap call and do the
"-1" encoding in here.

>> +cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
> 
> Ditto about "valid? ...".

Removed in v3.

> [...]
> 
>> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t
>> doorbell_address)
>> +{
>> +struct host_its *hw_its;
>> +
>> +list_for_each_entry(hw_its, _its_list, entry)
>> +{
>> +if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
> 
> Why not storing the ITS address rather than the doorbell to avoid this
> check?

Because the doorbell address is a nice architectural property of MSIs in
general. And we need this check anyway, it's just the addition of the
doorbell offset that is different.

> [...]
> 
>> +int gicv3_its_map_guest_device(struct domain *d,
>> +   paddr_t host_doorbell, uint32_t
>> host_devid,
>> +   paddr_t guest_doorbell, uint32_t
>> guest_devid,
>> +   uint32_t nr_events, bool valid)
>> +{
>> +void *itt_addr = NULL;
>> +struct host_its *hw_its;
>> +struct its_devices *dev = NULL, *temp;
>> +struct rb_node **new = >arch.vgic.its_devices.rb_node, *parent
>> = NULL;
>> +int ret = -ENOENT;
>> +
>> +hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>> +if ( !hw_its )
>> +return ret;
>> +
>> +/* check for already existing mappings */
>> +spin_lock(>arch.vgic.its_devices_lock);
>> +while ( *new )
>> +{
>> +temp = rb_entry(*new, struct its_devices, rbnode);
>> +
>> +parent = *new;
>> +if ( !compare_its_guest_devices(temp, guest_doorbell,
>> guest_devid) )
>> +{
>> +if ( !valid )
>> +rb_erase(>rbnode, >arch.vgic.its_devices);
>> +
>> +spin_unlock(>arch.vgic.its_devices_lock);
>> +
>> +if ( valid )
> 
> Again, a printk(XENLOG_GUEST...) 

Re: [Xen-devel] [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping

[Andre Przywara]

Hi,

On 22/03/17 22:45, Stefano Stabellini wrote:
> On Thu, 16 Mar 2017, Andre Przywara wrote:
>> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
>> those IDs, which we directly pass on to the host.
>> For this we have to map each device that Dom0 may request to a host
>> ITS device with the same identifier.
>> Allocate the respective memory and enter each device into an rbtree to
>> later be able to iterate over it or to easily teardown guests.
>>
>> Signed-off-by: Andre Przywara 
>> ---
>>  xen/arch/arm/gic-v3-its.c| 207 
>> +++
>>  xen/arch/arm/vgic-v3.c   |   3 +
>>  xen/include/asm-arm/domain.h |   3 +
>>  xen/include/asm-arm/gic_v3_its.h |  18 
>>  4 files changed, 231 insertions(+)
>>
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index 5c11b0d..60b15b5 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -21,6 +21,8 @@
>>  #include 
>>  #include 
>>  #include 
>> +#include 
>> +#include 
>>  #include 
>>  #include 
>>  #include 
>> @@ -32,6 +34,17 @@
>>  
>>  LIST_HEAD(host_its_list);
>>  
>> +struct its_devices {
>> +struct rb_node rbnode;
>> +struct host_its *hw_its;
>> +void *itt_addr;
>> +paddr_t guest_doorbell;
>> +uint32_t host_devid;
>> +uint32_t guest_devid;
>> +uint32_t eventids;
>> +uint32_t *host_lpis;
>> +};
>> +
>>  bool gicv3_its_host_has_its(void)
>>  {
>>  return !list_empty(_its_list);
>> @@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, 
>> uint32_t collection_id,
>>  return its_send_command(its, cmd);
>>  }
>>  
>> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
>> + uint8_t size_bits, paddr_t itt_addr, bool 
>> valid)
>> +{
>> +uint64_t cmd[4];
>> +
>> +if ( valid )
>> +{
>> +ASSERT(size_bits < 32);
>> +ASSERT(!(itt_addr & ~GENMASK(51, 8)));
>> +}
>> +cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
>> +cmd[1] = valid ? size_bits : 0x00;
>> +cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
>> +cmd[3] = 0x00;
>> +
>> +return its_send_command(its, cmd);
>> +}
>> +
>>  /* Set up the (1:1) collection mapping for the given host CPU. */
>>  int gicv3_its_setup_collection(unsigned int cpu)
>>  {
>> @@ -379,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its 
>> *hw_its)
>>  devid_bits = min(devid_bits, max_its_device_bits);
>>  if ( reg & GITS_TYPER_PTA )
>>  hw_its->flags |= HOST_ITS_USES_PTA;
>> +hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
>>  
>>  for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
>>  {
>> @@ -428,6 +460,180 @@ int gicv3_its_init(void)
>>  return 0;
>>  }
>>  
>> +static int remove_mapped_guest_device(struct its_devices *dev)
>> +{
>> +int ret;
>> +
>> +if ( dev->hw_its )
>> +{
>> +int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, 
>> false);
>> +if ( ret )
>> +return ret;
>> +}
>> +
>> +ret = gicv3_its_wait_commands(dev->hw_its);
>> +if ( ret )
>> +return ret;
>> +
>> +xfree(dev->itt_addr);
>> +xfree(dev);
>> +
>> +return 0;
>> +}
>> +
>> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
>> +{
>> +struct host_its *hw_its;
>> +
>> +list_for_each_entry(hw_its, _its_list, entry)
> 
> Does this need to take a spinlock to protect host_its_list? I guess not
> because the list is not modified after boot?

Exactly, I added a comment in v4 explaining this.

>> +{
>> +if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
>> +return hw_its;
>> +}
>> +
>> +return NULL;
>> +}
>> +
>> +static int compare_its_guest_devices(struct its_devices *dev,
>> + paddr_t doorbell, uint32_t devid)
>> +{
>> +if ( dev->guest_doorbell < doorbell )
>> +return -1;
>> +
>> +if ( dev->guest_doorbell > doorbell )
>> +return 1;
>> +
>> +if ( dev->guest_devid < devid )
>> +return -1;
>> +
>> +if ( dev->guest_devid > devid )
>> +return 1;
>> +
>> +return 0;
>> +}
>> +
>> +/*
>> + * Map a hardware device, identified by a certain host ITS and its device ID
>> + * to domain d, a guest ITS (identified by its doorbell address) and device 
>> ID.
>> + * Also provide the number of events (MSIs) needed for that device.
>> + * This does not check if this particular hardware device is already mapped
>> + * at another domain, it is expected that this would be done by the caller.
>> + */
>> +int gicv3_its_map_guest_device(struct domain *d,
>> +   paddr_t host_doorbell, uint32_t host_devid,
>> +   paddr_t guest_doorbell, uint32_t guest_devid,
>> +   uint32_t nr_events, bool valid)
>> +{
>> +void *itt_addr = NULL;
>> 

Re: [Xen-devel] [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping

[Vijay Kilari]

Hi Andre,

On Thu, Mar 16, 2017 at 4:50 PM, Andre Przywara  wrote:
> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
> those IDs, which we directly pass on to the host.
> For this we have to map each device that Dom0 may request to a host
> ITS device with the same identifier.
> Allocate the respective memory and enter each device into an rbtree to
> later be able to iterate over it or to easily teardown guests.
>
> Signed-off-by: Andre Przywara 
> ---
>  xen/arch/arm/gic-v3-its.c| 207 
> +++
>  xen/arch/arm/vgic-v3.c   |   3 +
>  xen/include/asm-arm/domain.h |   3 +
>  xen/include/asm-arm/gic_v3_its.h |  18 
>  4 files changed, 231 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 5c11b0d..60b15b5 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -21,6 +21,8 @@
>  #include 
>  #include 
>  #include 
> +#include 
> +#include 
>  #include 
>  #include 
>  #include 
> @@ -32,6 +34,17 @@
>
>  LIST_HEAD(host_its_list);
>
> +struct its_devices {
> +struct rb_node rbnode;
> +struct host_its *hw_its;
> +void *itt_addr;
> +paddr_t guest_doorbell;
> +uint32_t host_devid;
> +uint32_t guest_devid;
> +uint32_t eventids;
> +uint32_t *host_lpis;
> +};
> +
>  bool gicv3_its_host_has_its(void)
>  {
>  return !list_empty(_its_list);
> @@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, 
> uint32_t collection_id,
>  return its_send_command(its, cmd);
>  }
>
> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
> + uint8_t size_bits, paddr_t itt_addr, bool valid)
> +{
> +uint64_t cmd[4];
> +
> +if ( valid )
> +{
> +ASSERT(size_bits < 32);
> +ASSERT(!(itt_addr & ~GENMASK(51, 8)));
> +}
> +cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
> +cmd[1] = valid ? size_bits : 0x00;
> +cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
> +cmd[3] = 0x00;
> +
> +return its_send_command(its, cmd);
> +}
> +
>  /* Set up the (1:1) collection mapping for the given host CPU. */
>  int gicv3_its_setup_collection(unsigned int cpu)
>  {
> @@ -379,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its 
> *hw_its)
>  devid_bits = min(devid_bits, max_its_device_bits);
>  if ( reg & GITS_TYPER_PTA )
>  hw_its->flags |= HOST_ITS_USES_PTA;
> +hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
>
>  for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
>  {
> @@ -428,6 +460,180 @@ int gicv3_its_init(void)
>  return 0;
>  }
>
> +static int remove_mapped_guest_device(struct its_devices *dev)
> +{
> +int ret;
> +
> +if ( dev->hw_its )
> +{
> +int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, 
> false);
> +if ( ret )
> +return ret;
> +}
> +
> +ret = gicv3_its_wait_commands(dev->hw_its);
> +if ( ret )
> +return ret;
> +
> +xfree(dev->itt_addr);
> +xfree(dev);
> +
> +return 0;
> +}
> +
> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
> +{
> +struct host_its *hw_its;
> +
> +list_for_each_entry(hw_its, _its_list, entry)
> +{
> +if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
> +return hw_its;
> +}
> +
> +return NULL;
> +}
> +
> +static int compare_its_guest_devices(struct its_devices *dev,
> + paddr_t doorbell, uint32_t devid)
> +{
> +if ( dev->guest_doorbell < doorbell )
> +return -1;
> +
> +if ( dev->guest_doorbell > doorbell )
> +return 1;
> +
> +if ( dev->guest_devid < devid )
> +return -1;
> +
> +if ( dev->guest_devid > devid )
> +return 1;
> +
> +return 0;
> +}
> +
> +/*
> + * Map a hardware device, identified by a certain host ITS and its device ID
> + * to domain d, a guest ITS (identified by its doorbell address) and device 
> ID.
> + * Also provide the number of events (MSIs) needed for that device.
> + * This does not check if this particular hardware device is already mapped
> + * at another domain, it is expected that this would be done by the caller.
> + */
> +int gicv3_its_map_guest_device(struct domain *d,
> +   paddr_t host_doorbell, uint32_t host_devid,
> +   paddr_t guest_doorbell, uint32_t guest_devid,
> +   uint32_t nr_events, bool valid)
> +{
> +void *itt_addr = NULL;
> +struct host_its *hw_its;
> +struct its_devices *dev = NULL, *temp;
> +struct rb_node **new = >arch.vgic.its_devices.rb_node, *parent = NULL;
> +int ret = -ENOENT;
> +
> +hw_its = gicv3_its_find_by_doorbell(host_doorbell);
> +if ( !hw_its )
> +return ret;
> +
> +/* check for already existing mappings */
> +

Re: [Xen-devel] [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping

[Stefano Stabellini]

On Thu, 16 Mar 2017, Andre Przywara wrote:
> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
> those IDs, which we directly pass on to the host.
> For this we have to map each device that Dom0 may request to a host
> ITS device with the same identifier.
> Allocate the respective memory and enter each device into an rbtree to
> later be able to iterate over it or to easily teardown guests.
> 
> Signed-off-by: Andre Przywara 
> ---
>  xen/arch/arm/gic-v3-its.c| 207 
> +++
>  xen/arch/arm/vgic-v3.c   |   3 +
>  xen/include/asm-arm/domain.h |   3 +
>  xen/include/asm-arm/gic_v3_its.h |  18 
>  4 files changed, 231 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 5c11b0d..60b15b5 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -21,6 +21,8 @@
>  #include 
>  #include 
>  #include 
> +#include 
> +#include 
>  #include 
>  #include 
>  #include 
> @@ -32,6 +34,17 @@
>  
>  LIST_HEAD(host_its_list);
>  
> +struct its_devices {
> +struct rb_node rbnode;
> +struct host_its *hw_its;
> +void *itt_addr;
> +paddr_t guest_doorbell;
> +uint32_t host_devid;
> +uint32_t guest_devid;
> +uint32_t eventids;
> +uint32_t *host_lpis;
> +};
> +
>  bool gicv3_its_host_has_its(void)
>  {
>  return !list_empty(_its_list);
> @@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, 
> uint32_t collection_id,
>  return its_send_command(its, cmd);
>  }
>  
> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
> + uint8_t size_bits, paddr_t itt_addr, bool valid)
> +{
> +uint64_t cmd[4];
> +
> +if ( valid )
> +{
> +ASSERT(size_bits < 32);
> +ASSERT(!(itt_addr & ~GENMASK(51, 8)));
> +}
> +cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
> +cmd[1] = valid ? size_bits : 0x00;
> +cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
> +cmd[3] = 0x00;
> +
> +return its_send_command(its, cmd);
> +}
> +
>  /* Set up the (1:1) collection mapping for the given host CPU. */
>  int gicv3_its_setup_collection(unsigned int cpu)
>  {
> @@ -379,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its 
> *hw_its)
>  devid_bits = min(devid_bits, max_its_device_bits);
>  if ( reg & GITS_TYPER_PTA )
>  hw_its->flags |= HOST_ITS_USES_PTA;
> +hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
>  
>  for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
>  {
> @@ -428,6 +460,180 @@ int gicv3_its_init(void)
>  return 0;
>  }
>  
> +static int remove_mapped_guest_device(struct its_devices *dev)
> +{
> +int ret;
> +
> +if ( dev->hw_its )
> +{
> +int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, 
> false);
> +if ( ret )
> +return ret;
> +}
> +
> +ret = gicv3_its_wait_commands(dev->hw_its);
> +if ( ret )
> +return ret;
> +
> +xfree(dev->itt_addr);
> +xfree(dev);
> +
> +return 0;
> +}
> +
> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
> +{
> +struct host_its *hw_its;
> +
> +list_for_each_entry(hw_its, _its_list, entry)

Does this need to take a spinlock to protect host_its_list? I guess not
because the list is not modified after boot?


> +{
> +if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
> +return hw_its;
> +}
> +
> +return NULL;
> +}
> +
> +static int compare_its_guest_devices(struct its_devices *dev,
> + paddr_t doorbell, uint32_t devid)
> +{
> +if ( dev->guest_doorbell < doorbell )
> +return -1;
> +
> +if ( dev->guest_doorbell > doorbell )
> +return 1;
> +
> +if ( dev->guest_devid < devid )
> +return -1;
> +
> +if ( dev->guest_devid > devid )
> +return 1;
> +
> +return 0;
> +}
> +
> +/*
> + * Map a hardware device, identified by a certain host ITS and its device ID
> + * to domain d, a guest ITS (identified by its doorbell address) and device 
> ID.
> + * Also provide the number of events (MSIs) needed for that device.
> + * This does not check if this particular hardware device is already mapped
> + * at another domain, it is expected that this would be done by the caller.
> + */
> +int gicv3_its_map_guest_device(struct domain *d,
> +   paddr_t host_doorbell, uint32_t host_devid,
> +   paddr_t guest_doorbell, uint32_t guest_devid,
> +   uint32_t nr_events, bool valid)
> +{
> +void *itt_addr = NULL;
> +struct host_its *hw_its;
> +struct its_devices *dev = NULL, *temp;
> +struct rb_node **new = >arch.vgic.its_devices.rb_node, *parent = NULL;
> +int ret = -ENOENT;
> +
> +hw_its = gicv3_its_find_by_doorbell(host_doorbell);
> +if ( !hw_its )

Re: [Xen-devel] [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping

[Julien Grall]

Hi Andre,

On 16/03/17 11:20, Andre Przywara wrote:

The ITS uses device IDs to map LPIs to a device. Dom0 will later use
those IDs, which we directly pass on to the host.
For this we have to map each device that Dom0 may request to a host
ITS device with the same identifier.
Allocate the respective memory and enter each device into an rbtree to
later be able to iterate over it or to easily teardown guests.

Signed-off-by: Andre Przywara 
---
 xen/arch/arm/gic-v3-its.c| 207 +++
 xen/arch/arm/vgic-v3.c   |   3 +
 xen/include/asm-arm/domain.h |   3 +
 xen/include/asm-arm/gic_v3_its.h |  18 
 4 files changed, 231 insertions(+)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 5c11b0d..60b15b5 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -21,6 +21,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -32,6 +34,17 @@

 LIST_HEAD(host_its_list);

+struct its_devices {
+struct rb_node rbnode;
+struct host_its *hw_its;
+void *itt_addr;
+paddr_t guest_doorbell;


I think it would be worth to explain in the commit message why you need 
the guest_doorbell in the struct its_devices and how you plan to use it.



+uint32_t host_devid;
+uint32_t guest_devid;
+uint32_t eventids;
+uint32_t *host_lpis;
+};
+
 bool gicv3_its_host_has_its(void)
 {
 return !list_empty(_its_list);
@@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, 
uint32_t collection_id,
 return its_send_command(its, cmd);
 }

+static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
+ uint8_t size_bits, paddr_t itt_addr, bool valid)
+{
+uint64_t cmd[4];
+
+if ( valid )
+{
+ASSERT(size_bits < 32);


It would be better if you do the check against the real number in 
hardware (i.e GITS_TYPER.ID_bits).




+ASSERT(!(itt_addr & ~GENMASK(51, 8)));
+}
+cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
+cmd[1] = valid ? size_bits : 0x00;


This is really confusing. The check was not on the previous version. So 
why do you need that?


Also, it would have been better to hide the "size - 1" in the helper 
avoiding to really on the caller to do the right thing.



+cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;


Ditto about "valid? ...".

[...]


+static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
+{
+struct host_its *hw_its;
+
+list_for_each_entry(hw_its, _its_list, entry)
+{
+if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )


Why not storing the ITS address rather than the doorbell to avoid this 
check?



[...]


+int gicv3_its_map_guest_device(struct domain *d,
+   paddr_t host_doorbell, uint32_t host_devid,
+   paddr_t guest_doorbell, uint32_t guest_devid,
+   uint32_t nr_events, bool valid)
+{
+void *itt_addr = NULL;
+struct host_its *hw_its;
+struct its_devices *dev = NULL, *temp;
+struct rb_node **new = >arch.vgic.its_devices.rb_node, *parent = NULL;
+int ret = -ENOENT;
+
+hw_its = gicv3_its_find_by_doorbell(host_doorbell);
+if ( !hw_its )
+return ret;
+
+/* check for already existing mappings */
+spin_lock(>arch.vgic.its_devices_lock);
+while ( *new )
+{
+temp = rb_entry(*new, struct its_devices, rbnode);
+
+parent = *new;
+if ( !compare_its_guest_devices(temp, guest_doorbell, guest_devid) )
+{
+if ( !valid )
+rb_erase(>rbnode, >arch.vgic.its_devices);
+
+spin_unlock(>arch.vgic.its_devices_lock);
+
+if ( valid )


Again, a printk(XENLOG_GUEST...) here would be useful to know which host 
DeviceID was associated to the guest DeviceID.



+return -EBUSY;
+
+return remove_mapped_guest_device(temp);


Just above you removed the device from the RB-tree but this function may 
fail and never free the memory. This means that memory will be leaked 
leading to a potential denial of service.



+}
+
+if ( compare_its_guest_devices(temp, guest_doorbell, guest_devid) > 0 )
+new = &((*new)->rb_left);
+else
+new = &((*new)->rb_right);
+}
+
+if ( !valid )
+goto out_unlock;
+
+ret = -ENOMEM;
+
+/* An Interrupt Translation Table needs to be 256-byte aligned. */
+itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);
+if ( !itt_addr )
+goto out_unlock;
+
+dev = xzalloc(struct its_devices);
+if ( !dev )
+goto out_unlock;
+
+ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,


I don't understand why nr_events - 1. Can you explain?

[...]


+/* Removing any connections a domain had to any ITS in the system. */
+void 

[Xen-devel] [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping

[Andre Przywara]

The ITS uses device IDs to map LPIs to a device. Dom0 will later use
those IDs, which we directly pass on to the host.
For this we have to map each device that Dom0 may request to a host
ITS device with the same identifier.
Allocate the respective memory and enter each device into an rbtree to
later be able to iterate over it or to easily teardown guests.

Signed-off-by: Andre Przywara 
---
 xen/arch/arm/gic-v3-its.c| 207 +++
 xen/arch/arm/vgic-v3.c   |   3 +
 xen/include/asm-arm/domain.h |   3 +
 xen/include/asm-arm/gic_v3_its.h |  18 
 4 files changed, 231 insertions(+)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 5c11b0d..60b15b5 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -21,6 +21,8 @@
 #include 
 #include 
 #include 
+#include 
+#include 
 #include 
 #include 
 #include 
@@ -32,6 +34,17 @@
 
 LIST_HEAD(host_its_list);
 
+struct its_devices {
+struct rb_node rbnode;
+struct host_its *hw_its;
+void *itt_addr;
+paddr_t guest_doorbell;
+uint32_t host_devid;
+uint32_t guest_devid;
+uint32_t eventids;
+uint32_t *host_lpis;
+};
+
 bool gicv3_its_host_has_its(void)
 {
 return !list_empty(_its_list);
@@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, 
uint32_t collection_id,
 return its_send_command(its, cmd);
 }
 
+static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
+ uint8_t size_bits, paddr_t itt_addr, bool valid)
+{
+uint64_t cmd[4];
+
+if ( valid )
+{
+ASSERT(size_bits < 32);
+ASSERT(!(itt_addr & ~GENMASK(51, 8)));
+}
+cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
+cmd[1] = valid ? size_bits : 0x00;
+cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
+cmd[3] = 0x00;
+
+return its_send_command(its, cmd);
+}
+
 /* Set up the (1:1) collection mapping for the given host CPU. */
 int gicv3_its_setup_collection(unsigned int cpu)
 {
@@ -379,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its 
*hw_its)
 devid_bits = min(devid_bits, max_its_device_bits);
 if ( reg & GITS_TYPER_PTA )
 hw_its->flags |= HOST_ITS_USES_PTA;
+hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
 
 for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
 {
@@ -428,6 +460,180 @@ int gicv3_its_init(void)
 return 0;
 }
 
+static int remove_mapped_guest_device(struct its_devices *dev)
+{
+int ret;
+
+if ( dev->hw_its )
+{
+int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, false);
+if ( ret )
+return ret;
+}
+
+ret = gicv3_its_wait_commands(dev->hw_its);
+if ( ret )
+return ret;
+
+xfree(dev->itt_addr);
+xfree(dev);
+
+return 0;
+}
+
+static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
+{
+struct host_its *hw_its;
+
+list_for_each_entry(hw_its, _its_list, entry)
+{
+if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
+return hw_its;
+}
+
+return NULL;
+}
+
+static int compare_its_guest_devices(struct its_devices *dev,
+ paddr_t doorbell, uint32_t devid)
+{
+if ( dev->guest_doorbell < doorbell )
+return -1;
+
+if ( dev->guest_doorbell > doorbell )
+return 1;
+
+if ( dev->guest_devid < devid )
+return -1;
+
+if ( dev->guest_devid > devid )
+return 1;
+
+return 0;
+}
+
+/*
+ * Map a hardware device, identified by a certain host ITS and its device ID
+ * to domain d, a guest ITS (identified by its doorbell address) and device ID.
+ * Also provide the number of events (MSIs) needed for that device.
+ * This does not check if this particular hardware device is already mapped
+ * at another domain, it is expected that this would be done by the caller.
+ */
+int gicv3_its_map_guest_device(struct domain *d,
+   paddr_t host_doorbell, uint32_t host_devid,
+   paddr_t guest_doorbell, uint32_t guest_devid,
+   uint32_t nr_events, bool valid)
+{
+void *itt_addr = NULL;
+struct host_its *hw_its;
+struct its_devices *dev = NULL, *temp;
+struct rb_node **new = >arch.vgic.its_devices.rb_node, *parent = NULL;
+int ret = -ENOENT;
+
+hw_its = gicv3_its_find_by_doorbell(host_doorbell);
+if ( !hw_its )
+return ret;
+
+/* check for already existing mappings */
+spin_lock(>arch.vgic.its_devices_lock);
+while ( *new )
+{
+temp = rb_entry(*new, struct its_devices, rbnode);
+
+parent = *new;
+if ( !compare_its_guest_devices(temp, guest_doorbell, guest_devid) )
+{
+if ( !valid )
+rb_erase(>rbnode, >arch.vgic.its_devices);
+
+spin_unlock(>arch.vgic.its_devices_lock);
+
+if (