Re: [PATCH 1/2] iommu: Add iommu_group_get/set_domain()

2020-07-06 Thread Lu Baolu

On 7/2/20 10:36 AM, Lu Baolu wrote:

Hi Robin,

On 7/1/20 8:18 PM, Robin Murphy wrote:

On 2020-07-01 08:32, Lu Baolu wrote:

Hi Robin,

On 2020/7/1 0:51, Robin Murphy wrote:

On 2020-06-30 02:03, Lu Baolu wrote:

Hi Robin,

On 6/29/20 7:56 PM, Robin Murphy wrote:

On 2020-06-27 04:15, Lu Baolu wrote:

The hardware assistant vfio mediated device is a use case of iommu
aux-domain. The interactions between vfio/mdev and iommu during mdev
creation and passthr are:

- Create a group for mdev with iommu_group_alloc();
- Add the device to the group with
 group = iommu_group_alloc();
 if (IS_ERR(group))
 return PTR_ERR(group);

 ret = iommu_group_add_device(group, >dev);
 if (!ret)
 dev_info(>dev, "MDEV: group_id = %d\n",
  iommu_group_id(group));
- Allocate an aux-domain
iommu_domain_alloc()
- Attach the aux-domain to the physical device from which the 
mdev is

   created.
iommu_aux_attach_device()

In the whole process, an iommu group was allocated for the mdev 
and an

iommu domain was attached to the group, but the group->domain leaves
NULL. As the result, iommu_get_domain_for_dev() doesn't work 
anymore.


This adds iommu_group_get/set_domain() so that group->domain 
could be
managed whenever a domain is attached or detached through the 
aux-domain

api's.


Letting external callers poke around directly in the internals of 
iommu_group doesn't look right to me.


Unfortunately, it seems that the vifo iommu abstraction is deeply 
bound

to the IOMMU subsystem. We can easily find other examples:

iommu_group_get/set_iommudata()
iommu_group_get/set_name()
...


Sure, but those are ways for users of a group to attach useful 
information of their own to it, that doesn't matter to the IOMMU 
subsystem itself. The interface you've proposed gives callers rich 
new opportunities to fundamentally break correct operation of the API:


 dom = iommu_domain_alloc();
 iommu_attach_group(dom, grp);
 ...
 iommu_group_set_domain(grp, NULL);
 // oops, leaked and can't ever detach properly now

or perhaps:

 grp = iommu_group_alloc();
 iommu_group_add_device(grp, dev);
 iommu_group_set_domain(grp, dom);
 ...
 iommu_detach_group(dom, grp);
 // oops, IOMMU driver might not handle this

If a regular device is attached to one or more aux domains for 
PASID use, iommu_get_domain_for_dev() is still going to return the 
primary domain, so why should it be expected to behave differently 
for mediated


Unlike the normal device attach, we will encounter two devices when it
comes to aux-domain.

- Parent physical device - this might be, for example, a PCIe device
with PASID feature support, hence it is able to tag an unique PASID
for DMA transfers originated from its subset. The device driver hence
is able to wrapper this subset into an isolated:

- Mediated device - a fake device created by the device driver 
mentioned

above.

Yes. All you mentioned are right for the parent device. But for 
mediated

device, iommu_get_domain_for_dev() doesn't work even it has an valid
iommu_group and iommu_domain.

iommu_get_domain_for_dev() is a necessary interface for device drivers
which want to support aux-domain. For example,


Only if they want to follow this very specific notion of using 
made-up devices and groups to represent aux attachments. Even if a 
driver managing its own aux domains entirely privately does create 
child devices for them, it's not like it can't keep its domain 
pointers in drvdata if it wants to ;)


Let's not conflate the current implementation of vfio_mdev with the 
general concepts involved here.



   struct iommu_domain *domain;
   struct device *dev = mdev_dev(mdev);
   unsigned long pasid;

   domain = iommu_get_domain_for_dev(dev);
   if (!domain)
   return -ENODEV;

   pasid = iommu_aux_get_pasid(domain, dev->parent);
   if (pasid == IOASID_INVALID)
   return -EINVAL;

   /* Program the device context with the PASID value */
   

Without this fix, iommu_get_domain_for_dev() always returns NULL 
and the

device driver has no means to support aux-domain.


So either the IOMMU API itself is missing the ability to do the 
right thing internally, or the mdev layer isn't using it 
appropriately. Either way, simply punching holes in the API for mdev 
to hack around its own mess doesn't seem like the best thing to do.


The initial impression I got was that it's implicitly assumed here 
that the mdev itself is attached to exactly one aux domain and 
nothing else, at which point I would wonder why it's using aux at 
all, but are you saying that in fact no attach happens with the mdev 
group either way, only to the parent device?


I'll admit I'm not hugely familiar with any of this, but it seems to 
me that the logical flow should be:


 - allocate domain
 - attach as aux to parent
 - 

Re: [PATCH 1/2] iommu: Add iommu_group_get/set_domain()

2020-07-01 Thread Lu Baolu

Hi Robin,

On 7/1/20 8:18 PM, Robin Murphy wrote:

On 2020-07-01 08:32, Lu Baolu wrote:

Hi Robin,

On 2020/7/1 0:51, Robin Murphy wrote:

On 2020-06-30 02:03, Lu Baolu wrote:

Hi Robin,

On 6/29/20 7:56 PM, Robin Murphy wrote:

On 2020-06-27 04:15, Lu Baolu wrote:

The hardware assistant vfio mediated device is a use case of iommu
aux-domain. The interactions between vfio/mdev and iommu during mdev
creation and passthr are:

- Create a group for mdev with iommu_group_alloc();
- Add the device to the group with
 group = iommu_group_alloc();
 if (IS_ERR(group))
 return PTR_ERR(group);

 ret = iommu_group_add_device(group, >dev);
 if (!ret)
 dev_info(>dev, "MDEV: group_id = %d\n",
  iommu_group_id(group));
- Allocate an aux-domain
iommu_domain_alloc()
- Attach the aux-domain to the physical device from which the mdev is
   created.
iommu_aux_attach_device()

In the whole process, an iommu group was allocated for the mdev 
and an

iommu domain was attached to the group, but the group->domain leaves
NULL. As the result, iommu_get_domain_for_dev() doesn't work anymore.

This adds iommu_group_get/set_domain() so that group->domain could be
managed whenever a domain is attached or detached through the 
aux-domain

api's.


Letting external callers poke around directly in the internals of 
iommu_group doesn't look right to me.


Unfortunately, it seems that the vifo iommu abstraction is deeply bound
to the IOMMU subsystem. We can easily find other examples:

iommu_group_get/set_iommudata()
iommu_group_get/set_name()
...


Sure, but those are ways for users of a group to attach useful 
information of their own to it, that doesn't matter to the IOMMU 
subsystem itself. The interface you've proposed gives callers rich 
new opportunities to fundamentally break correct operation of the API:


 dom = iommu_domain_alloc();
 iommu_attach_group(dom, grp);
 ...
 iommu_group_set_domain(grp, NULL);
 // oops, leaked and can't ever detach properly now

or perhaps:

 grp = iommu_group_alloc();
 iommu_group_add_device(grp, dev);
 iommu_group_set_domain(grp, dom);
 ...
 iommu_detach_group(dom, grp);
 // oops, IOMMU driver might not handle this

If a regular device is attached to one or more aux domains for 
PASID use, iommu_get_domain_for_dev() is still going to return the 
primary domain, so why should it be expected to behave differently 
for mediated


Unlike the normal device attach, we will encounter two devices when it
comes to aux-domain.

- Parent physical device - this might be, for example, a PCIe device
with PASID feature support, hence it is able to tag an unique PASID
for DMA transfers originated from its subset. The device driver hence
is able to wrapper this subset into an isolated:

- Mediated device - a fake device created by the device driver 
mentioned

above.

Yes. All you mentioned are right for the parent device. But for 
mediated

device, iommu_get_domain_for_dev() doesn't work even it has an valid
iommu_group and iommu_domain.

iommu_get_domain_for_dev() is a necessary interface for device drivers
which want to support aux-domain. For example,


Only if they want to follow this very specific notion of using 
made-up devices and groups to represent aux attachments. Even if a 
driver managing its own aux domains entirely privately does create 
child devices for them, it's not like it can't keep its domain 
pointers in drvdata if it wants to ;)


Let's not conflate the current implementation of vfio_mdev with the 
general concepts involved here.



   struct iommu_domain *domain;
   struct device *dev = mdev_dev(mdev);
   unsigned long pasid;

   domain = iommu_get_domain_for_dev(dev);
   if (!domain)
   return -ENODEV;

   pasid = iommu_aux_get_pasid(domain, dev->parent);
   if (pasid == IOASID_INVALID)
   return -EINVAL;

   /* Program the device context with the PASID value */
   

Without this fix, iommu_get_domain_for_dev() always returns NULL and 
the

device driver has no means to support aux-domain.


So either the IOMMU API itself is missing the ability to do the right 
thing internally, or the mdev layer isn't using it appropriately. 
Either way, simply punching holes in the API for mdev to hack around 
its own mess doesn't seem like the best thing to do.


The initial impression I got was that it's implicitly assumed here 
that the mdev itself is attached to exactly one aux domain and 
nothing else, at which point I would wonder why it's using aux at 
all, but are you saying that in fact no attach happens with the mdev 
group either way, only to the parent device?


I'll admit I'm not hugely familiar with any of this, but it seems to 
me that the logical flow should be:


 - allocate domain
 - attach as aux to parent
 - retrieve aux domain PASID
 - create mdev 

Re: [PATCH 1/2] iommu: Add iommu_group_get/set_domain()

2020-07-01 Thread Lu Baolu

Hello,

On 7/1/20 8:18 PM, Robin Murphy wrote:

On 2020-07-01 08:32, Lu Baolu wrote:

Hi Robin,

On 2020/7/1 0:51, Robin Murphy wrote:

On 2020-06-30 02:03, Lu Baolu wrote:

Hi Robin,

On 6/29/20 7:56 PM, Robin Murphy wrote:

On 2020-06-27 04:15, Lu Baolu wrote:

The hardware assistant vfio mediated device is a use case of iommu
aux-domain. The interactions between vfio/mdev and iommu during mdev
creation and passthr are:

- Create a group for mdev with iommu_group_alloc();
- Add the device to the group with
 group = iommu_group_alloc();
 if (IS_ERR(group))
 return PTR_ERR(group);

 ret = iommu_group_add_device(group, >dev);
 if (!ret)
 dev_info(>dev, "MDEV: group_id = %d\n",
  iommu_group_id(group));
- Allocate an aux-domain
iommu_domain_alloc()
- Attach the aux-domain to the physical device from which the mdev is
   created.
iommu_aux_attach_device()

In the whole process, an iommu group was allocated for the mdev 
and an

iommu domain was attached to the group, but the group->domain leaves
NULL. As the result, iommu_get_domain_for_dev() doesn't work anymore.

This adds iommu_group_get/set_domain() so that group->domain could be
managed whenever a domain is attached or detached through the 
aux-domain

api's.


Letting external callers poke around directly in the internals of 
iommu_group doesn't look right to me.


Unfortunately, it seems that the vifo iommu abstraction is deeply bound
to the IOMMU subsystem. We can easily find other examples:

iommu_group_get/set_iommudata()
iommu_group_get/set_name()
...


Sure, but those are ways for users of a group to attach useful 
information of their own to it, that doesn't matter to the IOMMU 
subsystem itself. The interface you've proposed gives callers rich 
new opportunities to fundamentally break correct operation of the API:


 dom = iommu_domain_alloc();
 iommu_attach_group(dom, grp);
 ...
 iommu_group_set_domain(grp, NULL);
 // oops, leaked and can't ever detach properly now

or perhaps:

 grp = iommu_group_alloc();
 iommu_group_add_device(grp, dev);
 iommu_group_set_domain(grp, dom);
 ...
 iommu_detach_group(dom, grp);
 // oops, IOMMU driver might not handle this

If a regular device is attached to one or more aux domains for 
PASID use, iommu_get_domain_for_dev() is still going to return the 
primary domain, so why should it be expected to behave differently 
for mediated


Unlike the normal device attach, we will encounter two devices when it
comes to aux-domain.

- Parent physical device - this might be, for example, a PCIe device
with PASID feature support, hence it is able to tag an unique PASID
for DMA transfers originated from its subset. The device driver hence
is able to wrapper this subset into an isolated:

- Mediated device - a fake device created by the device driver 
mentioned

above.

Yes. All you mentioned are right for the parent device. But for 
mediated

device, iommu_get_domain_for_dev() doesn't work even it has an valid
iommu_group and iommu_domain.

iommu_get_domain_for_dev() is a necessary interface for device drivers
which want to support aux-domain. For example,


Only if they want to follow this very specific notion of using 
made-up devices and groups to represent aux attachments. Even if a 
driver managing its own aux domains entirely privately does create 
child devices for them, it's not like it can't keep its domain 
pointers in drvdata if it wants to ;)


Let's not conflate the current implementation of vfio_mdev with the 
general concepts involved here.



   struct iommu_domain *domain;
   struct device *dev = mdev_dev(mdev);
   unsigned long pasid;

   domain = iommu_get_domain_for_dev(dev);
   if (!domain)
   return -ENODEV;

   pasid = iommu_aux_get_pasid(domain, dev->parent);
   if (pasid == IOASID_INVALID)
   return -EINVAL;

   /* Program the device context with the PASID value */
   

Without this fix, iommu_get_domain_for_dev() always returns NULL and 
the

device driver has no means to support aux-domain.


So either the IOMMU API itself is missing the ability to do the right 
thing internally, or the mdev layer isn't using it appropriately. 
Either way, simply punching holes in the API for mdev to hack around 
its own mess doesn't seem like the best thing to do.


The initial impression I got was that it's implicitly assumed here 
that the mdev itself is attached to exactly one aux domain and 
nothing else, at which point I would wonder why it's using aux at 
all, but are you saying that in fact no attach happens with the mdev 
group either way, only to the parent device?


I'll admit I'm not hugely familiar with any of this, but it seems to 
me that the logical flow should be:


 - allocate domain
 - attach as aux to parent
 - retrieve aux domain PASID
 - create mdev 

Re: [PATCH 1/2] iommu: Add iommu_group_get/set_domain()

2020-07-01 Thread Robin Murphy

On 2020-07-01 08:32, Lu Baolu wrote:

Hi Robin,

On 2020/7/1 0:51, Robin Murphy wrote:

On 2020-06-30 02:03, Lu Baolu wrote:

Hi Robin,

On 6/29/20 7:56 PM, Robin Murphy wrote:

On 2020-06-27 04:15, Lu Baolu wrote:

The hardware assistant vfio mediated device is a use case of iommu
aux-domain. The interactions between vfio/mdev and iommu during mdev
creation and passthr are:

- Create a group for mdev with iommu_group_alloc();
- Add the device to the group with
 group = iommu_group_alloc();
 if (IS_ERR(group))
 return PTR_ERR(group);

 ret = iommu_group_add_device(group, >dev);
 if (!ret)
 dev_info(>dev, "MDEV: group_id = %d\n",
  iommu_group_id(group));
- Allocate an aux-domain
iommu_domain_alloc()
- Attach the aux-domain to the physical device from which the mdev is
   created.
iommu_aux_attach_device()

In the whole process, an iommu group was allocated for the mdev and an
iommu domain was attached to the group, but the group->domain leaves
NULL. As the result, iommu_get_domain_for_dev() doesn't work anymore.

This adds iommu_group_get/set_domain() so that group->domain could be
managed whenever a domain is attached or detached through the 
aux-domain

api's.


Letting external callers poke around directly in the internals of 
iommu_group doesn't look right to me.


Unfortunately, it seems that the vifo iommu abstraction is deeply bound
to the IOMMU subsystem. We can easily find other examples:

iommu_group_get/set_iommudata()
iommu_group_get/set_name()
...


Sure, but those are ways for users of a group to attach useful 
information of their own to it, that doesn't matter to the IOMMU 
subsystem itself. The interface you've proposed gives callers rich new 
opportunities to fundamentally break correct operation of the API:


 dom = iommu_domain_alloc();
 iommu_attach_group(dom, grp);
 ...
 iommu_group_set_domain(grp, NULL);
 // oops, leaked and can't ever detach properly now

or perhaps:

 grp = iommu_group_alloc();
 iommu_group_add_device(grp, dev);
 iommu_group_set_domain(grp, dom);
 ...
 iommu_detach_group(dom, grp);
 // oops, IOMMU driver might not handle this

If a regular device is attached to one or more aux domains for PASID 
use, iommu_get_domain_for_dev() is still going to return the primary 
domain, so why should it be expected to behave differently for mediated


Unlike the normal device attach, we will encounter two devices when it
comes to aux-domain.

- Parent physical device - this might be, for example, a PCIe device
with PASID feature support, hence it is able to tag an unique PASID
for DMA transfers originated from its subset. The device driver hence
is able to wrapper this subset into an isolated:

- Mediated device - a fake device created by the device driver mentioned
above.

Yes. All you mentioned are right for the parent device. But for mediated
device, iommu_get_domain_for_dev() doesn't work even it has an valid
iommu_group and iommu_domain.

iommu_get_domain_for_dev() is a necessary interface for device drivers
which want to support aux-domain. For example,


Only if they want to follow this very specific notion of using made-up 
devices and groups to represent aux attachments. Even if a driver 
managing its own aux domains entirely privately does create child 
devices for them, it's not like it can't keep its domain pointers in 
drvdata if it wants to ;)


Let's not conflate the current implementation of vfio_mdev with the 
general concepts involved here.



   struct iommu_domain *domain;
   struct device *dev = mdev_dev(mdev);
   unsigned long pasid;

   domain = iommu_get_domain_for_dev(dev);
   if (!domain)
   return -ENODEV;

   pasid = iommu_aux_get_pasid(domain, dev->parent);
   if (pasid == IOASID_INVALID)
   return -EINVAL;

   /* Program the device context with the PASID value */
   

Without this fix, iommu_get_domain_for_dev() always returns NULL and the
device driver has no means to support aux-domain.


So either the IOMMU API itself is missing the ability to do the right 
thing internally, or the mdev layer isn't using it appropriately. 
Either way, simply punching holes in the API for mdev to hack around 
its own mess doesn't seem like the best thing to do.


The initial impression I got was that it's implicitly assumed here 
that the mdev itself is attached to exactly one aux domain and nothing 
else, at which point I would wonder why it's using aux at all, but are 
you saying that in fact no attach happens with the mdev group either 
way, only to the parent device?


I'll admit I'm not hugely familiar with any of this, but it seems to 
me that the logical flow should be:


 - allocate domain
 - attach as aux to parent
 - retrieve aux domain PASID
 - create mdev child based on PASID
 - attach mdev to domain (normally)


Re: [PATCH 1/2] iommu: Add iommu_group_get/set_domain()

2020-07-01 Thread Lu Baolu

Hi Robin,

On 2020/7/1 0:51, Robin Murphy wrote:

On 2020-06-30 02:03, Lu Baolu wrote:

Hi Robin,

On 6/29/20 7:56 PM, Robin Murphy wrote:

On 2020-06-27 04:15, Lu Baolu wrote:

The hardware assistant vfio mediated device is a use case of iommu
aux-domain. The interactions between vfio/mdev and iommu during mdev
creation and passthr are:

- Create a group for mdev with iommu_group_alloc();
- Add the device to the group with
 group = iommu_group_alloc();
 if (IS_ERR(group))
 return PTR_ERR(group);

 ret = iommu_group_add_device(group, >dev);
 if (!ret)
 dev_info(>dev, "MDEV: group_id = %d\n",
  iommu_group_id(group));
- Allocate an aux-domain
iommu_domain_alloc()
- Attach the aux-domain to the physical device from which the mdev is
   created.
iommu_aux_attach_device()

In the whole process, an iommu group was allocated for the mdev and an
iommu domain was attached to the group, but the group->domain leaves
NULL. As the result, iommu_get_domain_for_dev() doesn't work anymore.

This adds iommu_group_get/set_domain() so that group->domain could be
managed whenever a domain is attached or detached through the 
aux-domain

api's.


Letting external callers poke around directly in the internals of 
iommu_group doesn't look right to me.


Unfortunately, it seems that the vifo iommu abstraction is deeply bound
to the IOMMU subsystem. We can easily find other examples:

iommu_group_get/set_iommudata()
iommu_group_get/set_name()
...


Sure, but those are ways for users of a group to attach useful 
information of their own to it, that doesn't matter to the IOMMU 
subsystem itself. The interface you've proposed gives callers rich new 
opportunities to fundamentally break correct operation of the API:


 dom = iommu_domain_alloc();
 iommu_attach_group(dom, grp);
 ...
 iommu_group_set_domain(grp, NULL);
 // oops, leaked and can't ever detach properly now

or perhaps:

 grp = iommu_group_alloc();
 iommu_group_add_device(grp, dev);
 iommu_group_set_domain(grp, dom);
 ...
 iommu_detach_group(dom, grp);
 // oops, IOMMU driver might not handle this

If a regular device is attached to one or more aux domains for PASID 
use, iommu_get_domain_for_dev() is still going to return the primary 
domain, so why should it be expected to behave differently for mediated


Unlike the normal device attach, we will encounter two devices when it
comes to aux-domain.

- Parent physical device - this might be, for example, a PCIe device
with PASID feature support, hence it is able to tag an unique PASID
for DMA transfers originated from its subset. The device driver hence
is able to wrapper this subset into an isolated:

- Mediated device - a fake device created by the device driver mentioned
above.

Yes. All you mentioned are right for the parent device. But for mediated
device, iommu_get_domain_for_dev() doesn't work even it has an valid
iommu_group and iommu_domain.

iommu_get_domain_for_dev() is a necessary interface for device drivers
which want to support aux-domain. For example,


Only if they want to follow this very specific notion of using made-up 
devices and groups to represent aux attachments. Even if a driver 
managing its own aux domains entirely privately does create child 
devices for them, it's not like it can't keep its domain pointers in 
drvdata if it wants to ;)


Let's not conflate the current implementation of vfio_mdev with the 
general concepts involved here.



   struct iommu_domain *domain;
   struct device *dev = mdev_dev(mdev);
   unsigned long pasid;

   domain = iommu_get_domain_for_dev(dev);
   if (!domain)
   return -ENODEV;

   pasid = iommu_aux_get_pasid(domain, dev->parent);
   if (pasid == IOASID_INVALID)
   return -EINVAL;

   /* Program the device context with the PASID value */
   

Without this fix, iommu_get_domain_for_dev() always returns NULL and the
device driver has no means to support aux-domain.


So either the IOMMU API itself is missing the ability to do the right 
thing internally, or the mdev layer isn't using it appropriately. Either 
way, simply punching holes in the API for mdev to hack around its own 
mess doesn't seem like the best thing to do.


The initial impression I got was that it's implicitly assumed here that 
the mdev itself is attached to exactly one aux domain and nothing else, 
at which point I would wonder why it's using aux at all, but are you 
saying that in fact no attach happens with the mdev group either way, 
only to the parent device?


I'll admit I'm not hugely familiar with any of this, but it seems to me 
that the logical flow should be:


 - allocate domain
 - attach as aux to parent
 - retrieve aux domain PASID
 - create mdev child based on PASID
 - attach mdev to domain (normally)

Of course that might require giving 

Re: [PATCH 1/2] iommu: Add iommu_group_get/set_domain()

2020-06-30 Thread Robin Murphy

On 2020-06-30 02:03, Lu Baolu wrote:

Hi Robin,

On 6/29/20 7:56 PM, Robin Murphy wrote:

On 2020-06-27 04:15, Lu Baolu wrote:

The hardware assistant vfio mediated device is a use case of iommu
aux-domain. The interactions between vfio/mdev and iommu during mdev
creation and passthr are:

- Create a group for mdev with iommu_group_alloc();
- Add the device to the group with
 group = iommu_group_alloc();
 if (IS_ERR(group))
 return PTR_ERR(group);

 ret = iommu_group_add_device(group, >dev);
 if (!ret)
 dev_info(>dev, "MDEV: group_id = %d\n",
  iommu_group_id(group));
- Allocate an aux-domain
iommu_domain_alloc()
- Attach the aux-domain to the physical device from which the mdev is
   created.
iommu_aux_attach_device()

In the whole process, an iommu group was allocated for the mdev and an
iommu domain was attached to the group, but the group->domain leaves
NULL. As the result, iommu_get_domain_for_dev() doesn't work anymore.

This adds iommu_group_get/set_domain() so that group->domain could be
managed whenever a domain is attached or detached through the aux-domain
api's.


Letting external callers poke around directly in the internals of 
iommu_group doesn't look right to me.


Unfortunately, it seems that the vifo iommu abstraction is deeply bound
to the IOMMU subsystem. We can easily find other examples:

iommu_group_get/set_iommudata()
iommu_group_get/set_name()
...


Sure, but those are ways for users of a group to attach useful 
information of their own to it, that doesn't matter to the IOMMU 
subsystem itself. The interface you've proposed gives callers rich new 
opportunities to fundamentally break correct operation of the API:


dom = iommu_domain_alloc();
iommu_attach_group(dom, grp);
...
iommu_group_set_domain(grp, NULL);
// oops, leaked and can't ever detach properly now

or perhaps:

grp = iommu_group_alloc();
iommu_group_add_device(grp, dev);
iommu_group_set_domain(grp, dom);
...
iommu_detach_group(dom, grp);
// oops, IOMMU driver might not handle this

If a regular device is attached to one or more aux domains for PASID 
use, iommu_get_domain_for_dev() is still going to return the primary 
domain, so why should it be expected to behave differently for mediated


Unlike the normal device attach, we will encounter two devices when it
comes to aux-domain.

- Parent physical device - this might be, for example, a PCIe device
with PASID feature support, hence it is able to tag an unique PASID
for DMA transfers originated from its subset. The device driver hence
is able to wrapper this subset into an isolated:

- Mediated device - a fake device created by the device driver mentioned
above.

Yes. All you mentioned are right for the parent device. But for mediated
device, iommu_get_domain_for_dev() doesn't work even it has an valid
iommu_group and iommu_domain.

iommu_get_domain_for_dev() is a necessary interface for device drivers
which want to support aux-domain. For example,


Only if they want to follow this very specific notion of using made-up 
devices and groups to represent aux attachments. Even if a driver 
managing its own aux domains entirely privately does create child 
devices for them, it's not like it can't keep its domain pointers in 
drvdata if it wants to ;)


Let's not conflate the current implementation of vfio_mdev with the 
general concepts involved here.



   struct iommu_domain *domain;
   struct device *dev = mdev_dev(mdev);
   unsigned long pasid;

   domain = iommu_get_domain_for_dev(dev);
   if (!domain)
   return -ENODEV;

   pasid = iommu_aux_get_pasid(domain, dev->parent);
   if (pasid == IOASID_INVALID)
   return -EINVAL;

   /* Program the device context with the PASID value */
   

Without this fix, iommu_get_domain_for_dev() always returns NULL and the
device driver has no means to support aux-domain.


So either the IOMMU API itself is missing the ability to do the right 
thing internally, or the mdev layer isn't using it appropriately. Either 
way, simply punching holes in the API for mdev to hack around its own 
mess doesn't seem like the best thing to do.


The initial impression I got was that it's implicitly assumed here that 
the mdev itself is attached to exactly one aux domain and nothing else, 
at which point I would wonder why it's using aux at all, but are you 
saying that in fact no attach happens with the mdev group either way, 
only to the parent device?


I'll admit I'm not hugely familiar with any of this, but it seems to me 
that the logical flow should be:


- allocate domain
- attach as aux to parent
- retrieve aux domain PASID
- create mdev child based on PASID
- attach mdev to domain (normally)

Of course that might require giving the 

Re: [PATCH 1/2] iommu: Add iommu_group_get/set_domain()

2020-06-29 Thread Lu Baolu

Hi Robin,

On 6/29/20 7:56 PM, Robin Murphy wrote:

On 2020-06-27 04:15, Lu Baolu wrote:

The hardware assistant vfio mediated device is a use case of iommu
aux-domain. The interactions between vfio/mdev and iommu during mdev
creation and passthr are:

- Create a group for mdev with iommu_group_alloc();
- Add the device to the group with
 group = iommu_group_alloc();
 if (IS_ERR(group))
 return PTR_ERR(group);

 ret = iommu_group_add_device(group, >dev);
 if (!ret)
 dev_info(>dev, "MDEV: group_id = %d\n",
  iommu_group_id(group));
- Allocate an aux-domain
iommu_domain_alloc()
- Attach the aux-domain to the physical device from which the mdev is
   created.
iommu_aux_attach_device()

In the whole process, an iommu group was allocated for the mdev and an
iommu domain was attached to the group, but the group->domain leaves
NULL. As the result, iommu_get_domain_for_dev() doesn't work anymore.

This adds iommu_group_get/set_domain() so that group->domain could be
managed whenever a domain is attached or detached through the aux-domain
api's.


Letting external callers poke around directly in the internals of 
iommu_group doesn't look right to me.


Unfortunately, it seems that the vifo iommu abstraction is deeply bound
to the IOMMU subsystem. We can easily find other examples:

iommu_group_get/set_iommudata()
iommu_group_get/set_name()
...



If a regular device is attached to one or more aux domains for PASID 
use, iommu_get_domain_for_dev() is still going to return the primary 
domain, so why should it be expected to behave differently for mediated


Unlike the normal device attach, we will encounter two devices when it
comes to aux-domain.

- Parent physical device - this might be, for example, a PCIe device
with PASID feature support, hence it is able to tag an unique PASID
for DMA transfers originated from its subset. The device driver hence
is able to wrapper this subset into an isolated:

- Mediated device - a fake device created by the device driver mentioned
above.

Yes. All you mentioned are right for the parent device. But for mediated
device, iommu_get_domain_for_dev() doesn't work even it has an valid
iommu_group and iommu_domain.

iommu_get_domain_for_dev() is a necessary interface for device drivers
which want to support aux-domain. For example,

  struct iommu_domain *domain;
  struct device *dev = mdev_dev(mdev);
  unsigned long pasid;

  domain = iommu_get_domain_for_dev(dev);
  if (!domain)
  return -ENODEV;

  pasid = iommu_aux_get_pasid(domain, dev->parent);
  if (pasid == IOASID_INVALID)
  return -EINVAL;

  /* Program the device context with the PASID value */
  

Without this fix, iommu_get_domain_for_dev() always returns NULL and the
device driver has no means to support aux-domain.

Best regards,
baolu

devices? AFAICS it's perfectly legitimate to have no primary domain if 
traffic-without-PASID is invalid.


Robin.

___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 1/2] iommu: Add iommu_group_get/set_domain()

2020-06-29 Thread Robin Murphy

On 2020-06-27 04:15, Lu Baolu wrote:

The hardware assistant vfio mediated device is a use case of iommu
aux-domain. The interactions between vfio/mdev and iommu during mdev
creation and passthr are:

- Create a group for mdev with iommu_group_alloc();
- Add the device to the group with
 group = iommu_group_alloc();
 if (IS_ERR(group))
 return PTR_ERR(group);

 ret = iommu_group_add_device(group, >dev);
 if (!ret)
 dev_info(>dev, "MDEV: group_id = %d\n",
  iommu_group_id(group));
- Allocate an aux-domain
iommu_domain_alloc()
- Attach the aux-domain to the physical device from which the mdev is
   created.
iommu_aux_attach_device()

In the whole process, an iommu group was allocated for the mdev and an
iommu domain was attached to the group, but the group->domain leaves
NULL. As the result, iommu_get_domain_for_dev() doesn't work anymore.

This adds iommu_group_get/set_domain() so that group->domain could be
managed whenever a domain is attached or detached through the aux-domain
api's.


Letting external callers poke around directly in the internals of 
iommu_group doesn't look right to me.


If a regular device is attached to one or more aux domains for PASID 
use, iommu_get_domain_for_dev() is still going to return the primary 
domain, so why should it be expected to behave differently for mediated 
devices? AFAICS it's perfectly legitimate to have no primary domain if 
traffic-without-PASID is invalid.


Robin.


Fixes: 7bd50f0cd2fd5 ("vfio/type1: Add domain at(de)taching group helpers")
Signed-off-by: Lu Baolu 
---
  drivers/iommu/iommu.c | 28 
  include/linux/iommu.h | 14 ++
  2 files changed, 42 insertions(+)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index d43120eb1dc5..e2b665303d70 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -715,6 +715,34 @@ int iommu_group_set_name(struct iommu_group *group, const 
char *name)
  }
  EXPORT_SYMBOL_GPL(iommu_group_set_name);
  
+/**

+ * iommu_group_get_domain - get domain of a group
+ * @group: the group
+ *
+ * This is called to get the domain of a group.
+ */
+struct iommu_domain *iommu_group_get_domain(struct iommu_group *group)
+{
+   return group->domain;
+}
+EXPORT_SYMBOL_GPL(iommu_group_get_domain);
+
+/**
+ * iommu_group_set_domain - set domain for a group
+ * @group: the group
+ * @domain: iommu domain
+ *
+ * This is called to set the domain for a group. In aux-domain case, a domain
+ * might attach or detach to an iommu group through the aux-domain apis, but
+ * the group->domain doesn't get a chance to be updated there.
+ */
+void iommu_group_set_domain(struct iommu_group *group,
+   struct iommu_domain *domain)
+{
+   group->domain = domain;
+}
+EXPORT_SYMBOL_GPL(iommu_group_set_domain);
+
  static int iommu_create_device_direct_mappings(struct iommu_group *group,
   struct device *dev)
  {
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 5f0b7859d2eb..ff88d548a870 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -496,6 +496,9 @@ extern void iommu_group_set_iommudata(struct iommu_group 
*group,
  void *iommu_data,
  void (*release)(void *iommu_data));
  extern int iommu_group_set_name(struct iommu_group *group, const char *name);
+extern struct iommu_domain *iommu_group_get_domain(struct iommu_group *group);
+extern void iommu_group_set_domain(struct iommu_group *group,
+  struct iommu_domain *domain);
  extern int iommu_group_add_device(struct iommu_group *group,
  struct device *dev);
  extern void iommu_group_remove_device(struct device *dev);
@@ -840,6 +843,17 @@ static inline int iommu_group_set_name(struct iommu_group 
*group,
return -ENODEV;
  }
  
+static inline

+struct iommu_domain *iommu_group_get_domain(struct iommu_group *group)
+{
+   return NULL;
+}
+
+static inline void iommu_group_set_domain(struct iommu_group *group,
+ struct iommu_domain *domain)
+{
+}
+
  static inline int iommu_group_add_device(struct iommu_group *group,
 struct device *dev)
  {


___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu