RE: mechanism to allow a driver to bind to any device
-Original Message- From: Alex Williamson [mailto:alex.william...@redhat.com] Sent: Wednesday, March 26, 2014 11:21 AM To: Alexander Graf Cc: Konrad Rzeszutek Wilk; k...@vger.kernel.org; jan.kis...@siemens.com; will.dea...@arm.com; Yoder Stuart-B08248; linux-ker...@vger.kernel.org; Michal Hocko; Bjorn Helgaas; Sethi Varun-B16395; kvm...@lists.cs.columbia.edu; Rafael J. Wysocki; Guenter Roeck; Dmitry Kasatkin; Joe Perches; Wood Scott-B07421; Antonios Motakis; t...@virtualopensystems.com; Toshi Kani; Greg KH; a.r...@virtualopensystems.com; iommu@lists.linux-foundation.org; Tejun Heo; christoffer.d...@linaro.org Subject: Re: mechanism to allow a driver to bind to any device On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? It's probably not unreasonable... I did not understand the drivers_autoprobe mechanism until now...didn't realize we had that. What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Yes, that is
Re: mechanism to allow a driver to bind to any device
On Mon, Mar 31, 2014 at 06:47:51PM +, Stuart Yoder wrote: I also, was at the point where I thought we should perhaps just go with current mechanisms and implement new_id for the platform bus...but Greg's recent response is 'platform devices suck' and it sounds like he would reject a new_id patch for the platform bus. So it kind of feels like we are stuck. ids mean nothing in the platform device model, so having a new_id file for them makes no sense. greg k-h ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: mechanism to allow a driver to bind to any device
-Original Message-
From: Greg KH [mailto:gre...@linuxfoundation.org]
Sent: Monday, March 31, 2014 2:47 PM
To: Yoder Stuart-B08248
Cc: Alex Williamson; Alexander Graf; k...@vger.kernel.org;
jan.kis...@siemens.com; will.dea...@arm.com;
a.r...@virtualopensystems.com; Michal Hocko; Wood Scott-B07421; Sethi
Varun-B16395; kvm...@lists.cs.columbia.edu; Rafael J. Wysocki; Guenter
Roeck; Dmitry Kasatkin; Tejun Heo; Bjorn Helgaas; Antonios Motakis;
t...@virtualopensystems.com; Toshi Kani; linux-ker...@vger.kernel.org;
iommu@lists.linux-foundation.org; Joe Perches;
christoffer.d...@linaro.org
Subject: Re: mechanism to allow a driver to bind to any device
On Mon, Mar 31, 2014 at 06:47:51PM +, Stuart Yoder wrote:
I also, was at the point where I thought we should perhaps just
go with current mechanisms and implement new_id for the platform
bus...but Greg's recent response is 'platform devices suck' and it
sounds
like he would reject a new_id patch for the platform bus. So it kind
of feels like we are stuck.
ids mean nothing in the platform device model, so having a new_id file
for them makes no sense.
They don't have IDs like PCI, but platform drivers have to match on
something. Platform device match tables are based on compatible strings.
Example from Freescale DMA driver:
static const struct of_device_id fsldma_of_ids[] = {
{ .compatible = fsl,elo3-dma, },
{ .compatible = fsl,eloplus-dma, },
{ .compatible = fsl,elo-dma, },
{}
};
The process of unbinding, setting a new_id, and binding to vfio would work
just like PCI:
echo ffe101300.dma /sys/bus/platform/devices/ffe101300.dma/driver/unbind
echo fsl,eloplus-dma /sys/bus/platform/drivers/vfio-platform/new_id
Thanks,
Stuart
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: mechanism to allow a driver to bind to any device
On Mon, 31 Mar 2014 20:23:36 +
Stuart Yoder stuart.yo...@freescale.com wrote:
From: Greg KH [mailto:gre...@linuxfoundation.org]
Sent: Monday, March 31, 2014 2:47 PM
On Mon, Mar 31, 2014 at 06:47:51PM +, Stuart Yoder wrote:
I also, was at the point where I thought we should perhaps just
go with current mechanisms and implement new_id for the platform
bus...but Greg's recent response is 'platform devices suck' and it
sounds
like he would reject a new_id patch for the platform bus. So it kind
of feels like we are stuck.
ids mean nothing in the platform device model, so having a new_id file
for them makes no sense.
They don't have IDs like PCI, but platform drivers have to match on
something. Platform device match tables are based on compatible strings.
Example from Freescale DMA driver:
static const struct of_device_id fsldma_of_ids[] = {
{ .compatible = fsl,elo3-dma, },
{ .compatible = fsl,eloplus-dma, },
{ .compatible = fsl,elo-dma, },
{}
};
The process of unbinding, setting a new_id, and binding to vfio would work
just like PCI:
echo ffe101300.dma /sys/bus/platform/devices/ffe101300.dma/driver/unbind
echo fsl,eloplus-dma /sys/bus/platform/drivers/vfio-platform/new_id
In platform device land, we don't want to pursue the
new_id/match-by-compatible methodology: we know exactly which specific
device (not device types) we want bound to which driver, so we just
want to be able to simply:
echo fff51000.ethernet | sudo tee -a
/sys/bus/platform/devices/fff51000.ethernet/driver/unbind
echo fff51000.ethernet | sudo tee -a
/sys/bus/platform/drivers/vfio-platform/bind
and not get involved with how PCI doesn't simply do that, independent
of autoprobe/hotplug.
Kim
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: mechanism to allow a driver to bind to any device
On Fri, 28 Mar 2014 11:10:23 -0600 Alex Williamson alex.william...@redhat.com wrote: On Fri, 2014-03-28 at 12:58 -0400, Konrad Rzeszutek Wilk wrote: On Wed, Mar 26, 2014 at 04:09:21PM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 10:21 -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Option 2. in Stuart's list does this by short-cutting ID matching so that a match is only found when using the sysfs bind path, option 3. enables a way
Re: mechanism to allow a driver to bind to any device
On Mon, 2014-03-31 at 17:36 -0500, Kim Phillips wrote: On Fri, 28 Mar 2014 11:10:23 -0600 Alex Williamson alex.william...@redhat.com wrote: On Fri, 2014-03-28 at 12:58 -0400, Konrad Rzeszutek Wilk wrote: On Wed, Mar 26, 2014 at 04:09:21PM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 10:21 -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special
Re: mechanism to allow a driver to bind to any device
On Wed, Mar 26, 2014 at 10:39:57PM +0100, Antonios Motakis wrote: Of note is that new_id doesn't work particularly well for platform devices. Nor should it. Platform devices suck horribly, and ids mean nothing to them, so you shouldn't even try this. Use a real bus and it should be fine. greg k-h ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: mechanism to allow a driver to bind to any device
On Wed, Mar 26, 2014 at 04:09:21PM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 10:21 -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Option 2. in Stuart's list does this by short-cutting ID matching so that a match is only found when using the sysfs bind path, option 3. enables a way for a driver to expose their own sysfs entry point for binding. The latter feels particularly chaotic since drivers get to make-up their own bind mechanism. Another twist I'll throw in is that devices can be hot added to IOMMU groups that are in-use by userspace. When that happens we'd like to be able to disable driver autoprobe of the device to avoid a host driver automatically
Re: mechanism to allow a driver to bind to any device
On Fri, 2014-03-28 at 12:58 -0400, Konrad Rzeszutek Wilk wrote: On Wed, Mar 26, 2014 at 04:09:21PM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 10:21 -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Option 2. in Stuart's list does this by short-cutting ID matching so that a match is only found when using the sysfs bind path, option 3. enables a way for a driver to expose their own sysfs entry point for binding. The latter feels particularly chaotic since drivers get to make-up their own bind mechanism. Another twist I'll throw in is that devices can be
Re: mechanism to allow a driver to bind to any device
On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. Which I think is what is currently being done. Why is that not sufficient? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Would like your comment on these options-- option #3 is preferred and is literally a 2 line patch. Thanks, Stuart ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: mechanism to allow a driver to bind to any device
Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. Alex Would like your comment on these options-- option #3 is preferred and is literally a 2 line patch. Thanks, Stuart ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: mechanism to allow a driver to bind to any device
-Original Message- From: Konrad Rzeszutek Wilk [mailto:konrad.w...@oracle.com] Sent: Wednesday, March 26, 2014 9:40 AM To: Yoder Stuart-B08248 Cc: Greg KH; k...@vger.kernel.org; jan.kis...@siemens.com; will.dea...@arm.com; linux-ker...@vger.kernel.org; Bjorn Helgaas; Sethi Varun-B16395; kvm...@lists.cs.columbia.edu; Rafael J. Wysocki; ag...@suse.de; Guenter Roeck; Dmitry Kasatkin; Tejun Heo; Wood Scott- B07421; Antonios Motakis; t...@virtualopensystems.com; Michal Hocko; Toshi Kani; a.r...@virtualopensystems.com; iommu@lists.linux- foundation.org; Joe Perches; christoffer.d...@linaro.org Subject: Re: mechanism to allow a driver to bind to any device On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. Sorry, I frankly had no idea of what you were talking about. Please explain with an example what steps a user would take to unbind a device from the host and bind it to say vfio-pci. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. How can you bind a device to vfio, when vfio is not aware of the device type? Does not work. There is no way the vfio driver can know ahead of time what device may be bound to it. Stuart ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: mechanism to allow a driver to bind to any device
On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Option 2. in Stuart's list does this by short-cutting ID matching so that a match is only found when using the sysfs bind path, option 3. enables a way for a driver to expose their own sysfs entry point for binding. The latter feels particularly chaotic since drivers get to make-up their own bind mechanism. Another twist I'll throw in is that devices can be hot added to IOMMU groups that are in-use by userspace. When that happens we'd like to be able to disable driver autoprobe of the device to avoid a host driver automatically binding to the device. I wonder if instead of looking at the problem from the driver perspective, if we were to instead look at it from the device perspective if we might find a solution that would address both. For instance, if devices had a driver_probe_id property that was by default set to their bus specific ID match ($VENDOR $DEVICE on PCI) could we use that to write new match
Re: mechanism to allow a driver to bind to any device
On Wed, Mar 26, 2014 at 11:06:02PM +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? you echo the BDF to a 'new_slot' to setup an pci match entry (so that it can lookup from the BDF the device/vendor id). Then you echo the BDF to the 'bind'. The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. But that would not happen if you use BDF. So if you switch from using device/vendor id then you don't have this problem. Alex Would like your comment on these options-- option #3 is preferred and is literally a 2 line patch. Thanks, Stuart ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: mechanism to allow a driver to bind to any device
On Wed, Mar 26, 2014 at 10:21:02AM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Option 2. in Stuart's list does this by short-cutting ID matching so that a match is only found when using the sysfs bind path, option 3. enables a way for a driver to expose their own sysfs entry point for binding. The latter feels particularly chaotic since drivers get to make-up their own bind mechanism. Another twist I'll throw in is that devices can be hot added to IOMMU groups that are in-use by userspace. When that happens we'd like to be able to disable driver autoprobe of the device to avoid a host driver automatically binding to the device. I wonder if instead of looking at the problem from the driver perspective, if we were to instead look at it from the device perspective if we might find a solution that would address
Re: mechanism to allow a driver to bind to any device
On Wed, 2014-03-26 at 12:32 -0400, Konrad Rzeszutek Wilk wrote: On Wed, Mar 26, 2014 at 10:21:02AM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Option 2. in Stuart's list does this by short-cutting ID matching so that a match is only found when using the sysfs bind path, option 3. enables a way for a driver to expose their own sysfs entry point for binding. The latter feels particularly chaotic since drivers get to make-up their own bind mechanism. Another twist I'll throw in is that devices can be hot added to IOMMU groups that are in-use by userspace. When that happens we'd like to be able to disable driver autoprobe of the device to avoid a host driver automatically
Re: mechanism to allow a driver to bind to any device
On Wed, Mar 26, 2014 at 10:49:52AM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 12:32 -0400, Konrad Rzeszutek Wilk wrote: On Wed, Mar 26, 2014 at 10:21:02AM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Option 2. in Stuart's list does this by short-cutting ID matching so that a match is only found when using the sysfs bind path, option 3. enables a way for a driver to expose their own sysfs entry point for binding. The latter feels particularly chaotic since drivers get to make-up their own bind mechanism. Another twist I'll throw in is that devices
Re: mechanism to allow a driver to bind to any device
On Wed, 2014-03-26 at 13:04 -0400, Konrad Rzeszutek Wilk wrote: On Wed, Mar 26, 2014 at 10:49:52AM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 12:32 -0400, Konrad Rzeszutek Wilk wrote: On Wed, Mar 26, 2014 at 10:21:02AM -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Option 2. in Stuart's list does this by short-cutting ID matching so that a match is only found when using the sysfs bind path, option 3. enables a way for a
RE: mechanism to allow a driver to bind to any device
The other option for this is to having some sort of priority on the device probing with hotplugging. That is you can could do the following: 1) add the device vendor/model in vfio 2) unbind the BDF from the original driver. 3) hotplug happens - any new device that has the device vendor/model gets owned by vfio instead of the original device. 4). bind the BDF to the vfio. Granted that is a bit silly too - as the admin might want to have the new hotplugged device be owned by the native driver. In which case, why not just switch out from using device vendor/model to just using BDF values? Did you read option #3 in my proposal-- using bus/dev/func # is one option proposed: 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind bus/dev/func In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 But in order to do that the vfio driver needs to call driver_probe_device() which it can't right now without the proposed 2 line patch...which Greg rejects. Stuart ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: mechanism to allow a driver to bind to any device
Hello, On Wed, Mar 26, 2014 at 2:40 AM, Stuart Yoder stuart.yo...@freescale.comwrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. Of note is that new_id doesn't work particularly well for platform devices. Before trying any of the other two solutions, a nasty hack was applied on the device tree used with the system in order to let vfio-platform match with a specific device, which is certainly not an acceptable solution. Implementing wildcard matching quickly reveals the problems mentioned, which motivates the other two approaches. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 Would like your comment on these options-- option #3 is preferred and is literally a 2 line patch. I would definitely agree with approach #3, for which Kim has already provided a patch. Not having this would make using VFIO with platform devices really inelegant and strange. Thanks, Stuart -- Antonios Motakis Virtual Open Systems ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: mechanism to allow a driver to bind to any device
(Resend because of html) On Wed, Mar 26, 2014 at 2:40 AM, Stuart Yoder stuart.yo...@freescale.com wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. Of note is that new_id doesn't work particularly well for platform devices. Before trying any of the other two solutions, a nasty hack was applied on the device tree used with the system in order to let vfio-platform match with a specific device, which is certainly not an acceptable solution. Implementing wildcard matching quickly reveals the problems mentioned, which motivates the other two approaches. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 Would like your comment on these options-- option #3 is preferred and is literally a 2 line patch. I would definitely agree with approach #3, for which Kim has already provided a patch. Not having this would make using VFIO with platform devices really inelegant and strange. Thanks, Stuart -- Antonios Motakis Virtual Open Systems ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: mechanism to allow a driver to bind to any device
On Wed, 2014-03-26 at 10:21 -0600, Alex Williamson wrote: On Wed, 2014-03-26 at 23:06 +0800, Alexander Graf wrote: Am 26.03.2014 um 22:40 schrieb Konrad Rzeszutek Wilk konrad.w...@oracle.com: On Wed, Mar 26, 2014 at 01:40:32AM +, Stuart Yoder wrote: Hi Greg, We (Linaro, Freescale, Virtual Open Systems) are trying get an issue closed that has been perculating for a while around creating a mechanism that will allow kernel drivers like vfio can bind to devices of any type. This thread with you: http://www.spinics.net/lists/kvm-arm/msg08370.html ...seems to have died out, so am trying to get your response and will summarize again. Vfio drivers in the kernel (regardless of bus type) need to bind to devices of any type. The driver's function is to simply export hardware resources of any type to user space. There are several approaches that have been proposed: You seem to have missed the one I proposed. 1. new_id -- (current approach) the user explicitly registers each new device type with the vfio driver using the new_id mechanism. Problem: multiple drivers will be resident that handle the same device type...and there is nothing user space hotplug infrastructure can do to help. 2. any id -- the vfio driver could specify a wildcard match of some kind in its ID match table which would allow it to match and bind to any possible device id. However, we don't want the vfio driver grabbing _all_ devices...just the ones we explicitly want to pass to user space. The proposed patch to support this was to create a new flag sysfs_bind_only in struct device_driver. When this flag is set, the driver can only bind to devices via the sysfs bind file. This would allow the wildcard match to work. Patch is here: https://lkml.org/lkml/2013/12/3/253 3. Driver initiated explicit bind -- with this approach the vfio driver would create a private 'bind' sysfs object and the user would echo the requested device into it: echo 0001:03:00.0 /sys/bus/pci/drivers/vfio-pci/vfio_bind In order to make that work, the driver would need to call driver_probe_device() and thus we need this patch: https://lkml.org/lkml/2014/2/8/175 4). Use the 'unbind' (from the original device) and 'bind' to vfio driver. This is approach 2, no? Which I think is what is currently being done. Why is that not sufficient? How would 'bind to vfio driver' look like? The only thing I see in the URL is That works, but it is ugly. There is some mention of race but I don't see how - if you do the 'unbind' on the original driver and then bind the BDF to the VFIO how would you get a race? Typically on PCI, you do a - add wildcard (pci id) match to vfio driver - unbind driver - reprobe - device attaches to vfio driver because it is the least recent match - remove wildcard match from vfio driver If in between you hotplug add a card of the same type, it gets attached to vfio - even though the logical default driver would be the device specific driver. I've mentioned drivers_autoprobe in the past, but I'm not sure we're really factoring it into the discussion. drivers_autoprobe allows us to toggle two points: a) When a new device is added whether we automatically give drivers a try at binding to it b) When a new driver is added whether it gets to try to bind to anything in the system So we do have a mechanism to avoid the race, but the problem is that it becomes the responsibility of userspace to: 1) turn off drivers_autoprobe 2) unbind/new_id/bind/remove_id 3) turn on drivers_autoprobe 4) call drivers_probe for anything added between 1) 3) Is the question about the ugliness of the current solution whether it's unreasonable to ask userspace to do this? What we seem to be asking for above is more like an autoprobe flag per driver where there's some way for this special driver to opt out of auto probing. Option 2. in Stuart's list does this by short-cutting ID matching so that a match is only found when using the sysfs bind path, option 3. enables a way for a driver to expose their own sysfs entry point for binding. The latter feels particularly chaotic since drivers get to make-up their own bind mechanism. Another twist I'll throw in is that devices can be hot added to IOMMU groups that are in-use by userspace. When that happens we'd like to be able to disable driver autoprobe of the device to avoid a host driver automatically binding to the device. I wonder if instead of looking at the problem from the driver perspective, if we were to instead look at it from the device perspective if we might find a solution that would address both.
