Re: [PATCH 2/3] ACPI: Add driver for the VIOT table
On Fri, Mar 19, 2021 at 11:44:26AM +0100, Auger Eric wrote: > add some kernel-doc comments matching the explanation in the commit message? Yes, I'll add some comments. I got rid of the other issues you pointed out while reworking the driver, should be more straightforward now Thanks, Jean ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 2/3] ACPI: Add driver for the VIOT table
On Thu, Mar 18, 2021 at 07:36:50PM +, Robin Murphy wrote:
> On 2021-03-16 19:16, Jean-Philippe Brucker wrote:
> > The ACPI Virtual I/O Translation Table describes topology of
> > para-virtual platforms. For now it describes the relation between
> > virtio-iommu and the endpoints it manages. Supporting that requires
> > three steps:
> >
> > (1) acpi_viot_init(): parse the VIOT table, build a list of endpoints
> > and vIOMMUs.
> >
> > (2) acpi_viot_set_iommu_ops(): when the vIOMMU driver is loaded and the
> > device probed, register it to the VIOT driver. This step is required
> > because unlike similar drivers, VIOT doesn't create the vIOMMU
> > device.
>
> Note that you're basically the same as the DT case in this regard, so I'd
> expect things to be closer to that pattern than to that of IORT.
>
> [...]
> > @@ -1506,12 +1507,17 @@ int acpi_dma_configure_id(struct device *dev, enum
> > dev_dma_attr attr,
> > {
> > const struct iommu_ops *iommu;
> > u64 dma_addr = 0, size = 0;
> > + int ret;
> > if (attr == DEV_DMA_NOT_SUPPORTED) {
> > set_dma_ops(dev, _dummy_ops);
> > return 0;
> > }
> > + ret = acpi_viot_dma_setup(dev, attr);
> > + if (ret)
> > + return ret > 0 ? 0 : ret;
>
> I think things could do with a fair bit of refactoring here. Ideally we want
> to process a possible _DMA method (acpi_dma_get_range()) regardless of which
> flavour of IOMMU table might be present, and the amount of duplication we
> fork into at this point is unfortunate.
>
> > +
> > iort_dma_setup(dev, _addr, );
>
> For starters I think most of that should be dragged out to this level here -
> it's really only the {rc,nc}_dma_get_range() bit that deserves to be the
> IORT-specific call.
Makes sense, though I'll move it to drivers/acpi/arm64/dma.c instead of
here, because it has only ever run on CONFIG_ARM64. I don't want to
accidentally break some x86 platform with an invalid _DMA method (same
reason 7ad426398082 and 18b709beb503 kept this code in IORT)
>
> > iommu = iort_iommu_configure_id(dev, input_id);
>
> Similarly, it feels like it's only the table scan part in the middle of that
> that needs dispatching between IORT/VIOT, and its head and tail pulled out
> into a common path.
Agreed
>
> [...]
> > +static const struct iommu_ops *viot_iommu_setup(struct device *dev)
> > +{
> > + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
> > + struct viot_iommu *viommu = NULL;
> > + struct viot_endpoint *ep;
> > + u32 epid;
> > + int ret;
> > +
> > + /* Already translated? */
> > + if (fwspec && fwspec->ops)
> > + return NULL;
> > +
> > + mutex_lock(_lock);
> > + list_for_each_entry(ep, _endpoints, list) {
> > + if (viot_device_match(dev, >dev_id, )) {
> > + epid += ep->endpoint_id;
> > + viommu = ep->viommu;
> > + break;
> > + }
> > + }
> > + mutex_unlock(_lock);
> > + if (!viommu)
> > + return NULL;
> > +
> > + /* We're not translating ourself */
> > + if (viot_device_match(dev, >dev_id, ))
> > + return NULL;
> > +
> > + /*
> > +* If we found a PCI range managed by the viommu, we're the one that has
> > +* to request ACS.
> > +*/
> > + if (dev_is_pci(dev))
> > + pci_request_acs();
> > +
> > + if (!viommu->ops || WARN_ON(!viommu->dev))
> > + return ERR_PTR(-EPROBE_DEFER);
>
> Can you create (or look up) a viommu->fwnode when initially parsing the VIOT
> to represent the IOMMU devices to wait for, such that the
> viot_device_match() lookup can resolve to that and let you fall into the
> standard iommu_ops_from_fwnode() path? That's what I mean about following
> the DT pattern - I guess it might need a bit of trickery to rewrite things
> if iommu_device_register() eventually turns up with a new fwnode, so I doubt
> we can get away without *some* kind of private interface between
> virtio-iommu and VIOT, but it would be nice for the common(ish) DMA paths to
> stay as unaware of the specifics as possible.
Yes I can reuse iommu_ops_from_fwnode(). Turns out it's really easy: if we
move the VIOT initialization after acpi_scan_init(), we can use
pci_get_domain_bus_and_slot() directly and create missing fwnodes. That
gets rid of any need for a private interface between virtio-iommu and
VIOT.
>
> > +
> > + ret = iommu_fwspec_init(dev, viommu->dev->fwnode, viommu->ops);
> > + if (ret)
> > + return ERR_PTR(ret);
> > +
> > + iommu_fwspec_add_ids(dev, , 1);
> > +
> > + /*
> > +* If we have reason to believe the IOMMU driver missed the initial
> > +* add_device callback for dev, replay it to get things in order.
> > +*/
> > + if (dev->bus && !device_iommu_mapped(dev))
> > + iommu_probe_device(dev);
> > +
> > + return viommu->ops;
> > +}
> > +
> > +/**
> > + * acpi_viot_dma_setup - Configure DMA for an endpoint described in
Re: [PATCH 2/3] ACPI: Add driver for the VIOT table
Hi Jean,
On 3/16/21 8:16 PM, Jean-Philippe Brucker wrote:
> The ACPI Virtual I/O Translation Table describes topology of
> para-virtual platforms. For now it describes the relation between
> virtio-iommu and the endpoints it manages. Supporting that requires
> three steps:
>
> (1) acpi_viot_init(): parse the VIOT table, build a list of endpoints
> and vIOMMUs.
>
> (2) acpi_viot_set_iommu_ops(): when the vIOMMU driver is loaded and the
> device probed, register it to the VIOT driver. This step is required
> because unlike similar drivers, VIOT doesn't create the vIOMMU
> device.
>
> (3) acpi_viot_dma_setup(): when the endpoint is initialized, find the
> vIOMMU and register the endpoint's DMA ops.
>
> If step (3) happens before step (2), it is deferred until the IOMMU is
> initialized, then retried.
>
> Signed-off-by: Jean-Philippe Brucker
> ---
> drivers/acpi/Kconfig | 3 +
> drivers/iommu/Kconfig| 1 +
> drivers/acpi/Makefile| 2 +
> include/linux/acpi_viot.h| 26 +++
> drivers/acpi/bus.c | 2 +
> drivers/acpi/scan.c | 6 +
> drivers/acpi/viot.c | 406 +++
> drivers/iommu/virtio-iommu.c | 3 +
> MAINTAINERS | 8 +
> 9 files changed, 457 insertions(+)
> create mode 100644 include/linux/acpi_viot.h
> create mode 100644 drivers/acpi/viot.c
>
> diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
> index eedec61e3476..3758c6940ed7 100644
> --- a/drivers/acpi/Kconfig
> +++ b/drivers/acpi/Kconfig
> @@ -526,6 +526,9 @@ endif
>
> source "drivers/acpi/pmic/Kconfig"
>
> +config ACPI_VIOT
> + bool
> +
> endif# ACPI
>
> config X86_PM_TIMER
> diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
> index 192ef8f61310..2819b5c8ec30 100644
> --- a/drivers/iommu/Kconfig
> +++ b/drivers/iommu/Kconfig
> @@ -403,6 +403,7 @@ config VIRTIO_IOMMU
> depends on ARM64
> select IOMMU_API
> select INTERVAL_TREE
> + select ACPI_VIOT if ACPI
> help
> Para-virtualised IOMMU driver with virtio.
>
> diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile
> index 700b41adf2db..a6e644c48987 100644
> --- a/drivers/acpi/Makefile
> +++ b/drivers/acpi/Makefile
> @@ -118,3 +118,5 @@ video-objs+= acpi_video.o
> video_detect.o
> obj-y+= dptf/
>
> obj-$(CONFIG_ARM64) += arm64/
> +
> +obj-$(CONFIG_ACPI_VIOT) += viot.o
> diff --git a/include/linux/acpi_viot.h b/include/linux/acpi_viot.h
> new file mode 100644
> index ..1f5837595488
> --- /dev/null
> +++ b/include/linux/acpi_viot.h
> @@ -0,0 +1,26 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +
> +#ifndef __ACPI_VIOT_H__
> +#define __ACPI_VIOT_H__
> +
> +#include
> +
> +#ifdef CONFIG_ACPI_VIOT
> +void __init acpi_viot_init(void);
> +int acpi_viot_dma_setup(struct device *dev, enum dev_dma_attr attr);
> +int acpi_viot_set_iommu_ops(struct device *dev, struct iommu_ops *ops);
> +#else
> +static inline void acpi_viot_init(void) {}
> +static inline int acpi_viot_dma_setup(struct device *dev,
> + enum dev_dma_attr attr)
> +{
> + return 0;
> +}
> +static inline int acpi_viot_set_iommu_ops(struct device *dev,
> + struct iommu_ops *ops)
> +{
> + return -ENODEV;
> +}
> +#endif
> +
> +#endif /* __ACPI_VIOT_H__ */
> diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
> index be7da23fad76..f9a965c6617e 100644
> --- a/drivers/acpi/bus.c
> +++ b/drivers/acpi/bus.c
> @@ -27,6 +27,7 @@
> #include
> #endif
> #include
> +#include
> #include
> #include
> #include
> @@ -1338,6 +1339,7 @@ static int __init acpi_init(void)
>
> pci_mmcfg_late_init();
> acpi_iort_init();
> + acpi_viot_init();
> acpi_scan_init();
> acpi_ec_init();
> acpi_debugfs_init();
> diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
> index a184529d8fa4..4af01fddd94c 100644
> --- a/drivers/acpi/scan.c
> +++ b/drivers/acpi/scan.c
> @@ -9,6 +9,7 @@
> #include
> #include
> #include
> +#include
> #include
> #include
> #include
> @@ -1506,12 +1507,17 @@ int acpi_dma_configure_id(struct device *dev, enum
> dev_dma_attr attr,
> {
> const struct iommu_ops *iommu;
> u64 dma_addr = 0, size = 0;
> + int ret;
>
> if (attr == DEV_DMA_NOT_SUPPORTED) {
> set_dma_ops(dev, _dummy_ops);
> return 0;
> }
>
> + ret = acpi_viot_dma_setup(dev, attr);
> + if (ret)
> + return ret > 0 ? 0 : ret;
> +
> iort_dma_setup(dev, _addr, );
>
> iommu = iort_iommu_configure_id(dev, input_id);
> diff --git a/drivers/acpi/viot.c b/drivers/acpi/viot.c
> new file mode 100644
> index ..57a092e8551b
> --- /dev/null
> +++ b/drivers/acpi/viot.c
> @@ -0,0 +1,406 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Virtual I/O
Re: [PATCH 2/3] ACPI: Add driver for the VIOT table
On 2021-03-16 19:16, Jean-Philippe Brucker wrote:
The ACPI Virtual I/O Translation Table describes topology of
para-virtual platforms. For now it describes the relation between
virtio-iommu and the endpoints it manages. Supporting that requires
three steps:
(1) acpi_viot_init(): parse the VIOT table, build a list of endpoints
and vIOMMUs.
(2) acpi_viot_set_iommu_ops(): when the vIOMMU driver is loaded and the
device probed, register it to the VIOT driver. This step is required
because unlike similar drivers, VIOT doesn't create the vIOMMU
device.
Note that you're basically the same as the DT case in this regard, so
I'd expect things to be closer to that pattern than to that of IORT.
[...]
@@ -1506,12 +1507,17 @@ int acpi_dma_configure_id(struct device *dev, enum
dev_dma_attr attr,
{
const struct iommu_ops *iommu;
u64 dma_addr = 0, size = 0;
+ int ret;
if (attr == DEV_DMA_NOT_SUPPORTED) {
set_dma_ops(dev, _dummy_ops);
return 0;
}
+ ret = acpi_viot_dma_setup(dev, attr);
+ if (ret)
+ return ret > 0 ? 0 : ret;
I think things could do with a fair bit of refactoring here. Ideally we
want to process a possible _DMA method (acpi_dma_get_range()) regardless
of which flavour of IOMMU table might be present, and the amount of
duplication we fork into at this point is unfortunate.
+
iort_dma_setup(dev, _addr, );
For starters I think most of that should be dragged out to this level
here - it's really only the {rc,nc}_dma_get_range() bit that deserves to
be the IORT-specific call.
iommu = iort_iommu_configure_id(dev, input_id);
Similarly, it feels like it's only the table scan part in the middle of
that that needs dispatching between IORT/VIOT, and its head and tail
pulled out into a common path.
[...]
+static const struct iommu_ops *viot_iommu_setup(struct device *dev)
+{
+ struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+ struct viot_iommu *viommu = NULL;
+ struct viot_endpoint *ep;
+ u32 epid;
+ int ret;
+
+ /* Already translated? */
+ if (fwspec && fwspec->ops)
+ return NULL;
+
+ mutex_lock(_lock);
+ list_for_each_entry(ep, _endpoints, list) {
+ if (viot_device_match(dev, >dev_id, )) {
+ epid += ep->endpoint_id;
+ viommu = ep->viommu;
+ break;
+ }
+ }
+ mutex_unlock(_lock);
+ if (!viommu)
+ return NULL;
+
+ /* We're not translating ourself */
+ if (viot_device_match(dev, >dev_id, ))
+ return NULL;
+
+ /*
+* If we found a PCI range managed by the viommu, we're the one that has
+* to request ACS.
+*/
+ if (dev_is_pci(dev))
+ pci_request_acs();
+
+ if (!viommu->ops || WARN_ON(!viommu->dev))
+ return ERR_PTR(-EPROBE_DEFER);
Can you create (or look up) a viommu->fwnode when initially parsing the
VIOT to represent the IOMMU devices to wait for, such that the
viot_device_match() lookup can resolve to that and let you fall into the
standard iommu_ops_from_fwnode() path? That's what I mean about
following the DT pattern - I guess it might need a bit of trickery to
rewrite things if iommu_device_register() eventually turns up with a new
fwnode, so I doubt we can get away without *some* kind of private
interface between virtio-iommu and VIOT, but it would be nice for the
common(ish) DMA paths to stay as unaware of the specifics as possible.
+
+ ret = iommu_fwspec_init(dev, viommu->dev->fwnode, viommu->ops);
+ if (ret)
+ return ERR_PTR(ret);
+
+ iommu_fwspec_add_ids(dev, , 1);
+
+ /*
+* If we have reason to believe the IOMMU driver missed the initial
+* add_device callback for dev, replay it to get things in order.
+*/
+ if (dev->bus && !device_iommu_mapped(dev))
+ iommu_probe_device(dev);
+
+ return viommu->ops;
+}
+
+/**
+ * acpi_viot_dma_setup - Configure DMA for an endpoint described in VIOT
+ * @dev: the endpoint
+ * @attr: coherency property of the endpoint
+ *
+ * Setup the DMA and IOMMU ops for an endpoint described by the VIOT table.
+ *
+ * Return:
+ * * 0 - @dev doesn't match any VIOT node
+ * * 1 - ops for @dev were successfully installed
+ * * -EPROBE_DEFER - ops for @dev aren't yet available
+ */
+int acpi_viot_dma_setup(struct device *dev, enum dev_dma_attr attr)
+{
+ const struct iommu_ops *iommu_ops = viot_iommu_setup(dev);
+
+ if (IS_ERR_OR_NULL(iommu_ops)) {
+ int ret = PTR_ERR(iommu_ops);
+
+ if (ret == -EPROBE_DEFER || ret == 0)
+ return ret;
+ dev_err(dev, "error %d while setting up virt IOMMU\n", ret);
+ return 0;
+ }
+
+#ifdef CONFIG_ARCH_HAS_SETUP_DMA_OPS
[PATCH 2/3] ACPI: Add driver for the VIOT table
The ACPI Virtual I/O Translation Table describes topology of
para-virtual platforms. For now it describes the relation between
virtio-iommu and the endpoints it manages. Supporting that requires
three steps:
(1) acpi_viot_init(): parse the VIOT table, build a list of endpoints
and vIOMMUs.
(2) acpi_viot_set_iommu_ops(): when the vIOMMU driver is loaded and the
device probed, register it to the VIOT driver. This step is required
because unlike similar drivers, VIOT doesn't create the vIOMMU
device.
(3) acpi_viot_dma_setup(): when the endpoint is initialized, find the
vIOMMU and register the endpoint's DMA ops.
If step (3) happens before step (2), it is deferred until the IOMMU is
initialized, then retried.
Signed-off-by: Jean-Philippe Brucker
---
drivers/acpi/Kconfig | 3 +
drivers/iommu/Kconfig| 1 +
drivers/acpi/Makefile| 2 +
include/linux/acpi_viot.h| 26 +++
drivers/acpi/bus.c | 2 +
drivers/acpi/scan.c | 6 +
drivers/acpi/viot.c | 406 +++
drivers/iommu/virtio-iommu.c | 3 +
MAINTAINERS | 8 +
9 files changed, 457 insertions(+)
create mode 100644 include/linux/acpi_viot.h
create mode 100644 drivers/acpi/viot.c
diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
index eedec61e3476..3758c6940ed7 100644
--- a/drivers/acpi/Kconfig
+++ b/drivers/acpi/Kconfig
@@ -526,6 +526,9 @@ endif
source "drivers/acpi/pmic/Kconfig"
+config ACPI_VIOT
+ bool
+
endif # ACPI
config X86_PM_TIMER
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 192ef8f61310..2819b5c8ec30 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -403,6 +403,7 @@ config VIRTIO_IOMMU
depends on ARM64
select IOMMU_API
select INTERVAL_TREE
+ select ACPI_VIOT if ACPI
help
Para-virtualised IOMMU driver with virtio.
diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile
index 700b41adf2db..a6e644c48987 100644
--- a/drivers/acpi/Makefile
+++ b/drivers/acpi/Makefile
@@ -118,3 +118,5 @@ video-objs += acpi_video.o video_detect.o
obj-y += dptf/
obj-$(CONFIG_ARM64)+= arm64/
+
+obj-$(CONFIG_ACPI_VIOT)+= viot.o
diff --git a/include/linux/acpi_viot.h b/include/linux/acpi_viot.h
new file mode 100644
index ..1f5837595488
--- /dev/null
+++ b/include/linux/acpi_viot.h
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef __ACPI_VIOT_H__
+#define __ACPI_VIOT_H__
+
+#include
+
+#ifdef CONFIG_ACPI_VIOT
+void __init acpi_viot_init(void);
+int acpi_viot_dma_setup(struct device *dev, enum dev_dma_attr attr);
+int acpi_viot_set_iommu_ops(struct device *dev, struct iommu_ops *ops);
+#else
+static inline void acpi_viot_init(void) {}
+static inline int acpi_viot_dma_setup(struct device *dev,
+ enum dev_dma_attr attr)
+{
+ return 0;
+}
+static inline int acpi_viot_set_iommu_ops(struct device *dev,
+ struct iommu_ops *ops)
+{
+ return -ENODEV;
+}
+#endif
+
+#endif /* __ACPI_VIOT_H__ */
diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
index be7da23fad76..f9a965c6617e 100644
--- a/drivers/acpi/bus.c
+++ b/drivers/acpi/bus.c
@@ -27,6 +27,7 @@
#include
#endif
#include
+#include
#include
#include
#include
@@ -1338,6 +1339,7 @@ static int __init acpi_init(void)
pci_mmcfg_late_init();
acpi_iort_init();
+ acpi_viot_init();
acpi_scan_init();
acpi_ec_init();
acpi_debugfs_init();
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index a184529d8fa4..4af01fddd94c 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -9,6 +9,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -1506,12 +1507,17 @@ int acpi_dma_configure_id(struct device *dev, enum
dev_dma_attr attr,
{
const struct iommu_ops *iommu;
u64 dma_addr = 0, size = 0;
+ int ret;
if (attr == DEV_DMA_NOT_SUPPORTED) {
set_dma_ops(dev, _dummy_ops);
return 0;
}
+ ret = acpi_viot_dma_setup(dev, attr);
+ if (ret)
+ return ret > 0 ? 0 : ret;
+
iort_dma_setup(dev, _addr, );
iommu = iort_iommu_configure_id(dev, input_id);
diff --git a/drivers/acpi/viot.c b/drivers/acpi/viot.c
new file mode 100644
index ..57a092e8551b
--- /dev/null
+++ b/drivers/acpi/viot.c
@@ -0,0 +1,406 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Virtual I/O topology
+ */
+#define pr_fmt(fmt) "ACPI: VIOT: " fmt
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+struct viot_dev_id {
+ unsigned inttype;
+#define VIOT_DEV_TYPE_PCI 1
+#define VIOT_DEV_TYPE_MMIO 2
+ union {
+ /* PCI endpoint
