Re: [RFC/PATCH 1/1] virtio: Introduce MMIO ops
* Jan Kiszka [2020-04-30 14:59:50]: > >I believe ivshmem2_virtio requires hypervisor to support PCI device emulation > >(for life-cycle management of VMs), which our hypervisor may not support. A > >simple shared memory and doorbell or message-queue based transport will work > >for > >us. > > As written in our private conversation, a mapping of the ivshmem2 device > discovery to platform mechanism (device tree etc.) and maybe even the > register access for doorbell and life-cycle management to something > hypercall-like would be imaginable. What would count more from virtio > perspective is a common mapping on a shared memory transport. Yes that sounds simpler for us. > That said, I also warned about all the features that PCI already defined > (such as message-based interrupts) which you may have to add when going a > different way for the shared memory device. Is it really required to present this shared memory as belonging to a PCI device? I would expect the device-tree to indicate the presence of this shared memory region, which we should be able to present to ivshmem2 as shared memory region to use (along with some handles for doorbell or message queue use). I understand the usefulness of modeling the shared memory as part of device so that hypervisor can send events related to peers going down or coming up. In our case, there will be other means to discover those events and avoiding this requirement on hypervisor (to emulate PCI) will simplify the solution for us. Any idea when we can expect virtio over ivshmem2 to become available?! -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC/PATCH 1/1] virtio: Introduce MMIO ops
* Will Deacon [2020-04-30 11:41:50]: > On Thu, Apr 30, 2020 at 04:04:46PM +0530, Srivatsa Vaddagiri wrote: > > If CONFIG_VIRTIO_MMIO_OPS is defined, then I expect this to be > > unconditionally > > set to 'magic_qcom_ops' that uses hypervisor-supported interface for IO (for > > example: message_queue_send() and message_queue_recevie() hypercalls). > > Hmm, but then how would such a kernel work as a guest under all the > spec-compliant hypervisors out there? Ok I see your point and yes for better binary compatibility, the ops have to be set based on runtime detection of hypervisor capabilities. > > Ok. I guess the other option is to standardize on a new virtio transport > > (like > > ivshmem2-virtio)? > > I haven't looked at that, but I suppose it depends on what your hypervisor > folks are willing to accomodate. I believe ivshmem2_virtio requires hypervisor to support PCI device emulation (for life-cycle management of VMs), which our hypervisor may not support. A simple shared memory and doorbell or message-queue based transport will work for us. -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC/PATCH 0/1] virtio_mmio: hypervisor specific interfaces for MMIO
* Will Deacon [2020-04-30 11:39:19]: > Hi Vatsa, > > On Thu, Apr 30, 2020 at 03:59:39PM +0530, Srivatsa Vaddagiri wrote: > > > What's stopping you from implementing the trapping support in the > > > hypervisor? Unlike the other patches you sent out, where the guest memory > > > is not accessible to the host, there doesn't seem to be any advantage to > > > not having trapping support, or am I missing something here? > > > > I have had this discussion with hypervisor folks. They seem to be > > concerned about complexity of having a VM's fault be handled in another > > untrusted VM. They are not keen to add MMIO support. > > Right, but I'm concerned about forking the implementation from the spec > and I'm not keen to add these hooks ;) > > What does your hook actually do? I'm assuming an HVC? Yes, it will issue message-queue related hypercalls > If so, then where the > fault is handled seems to be unrelated and whether the guest exit is due to > an HVC or a stage-2 fault should be immaterial. A stage-2 fault would be considered fatal normally and result in termination of guest. Modifying that behavior to allow resumption in case of virtio config space access, especially including the untrusted VM in this flow, is perhaps the concern. HVC calls OTOH are more vetted interfaces that the hypervisor has to do nothing additional to handle. > In other words, I don't > follow why the trapping mechanism necessitates the way in which the fault is > handled. Let me check with our hypervisor folks again. Thanks for your inputs. -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC/PATCH 0/1] virtio_mmio: hypervisor specific interfaces for MMIO
* Michael S. Tsirkin [2020-04-30 06:07:56]: > On Thu, Apr 30, 2020 at 03:32:55PM +0530, Srivatsa Vaddagiri wrote: > > The Type-1 hypervisor we are dealing with does not allow for MMIO > > transport. > > How about PCI then? Correct me if I am wrong, but basically virtio_pci uses the same low-level primitive as readl/writel on a platform such as ARM64? So similar issues there also. -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC/PATCH 1/1] virtio: Introduce MMIO ops
* Will Deacon [2020-04-30 11:14:32]:
> > +#ifdef CONFIG_VIRTIO_MMIO_OPS
> >
> > +static struct virtio_mmio_ops *mmio_ops;
> > +
> > +#define virtio_readb(a)mmio_ops->mmio_readl((a))
> > +#define virtio_readw(a)mmio_ops->mmio_readl((a))
> > +#define virtio_readl(a)mmio_ops->mmio_readl((a))
> > +#define virtio_writeb(val, a) mmio_ops->mmio_writeb((val), (a))
> > +#define virtio_writew(val, a) mmio_ops->mmio_writew((val), (a))
> > +#define virtio_writel(val, a) mmio_ops->mmio_writel((val), (a))
>
> How exactly are these ops hooked up? I'm envisaging something like:
>
> ops = spec_compliant_ops;
> [...]
> if (firmware_says_hypervisor_is_buggy())
> ops = magic_qcom_ops;
>
> am I wrong?
If CONFIG_VIRTIO_MMIO_OPS is defined, then I expect this to be unconditionally
set to 'magic_qcom_ops' that uses hypervisor-supported interface for IO (for
example: message_queue_send() and message_queue_recevie() hypercalls).
> > +int register_virtio_mmio_ops(struct virtio_mmio_ops *ops)
> > +{
> > + pr_info("Registered %s as mmio ops\n", ops->name);
> > + mmio_ops = ops;
>
> Not looking good, and really defeats the point of standardising this stuff
> imo.
Ok. I guess the other option is to standardize on a new virtio transport (like
ivshmem2-virtio)?
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC/PATCH 0/1] virtio_mmio: hypervisor specific interfaces for MMIO
* Will Deacon [2020-04-30 11:08:22]: > > This patch is meant to seek comments. If its considered to be in right > > direction, will work on making it more complete and send the next version! > > What's stopping you from implementing the trapping support in the > hypervisor? Unlike the other patches you sent out, where the guest memory > is not accessible to the host, there doesn't seem to be any advantage to > not having trapping support, or am I missing something here? Hi Will, I have had this discussion with hypervisor folks. They seem to be concerned about complexity of having a VM's fault be handled in another untrusted VM. They are not keen to add MMIO support. -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[RFC/PATCH 1/1] virtio: Introduce MMIO ops
Some hypervisors may not support MMIO transport i.e trap config
space access and have it be handled by backend driver. They may
allow other ways to interact with backend such as message-queue
or doorbell API. This patch allows for hypervisor specific
methods for config space IO.
Signed-off-by: Srivatsa Vaddagiri
---
drivers/virtio/virtio_mmio.c | 131 ++-
include/linux/virtio.h | 14 +
2 files changed, 94 insertions(+), 51 deletions(-)
diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
index 97d5725..69bfa35 100644
--- a/drivers/virtio/virtio_mmio.c
+++ b/drivers/virtio/virtio_mmio.c
@@ -100,7 +100,35 @@ struct virtio_mmio_vq_info {
struct list_head node;
};
+#ifdef CONFIG_VIRTIO_MMIO_OPS
+static struct virtio_mmio_ops *mmio_ops;
+
+#define virtio_readb(a)mmio_ops->mmio_readl((a))
+#define virtio_readw(a)mmio_ops->mmio_readl((a))
+#define virtio_readl(a)mmio_ops->mmio_readl((a))
+#define virtio_writeb(val, a) mmio_ops->mmio_writeb((val), (a))
+#define virtio_writew(val, a) mmio_ops->mmio_writew((val), (a))
+#define virtio_writel(val, a) mmio_ops->mmio_writel((val), (a))
+
+int register_virtio_mmio_ops(struct virtio_mmio_ops *ops)
+{
+ pr_info("Registered %s as mmio ops\n", ops->name);
+ mmio_ops = ops;
+
+ return 0;
+}
+
+#else /* CONFIG_VIRTIO_MMIO_OPS */
+
+#define virtio_readb(a)readb((a))
+#define virtio_readw(a)readw((a))
+#define virtio_readl(a)readl((a))
+#define virtio_writeb(val, a) writeb((val), (a))
+#define virtio_writew(val, a) writew((val), (a))
+#define virtio_writel(val, a) writel((val), (a))
+
+#endif /* CONFIG_VIRTIO_MMIO_OPS */
/* Configuration interface */
@@ -109,12 +137,12 @@ static u64 vm_get_features(struct virtio_device *vdev)
struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev);
u64 features;
- writel(1, vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES_SEL);
- features = readl(vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES);
+ virtio_writel(1, vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES_SEL);
+ features = virtio_readl(vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES);
features <<= 32;
- writel(0, vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES_SEL);
- features |= readl(vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES);
+ virtio_writel(0, vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES_SEL);
+ features |= virtio_readl(vm_dev->base + VIRTIO_MMIO_DEVICE_FEATURES);
return features;
}
@@ -133,12 +161,12 @@ static int vm_finalize_features(struct virtio_device
*vdev)
return -EINVAL;
}
- writel(1, vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES_SEL);
- writel((u32)(vdev->features >> 32),
+ virtio_writel(1, vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES_SEL);
+ virtio_writel((u32)(vdev->features >> 32),
vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES);
- writel(0, vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES_SEL);
- writel((u32)vdev->features,
+ virtio_writel(0, vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES_SEL);
+ virtio_writel((u32)vdev->features,
vm_dev->base + VIRTIO_MMIO_DRIVER_FEATURES);
return 0;
@@ -158,25 +186,25 @@ static void vm_get(struct virtio_device *vdev, unsigned
offset,
int i;
for (i = 0; i < len; i++)
- ptr[i] = readb(base + offset + i);
+ ptr[i] = virtio_readb(base + offset + i);
return;
}
switch (len) {
case 1:
- b = readb(base + offset);
+ b = virtio_readb(base + offset);
memcpy(buf, , sizeof b);
break;
case 2:
- w = cpu_to_le16(readw(base + offset));
+ w = cpu_to_le16(virtio_readw(base + offset));
memcpy(buf, , sizeof w);
break;
case 4:
- l = cpu_to_le32(readl(base + offset));
+ l = cpu_to_le32(virtio_readl(base + offset));
memcpy(buf, , sizeof l);
break;
case 8:
- l = cpu_to_le32(readl(base + offset));
+ l = cpu_to_le32(virtio_readl(base + offset));
memcpy(buf, , sizeof l);
l = cpu_to_le32(ioread32(base + offset + sizeof l));
memcpy(buf + sizeof l, , sizeof l);
@@ -200,7 +228,7 @@ static void vm_set(struct virtio_device *vdev, unsigned
offset,
int i;
for (i = 0; i < len; i++)
- writeb(ptr[i], base + offset + i);
+ virtio_writeb(ptr[i], base + offset + i);
return;
[RFC/PATCH 0/1] virtio_mmio: hypervisor specific interfaces for MMIO
The Type-1 hypervisor we are dealing with does not allow for MMIO transport. [1] summarizes some of the problems we have in making virtio work on such hypervisors. This patch proposes a solution for transport problem viz how we can do config space IO on such a hypervisor. Hypervisor specific methods introduced allows for seamless IO of config space. This patch is meant to seek comments. If its considered to be in right direction, will work on making it more complete and send the next version! 1. https://lkml.org/lkml/2020/4/28/427 Srivatsa Vaddagiri (1): virtio: Introduce MMIO ops drivers/virtio/virtio_mmio.c | 131 ++- include/linux/virtio.h | 14 + 2 files changed, 94 insertions(+), 51 deletions(-) -- 2.7.4 -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 5/5] virtio: Add bounce DMA ops
* Michael S. Tsirkin [2020-04-29 06:20:48]: > On Wed, Apr 29, 2020 at 03:39:53PM +0530, Srivatsa Vaddagiri wrote: > > That would still not work I think where swiotlb is used for pass-thr devices > > (when private memory is fine) as well as virtio devices (when shared memory > > is > > required). > > So that is a separate question. When there are multiple untrusted > devices, at the moment it looks like a single bounce buffer is used. > > Which to me seems like a security problem, I think we should protect > untrusted devices from each other. I think as first step, let me see if we can make swiotlb driver accept a target memory segment as its working area. That may suffice our needs I think. A subsequent step could be to make swiotlb driver recognize multiple pools. -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 5/5] virtio: Add bounce DMA ops
* Michael S. Tsirkin [2020-04-29 05:52:05]: > > > So it seems that with modern Linux, all one needs > > > to do on x86 is mark the device as untrusted. > > > It's already possible to do this with ACPI and with OF - would that be > > > sufficient for achieving what this patchset is trying to do? > > > > In my case, its not sufficient to just mark virtio device untrusted and thus > > activate the use of swiotlb. All of the secondary VM memory, including those > > allocate by swiotlb driver, is private to it. > > So why not make the bounce buffer memory shared then? Its a limitation by our hypervisor. When a secondary VM is created, two memory segments are allocated - one private and other shared. There is no provision for the secondary VM to make part of its private memory shared after it boots. I can perhaps consider a change in swiotlb driver to accept the second shared memory segment as its main working area (rather than allocate its own). That would still not work I think where swiotlb is used for pass-thr devices (when private memory is fine) as well as virtio devices (when shared memory is required). -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 5/5] virtio: Add bounce DMA ops
* Michael S. Tsirkin [2020-04-29 02:50:41]: > So it seems that with modern Linux, all one needs > to do on x86 is mark the device as untrusted. > It's already possible to do this with ACPI and with OF - would that be > sufficient for achieving what this patchset is trying to do? In my case, its not sufficient to just mark virtio device untrusted and thus activate the use of swiotlb. All of the secondary VM memory, including those allocate by swiotlb driver, is private to it. An additional piece of memory is available to secondary VM which is shared between VMs and which is where I need swiotlb driver to do its work. -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 5/5] virtio: Add bounce DMA ops
* Stefano Stabellini [2020-04-28 16:04:34]:
> > > Is swiotlb commonly used for multiple devices that may be on different
> > > trust
> > > boundaries (and not behind a hardware iommu)?
>
> The trust boundary is not a good way of describing the scenario and I
> think it leads to miscommunication.
>
> A better way to describe the scenario would be that the device can only
> DMA to/from a small reserved-memory region advertised on device tree.
>
> Do we have other instances of devices that can only DMA to/from very
> specific and non-configurable address ranges? If so, this series could
> follow their example.
AFAICT there is no such notion in current DMA API.
static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size,
bool is_ram)
{
return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit);
}
Only the max address a device can access is defined and not a range that we seem
to need here. I think we need to set the bus_dma_limit to 0 for virtio devices
which will force the use of swiotlb_map API. We should also have a per-device
swiotlb pool defined, so that swiotlb can use the pool meant for the given
device.
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 5/5] virtio: Add bounce DMA ops
* Michael S. Tsirkin [2020-04-28 16:41:04]:
> > Won't we still need some changes to virtio to make use of its own pool (to
> > bounce buffers)? Something similar to its own DMA ops proposed in this
> > patch?
>
> If you are doing this for all devices, you need to either find a way
> to do this without chaning DMA ops, or by doing some automatic change
> to all drivers.
Ok thanks for this input. I will see how we can obfuscate this in DMA APIs
itself.
Can you also comment on the virtio transport problem I cited? The hypervisor we
are dealing with does not support MMIO transport. It supports message queue
send/recv and also doorbell, which I think can be used if we can make some
change like this to virtio_mmio.c:
+static inline u32
+virtio_readl(struct virtio_mmio_device *vm_dev, u32 reg_offset)
+{
+return vm_dev->mmio_ops->readl(vm_dev, reg_offset);
+}
+
+static inline void
+virtio_writel(struct virtio_mmio_device *vm_dev, u32 reg_offset, u32 data)
+{
+vm_dev->mmio_ops->writel(vm_dev, reg_offset, data);
+}
/* Check magic value */
-magic = readl(vm_dev->base + VIRTIO_MMIO_MAGIC_VALUE);
+magic = vrito_readl(vm_dev, VIRTIO_MMIO_MAGIC_VALUE);
mmio_ops->readl on most platforms can default to readl itself, while on a
platform like us, it can boil down to message_queue send/recv. Would such a
change be acceptable?
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 5/5] virtio: Add bounce DMA ops
* Michael S. Tsirkin [2020-04-28 12:17:57]:
> Okay, but how is all this virtio specific? For example, why not allow
> separate swiotlbs for any type of device?
> For example, this might make sense if a given device is from a
> different, less trusted vendor.
Is swiotlb commonly used for multiple devices that may be on different trust
boundaries (and not behind a hardware iommu)? If so, then yes it sounds like a
good application of multiple swiotlb pools.
> All this can then maybe be hidden behind the DMA API.
Won't we still need some changes to virtio to make use of its own pool (to
bounce buffers)? Something similar to its own DMA ops proposed in this patch?
> > +void virtio_bounce_set_dma_ops(struct virtio_device *vdev)
> > +{
> > + if (!bounce_buf_paddr)
> > + return;
> > +
> > + set_dma_ops(vdev->dev.parent, _dma_ops);
>
>
> I don't think DMA API maintainers will be happy with new users
> of set_dma_ops.
Is there an alternate API that is more preffered?
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH 5/5] virtio: Add bounce DMA ops
For better security, its desirable that a guest VM's memory is
not accessible to any entity that executes outside the context of
guest VM. In case of virtio, backend drivers execute outside the
context of guest VM and in general will need access to complete
guest VM memory. One option to restrict the access provided to
backend driver is to make use of a bounce buffer. The bounce
buffer is accessible to both backend and frontend drivers. All IO
buffers that are in private space of guest VM are bounced to be
accessible to backend.
This patch proposes a new memory pool to be used for this bounce
purpose, rather than the default swiotlb memory pool. That will
avoid any conflicts that may arise in situations where a VM needs
to use swiotlb pool for driving any pass-through devices (in
which case swiotlb memory needs not be shared with another VM) as
well as virtio devices (which will require swiotlb memory to be
shared with backend VM). As a possible extension to this patch,
we can provide an option for virtio to make use of default
swiotlb memory pool itself, where no such conflicts may exist in
a given deployment.
Signed-off-by: Srivatsa Vaddagiri
---
drivers/virtio/Makefile| 2 +-
drivers/virtio/virtio.c| 2 +
drivers/virtio/virtio_bounce.c | 150 +
include/linux/virtio.h | 4 ++
4 files changed, 157 insertions(+), 1 deletion(-)
create mode 100644 drivers/virtio/virtio_bounce.c
diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile
index 29a1386e..3fd3515 100644
--- a/drivers/virtio/Makefile
+++ b/drivers/virtio/Makefile
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o
+obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o virtio_bounce.o
obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o
obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o
virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o
diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index a977e32..bc2f779 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -329,6 +329,7 @@ int register_virtio_device(struct virtio_device *dev)
dev->index = err;
dev_set_name(>dev, "virtio%u", dev->index);
+ virtio_bounce_set_dma_ops(dev);
spin_lock_init(>config_lock);
dev->config_enabled = false;
@@ -431,6 +432,7 @@ EXPORT_SYMBOL_GPL(virtio_device_restore);
static int virtio_init(void)
{
+ virtio_map_bounce_buffer();
if (bus_register(_bus) != 0)
panic("virtio bus registration failed");
return 0;
diff --git a/drivers/virtio/virtio_bounce.c b/drivers/virtio/virtio_bounce.c
new file mode 100644
index 000..3de8e0e
--- /dev/null
+++ b/drivers/virtio/virtio_bounce.c
@@ -0,0 +1,150 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Virtio DMA ops to bounce buffers
+ *
+ * Copyright (c) 2020, The Linux Foundation. All rights reserved.
+ *
+ * This module allows bouncing of IO buffers to a region which will be
+ * accessible to backend drivers.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+static phys_addr_t bounce_buf_paddr;
+static void *bounce_buf_vaddr;
+static size_t bounce_buf_size;
+struct swiotlb_pool *virtio_pool;
+
+#define VIRTIO_MAX_BOUNCE_SIZE (16*4096)
+
+static void *virtio_alloc_coherent(struct device *dev, size_t size,
+ dma_addr_t *dma_handle, gfp_t gfp_flags, unsigned long attrs)
+{
+ phys_addr_t addr;
+
+ if (!virtio_pool)
+ return NULL;
+
+ addr = swiotlb_alloc(virtio_pool, size, bounce_buf_paddr, ULONG_MAX);
+ if (addr == DMA_MAPPING_ERROR)
+ return NULL;
+
+ *dma_handle = (addr - bounce_buf_paddr);
+
+ return bounce_buf_vaddr + (addr - bounce_buf_paddr);
+}
+
+static void virtio_free_coherent(struct device *dev, size_t size, void *vaddr,
+ dma_addr_t dma_handle, unsigned long attrs)
+{
+ phys_addr_t addr = (dma_handle + bounce_buf_paddr);
+
+ swiotlb_free(virtio_pool, addr, size);
+}
+
+static dma_addr_t virtio_map_page(struct device *dev, struct page *page,
+ unsigned long offset, size_t size,
+ enum dma_data_direction dir, unsigned long attrs)
+{
+ void *ptr = page_address(page) + offset;
+ phys_addr_t paddr = virt_to_phys(ptr);
+ dma_addr_t handle;
+
+ if (!virtio_pool)
+ return DMA_MAPPING_ERROR;
+
+ handle = _swiotlb_tbl_map_single(virtio_pool, dev, bounce_buf_paddr,
+paddr, size, size, dir, attrs);
+ if (handle == (phys_addr_t)DMA_MAPPING_ERROR)
+ return DMA_MAPPING_ERROR;
+
+ return handle - bounce_buf_paddr;
+}
+
+static void virtio_unmap_page(struct device *dev, dma_addr_t dev_addr,
+ size_t size, enum dma_data_direction dir, unsigned long attrs)
+{
+ phys_addr_t
[PATCH 3/5] swiotlb: Add alloc and free APIs
Move the memory allocation and free portion of swiotlb driver
into independent routines. They will be useful for drivers that
need swiotlb driver to just allocate/free memory chunks and not
additionally bounce memory.
Signed-off-by: Srivatsa Vaddagiri
---
include/linux/swiotlb.h | 17 ++
kernel/dma/swiotlb.c| 151
2 files changed, 106 insertions(+), 62 deletions(-)
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index c634b4d..957697e 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -186,6 +186,10 @@ void __init swiotlb_exit(void);
unsigned int swiotlb_max_segment(void);
size_t swiotlb_max_mapping_size(struct device *dev);
bool is_swiotlb_active(void);
+extern phys_addr_t swiotlb_alloc(struct swiotlb_pool *pool, size_t alloc_size,
+ unsigned long tbl_dma_addr, unsigned long mask);
+extern void swiotlb_free(struct swiotlb_pool *pool,
+ phys_addr_t tlb_addr, size_t alloc_size);
#else
#define swiotlb_force SWIOTLB_NO_FORCE
@@ -219,6 +223,19 @@ static inline bool is_swiotlb_active(void)
{
return false;
}
+
+static inline phys_addr_t swiotlb_alloc(struct swiotlb_pool *pool,
+ size_t alloc_size, unsigned long tbl_dma_addr,
+ unsigned long mask)
+{
+ return DMA_MAPPING_ERROR;
+}
+
+static inline void swiotlb_free(struct swiotlb_pool *pool,
+ phys_addr_t tlb_addr, size_t alloc_size)
+{
+}
+
#endif /* CONFIG_SWIOTLB */
extern void swiotlb_print_info(void);
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 8cf0b57..7411ce5 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -444,37 +444,14 @@ static inline void *tlb_vaddr(struct swiotlb_pool *pool,
phys_addr_t tlb_addr)
return pool->io_tlb_vstart + (tlb_addr - pool->io_tlb_start);
}
-phys_addr_t _swiotlb_tbl_map_single(struct swiotlb_pool *pool,
- struct device *hwdev,
- dma_addr_t tbl_dma_addr,
- phys_addr_t orig_addr,
- size_t mapping_size,
- size_t alloc_size,
- enum dma_data_direction dir,
- unsigned long attrs)
+phys_addr_t swiotlb_alloc(struct swiotlb_pool *pool, size_t alloc_size,
+ unsigned long tbl_dma_addr, unsigned long mask)
{
unsigned long flags;
phys_addr_t tlb_addr;
- unsigned int nslots, stride, index, wrap;
- int i;
- unsigned long mask;
+ unsigned int i, nslots, stride, index, wrap;
unsigned long offset_slots;
unsigned long max_slots;
- unsigned long tmp_io_tlb_used;
-
- if (pool->no_iotlb_memory)
- panic("Can not allocate SWIOTLB buffer earlier and can't now
provide you with the DMA bounce buffer");
-
- if (mem_encrypt_active())
- pr_warn_once("Memory encryption is active and system is using
DMA bounce buffers\n");
-
- if (mapping_size > alloc_size) {
- dev_warn_once(hwdev, "Invalid sizes (mapping: %zd bytes, alloc:
%zd bytes)",
- mapping_size, alloc_size);
- return (phys_addr_t)DMA_MAPPING_ERROR;
- }
-
- mask = dma_get_seg_boundary(hwdev);
tbl_dma_addr &= mask;
@@ -555,54 +532,23 @@ phys_addr_t _swiotlb_tbl_map_single(struct swiotlb_pool
*pool,
} while (index != wrap);
not_found:
- tmp_io_tlb_used = pool->io_tlb_used;
-
spin_unlock_irqrestore(>io_tlb_lock, flags);
- if (!(attrs & DMA_ATTR_NO_WARN) && printk_ratelimit())
- dev_warn(hwdev, "swiotlb buffer is full (sz: %zd bytes), total
%lu (slots), used %lu (slots)\n",
-alloc_size, pool->io_tlb_nslabs, tmp_io_tlb_used);
return (phys_addr_t)DMA_MAPPING_ERROR;
+
found:
pool->io_tlb_used += nslots;
spin_unlock_irqrestore(>io_tlb_lock, flags);
- /*
-* Save away the mapping from the original address to the DMA address.
-* This is needed when we sync the memory. Then we sync the buffer if
-* needed.
-*/
- for (i = 0; i < nslots; i++)
- pool->io_tlb_orig_addr[index+i] = orig_addr +
- (i << IO_TLB_SHIFT);
- if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) &&
- (dir == DMA_TO_DEVICE || dir == DMA_BIDIRECTIONAL))
- swiotlb_bounce(orig_addr, tlb_vaddr(pool, tlb_addr),
- mapping_size, DMA_TO_DEVICE);
-
return tlb_addr;
}
-/*
- * tlb_addr is the physical address of the bounce buffer to unmap.
- */
-void _swiotlb_tbl_unmap_single(struct swiotlb_pool *pool,
-
[PATCH 1/5] swiotlb: Introduce concept of swiotlb_pool
Currently swiotlb driver manages a global pool of memory which
acts as bounce buffers for memory that is not accessible to some
devices. The core functions provides by this driver to
allocate/free/bounce memory chunks will be more
useful if this driver can manage more than one pool. An immediate
application of such extension to swiotlb driver is to bounce
virtio buffers between private and shared space of a VM.
This patch introduces the concept of a swiotlb memory pool and
reorganizes current driver to work with the default global pool.
There is no functional change introduced by this patch.
Subsequent patches allow the swiotlb driver to work with more
than one pool of memory.
Signed-off-by: Srivatsa Vaddagiri
---
drivers/xen/swiotlb-xen.c | 4 +-
include/linux/swiotlb.h | 129 -
kernel/dma/swiotlb.c | 359 +++---
3 files changed, 307 insertions(+), 185 deletions(-)
diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index b6d2776..c2dc9c8 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -190,8 +190,8 @@ int __ref xen_swiotlb_init(int verbose, bool early)
/*
* IO TLB memory already allocated. Just use it.
*/
- if (io_tlb_start != 0) {
- xen_io_tlb_start = phys_to_virt(io_tlb_start);
+ if (swiotlb_start()) {
+ xen_io_tlb_start = phys_to_virt(swiotlb_start());
goto end;
}
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 046bb94..8c7843f 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -44,7 +44,59 @@ enum dma_sync_target {
SYNC_FOR_DEVICE = 1,
};
-extern phys_addr_t swiotlb_tbl_map_single(struct device *hwdev,
+#define MAX_POOL_NAME_SIZE 16
+
+struct swiotlb_pool {
+ char name[MAX_POOL_NAME_SIZE];
+ bool no_iotlb_memory;
+ int late_alloc;
+
+ spinlock_t io_tlb_lock;
+
+ /*
+* Used to do a quick range check in swiotlb_tbl_unmap_single and
+* swiotlb_tbl_sync_single_*, to see if the memory was in fact allocated
+* by this API.
+*/
+
+ phys_addr_t io_tlb_start, io_tlb_end;
+
+ /*
+* The number of IO TLB blocks (in groups of 64) between io_tlb_start
+* and io_tlb_end. This is command line adjustable via
+* setup_io_tlb_npages.
+*/
+ unsigned long io_tlb_nslabs;
+
+ /*
+* The number of used IO TLB block
+*/
+ unsigned long io_tlb_used;
+
+ /*
+* This is a free list describing the number of free entries available
+* from each index
+*/
+ unsigned int *io_tlb_list;
+ unsigned int io_tlb_index;
+
+ /*
+* We need to save away the original address corresponding to a mapped
+* entry for the sync operations.
+*/
+ phys_addr_t *io_tlb_orig_addr;
+
+ /*
+* Max segment that we can provide which (if pages are contingous) will
+* not be bounced (unless SWIOTLB_FORCE is set).
+*/
+ unsigned int max_segment;
+};
+
+extern struct swiotlb_pool default_swiotlb_pool;
+
+extern phys_addr_t _swiotlb_tbl_map_single(struct swiotlb_pool *pool,
+ struct device *hwdev,
dma_addr_t tbl_dma_addr,
phys_addr_t phys,
size_t mapping_size,
@@ -52,28 +104,80 @@ extern phys_addr_t swiotlb_tbl_map_single(struct device
*hwdev,
enum dma_data_direction dir,
unsigned long attrs);
-extern void swiotlb_tbl_unmap_single(struct device *hwdev,
+extern void _swiotlb_tbl_unmap_single(struct swiotlb_pool *pool,
+struct device *hwdev,
phys_addr_t tlb_addr,
size_t mapping_size,
size_t alloc_size,
enum dma_data_direction dir,
unsigned long attrs);
-extern void swiotlb_tbl_sync_single(struct device *hwdev,
+extern void _swiotlb_tbl_sync_single(struct swiotlb_pool *pool,
+ struct device *hwdev,
phys_addr_t tlb_addr,
size_t size, enum dma_data_direction dir,
enum dma_sync_target target);
-dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys,
- size_t size, enum dma_data_direction dir, unsigned long attrs);
+dma_addr_t _swiotlb_map(struct swiotlb_pool *pool, struct device *dev,
+ phys_addr_t phys, size_t size, enum dma_data_direction dir,
+ unsigned long attrs);
+
+static inline phys_addr_t
[PATCH 4/5] swiotlb: Add API to register new pool
This patch adds an interface for the swiotlb driver to recognize
a new memory pool. Upon successful initialization of the pool,
swiotlb returns a handle, which needs to be passed as an argument
for any future operations on the pool (map/unmap/alloc/free).
Signed-off-by: Srivatsa Vaddagiri
---
include/linux/swiotlb.h | 9
kernel/dma/swiotlb.c| 60 +
2 files changed, 69 insertions(+)
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 957697e..97ac82a 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -182,6 +182,9 @@ static inline bool is_swiotlb_buffer(phys_addr_t paddr)
return paddr >= swiotlb_start() && paddr < swiotlb_end();
}
+extern struct swiotlb_pool *swiotlb_register_pool(char *name,
+ phys_addr_t start, void *vstart, size_t size);
+
void __init swiotlb_exit(void);
unsigned int swiotlb_max_segment(void);
size_t swiotlb_max_mapping_size(struct device *dev);
@@ -236,6 +239,12 @@ static inline void swiotlb_free(struct swiotlb_pool *pool,
{
}
+static struct swiotlb_pool *swiotlb_register_pool(char *name,
+ phys_addr_t start, void *vstart, size_t size)
+{
+ return NULL;
+}
+
#endif /* CONFIG_SWIOTLB */
extern void swiotlb_print_info(void);
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 7411ce5..9883744 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -36,6 +36,7 @@
#include
#include
#include
+#include
#ifdef CONFIG_DEBUG_FS
#include
#endif
@@ -736,6 +737,65 @@ size_t swiotlb_max_mapping_size(struct device *dev)
return ((size_t)1 << IO_TLB_SHIFT) * IO_TLB_SEGSIZE;
}
+struct swiotlb_pool *swiotlb_register_pool(char *name, phys_addr_t start,
+ void *vstart, size_t size)
+{
+ struct swiotlb_pool *pool;
+ unsigned long i, bytes;
+ unsigned long nslabs;
+
+ nslabs = size >> IO_TLB_SHIFT;
+ if (!nslabs)
+ return ERR_PTR(-EINVAL);
+
+ pool = kzalloc(sizeof(*pool), GFP_KERNEL);
+ if (!pool)
+ return ERR_PTR(-ENOMEM);
+
+ bytes = nslabs << IO_TLB_SHIFT;
+
+ strncpy(pool->name, name, sizeof(pool->name));
+ spin_lock_init(>io_tlb_lock);
+ pool->late_alloc = 1;
+ pool->io_tlb_start = start;
+ pool->io_tlb_end = start + bytes;
+ pool->io_tlb_vstart = vstart;
+ pool->io_tlb_nslabs = nslabs;
+ pool->max_segment = rounddown(bytes, PAGE_SIZE);
+
+ /*
+* Allocate and initialize the free list array. This array is used
+* to find contiguous free memory regions of size up to IO_TLB_SEGSIZE
+* between io_tlb_start and io_tlb_end.
+*/
+ pool->io_tlb_list = (unsigned int *)__get_free_pages(GFP_KERNEL,
+ get_order(pool->io_tlb_nslabs * sizeof(int)));
+ if (!pool->io_tlb_list)
+ goto cleanup;
+
+ pool->io_tlb_orig_addr = (phys_addr_t *)
+ __get_free_pages(GFP_KERNEL,
+get_order(pool->io_tlb_nslabs *
+ sizeof(phys_addr_t)));
+ if (!pool->io_tlb_orig_addr)
+ goto cleanup;
+
+ for (i = 0; i < pool->io_tlb_nslabs; i++) {
+ pool->io_tlb_list[i] = IO_TLB_SEGSIZE -
+ OFFSET(i, IO_TLB_SEGSIZE);
+ pool->io_tlb_orig_addr[i] = INVALID_PHYS_ADDR;
+ }
+
+ return pool;
+
+cleanup:
+ kfree(pool->io_tlb_list);
+ kfree(pool->io_tlb_orig_addr);
+ kfree(pool);
+
+ return ERR_PTR(-ENOMEM);
+}
+
bool is_swiotlb_active(void)
{
/*
--
2.7.4
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH 0/5] virtio on Type-1 hypervisor
We ran into several problems in using virtio for IO paravirtualization on a Type-1 hypervisor with these characteristics: * By default, all of a guests's memory is private to it (no other guest can access its memory). * One of the VM is considered as primary and has access to most IO devices. This is similar to dom0 VM in case of Xen. All other VMs are considered as secondary VMs * virtio-backend drivers for all secondary VMs need to be hosted in primary VM * Since secondary VM's memory is not accessible to primary VM, to make virtio backend driver work, instead an additional piece of memory is provisioned by the hypervisor that is shared between primary and secondary VMs. This shared memory can be used, for example, to host virtio-ring structures and also to bounce IO buffers of secondary VM. * Message-queue and doorbell interfaces available in hypervisor to support inter-VM communication. Messge-queue API (send/recv) allows one VM to send short messages to another VM. Doorbell interface allows a VM to inject an interrupt into another VM. * No support for MMIO transport i.e hypervisor does not support trapping MMIO config space access by front-end driver and having it handled in backend drivers. Few problem statements arising out of this: 1) How can we make use of the shared memory region effectively to make virtio work in this scenario? What is proposed in the patch series for this problem is a virtio bounce driver that recognizes a shared memory region (shared between VMs) and makes use of swiotlb driver interfaces to bounce IO buffers between private and shared space. Some changes are proposed to swiotlb driver in this regard, that can let us reuse swiotlb functions to work with the shared memory pool. swiotlb driver can now recognize more than one pool of memory and extend its functions (allocate/free/bounce memory chunks) for each pool. 2) What transport mechanism works best in this case? I realize that ivshmem2-virtio proposal has discussed the possibility of using shared memory + doorbell as a virtio transport option. We can consider using that as a transport in case it will be acceptable upstream. Other option we had considered was to modify virtio_mmio.c itself to use message_queue send/recv hypercall interface (in place of readl/writel). That could be abstracted via 'mmio_ops' structure providing suitable implementation of readl/writel. Another option suggested by Christopher Dall is to unmap the config space from kernel address space and as part of the fault handler, use hypervisor specific APIs to achieve config space IO. 3) Which virtio backend drivers to leverage? We realized there are multiple implementations of virtio backend drivers, bundled as part of separate projects (Qemu, lkvm etc). We think it would be nice if we had some consolidation in that regard so that we can make use of the backend drivers that are not tightly coupled with a VMM. In our case, we need to be able to run virtio backend drivers as standalone programs (and not coupled with any VMM). Srivatsa Vaddagiri (5): swiotlb: Introduce concept of swiotlb_pool swiotlb: Allow for non-linear mapping between paddr and vaddr swiotlb: Add alloc and free APIs swiotlb: Add API to register new pool virtio: Add bounce DMA ops drivers/virtio/Makefile| 2 +- drivers/virtio/virtio.c| 2 + drivers/virtio/virtio_bounce.c | 150 +++ drivers/xen/swiotlb-xen.c | 4 +- include/linux/swiotlb.h| 157 +++- include/linux/virtio.h | 4 + kernel/dma/swiotlb.c | 556 - 7 files changed, 638 insertions(+), 237 deletions(-) create mode 100644 drivers/virtio/virtio_bounce.c -- 2.7.4 -- QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH 2/5] swiotlb: Allow for non-linear mapping between paddr and vaddr
Some of the memory pool managed by swiotlb driver could fall
outside the direct-mapped range, made accessible via memremap()
routine. To facilitate easy conversion between virtual and
physical address of such memory, store the virtual address of
memory pool in addition to its physical address.
Signed-off-by: Srivatsa Vaddagiri
---
include/linux/swiotlb.h | 2 ++
kernel/dma/swiotlb.c| 20 ++--
2 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 8c7843f..c634b4d 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -61,6 +61,8 @@ struct swiotlb_pool {
phys_addr_t io_tlb_start, io_tlb_end;
+ void *io_tlb_vstart;
+
/*
* The number of IO TLB blocks (in groups of 64) between io_tlb_start
* and io_tlb_end. This is command line adjustable via
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 9c504d3..8cf0b57 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -178,6 +178,7 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned long
nslabs, int verbose)
default_swiotlb_pool.io_tlb_start = __pa(tlb);
default_swiotlb_pool.io_tlb_end =
default_swiotlb_pool.io_tlb_start + bytes;
+ default_swiotlb_pool.io_tlb_vstart = tlb;
/*
* Allocate and initialize the free list array. This array is used
@@ -307,6 +308,7 @@ static void swiotlb_cleanup(void)
default_swiotlb_pool.io_tlb_start = 0;
default_swiotlb_pool.io_tlb_nslabs = 0;
default_swiotlb_pool.max_segment = 0;
+ default_swiotlb_pool.io_tlb_vstart = NULL;
}
int
@@ -320,6 +322,7 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
default_swiotlb_pool.io_tlb_start = virt_to_phys(tlb);
default_swiotlb_pool.io_tlb_end =
default_swiotlb_pool.io_tlb_start + bytes;
+ default_swiotlb_pool.io_tlb_vstart = tlb;
set_memory_decrypted((unsigned long)tlb, bytes >> PAGE_SHIFT);
memset(tlb, 0, bytes);
@@ -400,11 +403,10 @@ void __init swiotlb_exit(void)
/*
* Bounce: copy the swiotlb buffer from or back to the original dma location
*/
-static void swiotlb_bounce(phys_addr_t orig_addr, phys_addr_t tlb_addr,
+static void swiotlb_bounce(phys_addr_t orig_addr, void *vaddr,
size_t size, enum dma_data_direction dir)
{
unsigned long pfn = PFN_DOWN(orig_addr);
- unsigned char *vaddr = phys_to_virt(tlb_addr);
if (PageHighMem(pfn_to_page(pfn))) {
/* The buffer does not have a mapping. Map it in and copy */
@@ -437,6 +439,11 @@ static void swiotlb_bounce(phys_addr_t orig_addr,
phys_addr_t tlb_addr,
}
}
+static inline void *tlb_vaddr(struct swiotlb_pool *pool, phys_addr_t tlb_addr)
+{
+ return pool->io_tlb_vstart + (tlb_addr - pool->io_tlb_start);
+}
+
phys_addr_t _swiotlb_tbl_map_single(struct swiotlb_pool *pool,
struct device *hwdev,
dma_addr_t tbl_dma_addr,
@@ -569,7 +576,7 @@ phys_addr_t _swiotlb_tbl_map_single(struct swiotlb_pool
*pool,
(i << IO_TLB_SHIFT);
if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) &&
(dir == DMA_TO_DEVICE || dir == DMA_BIDIRECTIONAL))
- swiotlb_bounce(orig_addr, tlb_addr,
+ swiotlb_bounce(orig_addr, tlb_vaddr(pool, tlb_addr),
mapping_size, DMA_TO_DEVICE);
return tlb_addr;
@@ -594,7 +601,8 @@ void _swiotlb_tbl_unmap_single(struct swiotlb_pool *pool,
if (orig_addr != INVALID_PHYS_ADDR &&
!(attrs & DMA_ATTR_SKIP_CPU_SYNC) &&
((dir == DMA_FROM_DEVICE) || (dir == DMA_BIDIRECTIONAL)))
- swiotlb_bounce(orig_addr, tlb_addr, mapping_size,
DMA_FROM_DEVICE);
+ swiotlb_bounce(orig_addr, tlb_vaddr(pool, tlb_addr),
+ mapping_size, DMA_FROM_DEVICE);
/*
* Return the buffer to the free list by setting the corresponding
@@ -643,14 +651,14 @@ void _swiotlb_tbl_sync_single(struct swiotlb_pool *pool,
switch (target) {
case SYNC_FOR_CPU:
if (likely(dir == DMA_FROM_DEVICE || dir == DMA_BIDIRECTIONAL))
- swiotlb_bounce(orig_addr, tlb_addr,
+ swiotlb_bounce(orig_addr, tlb_vaddr(pool, tlb_addr),
size, DMA_FROM_DEVICE);
else
BUG_ON(dir != DMA_TO_DEVICE);
break;
case SYNC_FOR_DEVICE:
if (likely(dir == DMA_TO_DEVICE || dir == DMA_BIDIRECTIONAL))
- swiotlb_bounce(orig_addr, tlb_addr,
+ swiotlb_bounce(orig_addr, tlb_vaddr(pool, tlb_addr),
