From: Krishna Prabhakaran <[email protected]>
When i915 needs to make an imported dma-buf coherent for GPU access on
non-LLC platforms, or for objects that bypass LLC, it currently calls
wbinvd_on_all_cpus(). get_pages() runs whenever an imported buffer is
pinned, so this triggers a whole-cache write-back and invalidate,
broadcast by IPI to every CPU, on every execbuf submission involving an
imported dma-buf. That stalls the entire machine for milliseconds and
starves latency-sensitive work on unrelated cores (e.g. USB isochronous
audio serviced on the VMM's main thread).
Flush only the pages that actually need it instead:
- If we imported one of our own dma-bufs, the backing object is
struct-page backed once migrated to SMEM, so flush it directly with
drm_clflush_sg(), exactly as we flush our other objects. This also
avoids re-entering the exporter through dma_buf_vmap(), which would
recurse into i915_gem_object_pin_map() on the source object we
already hold locked (and fails the igt_dmabuf_import_same_driver
selftest with -EBUSY).
- For a foreign dma-buf the sg_table is not guaranteed to be backed by
struct pages, and the importer has no way to tell, so drm_clflush_sg()
cannot be used. vmap the buffer and flush that virtual range with
drm_clflush_virt_range() instead: x86 uses PIPT caches, so flushing
one virtual alias evicts the cache lines for every alias of the same
physical pages. The dma_resv lock required by dma_buf_vmap() is
already held here via the imported object.
Fall back to wbinvd only when the buffer cannot be vmapped or is backed
by I/O memory, where there is no CPU-side range to clflush.
Fixes: a035154da45d ("drm/i915/dmabuf: add paranoid flush-on-acquire")
Signed-off-by: Krishna Prabhakaran <[email protected]>
---
v2:
- Flush our own imported dma-bufs directly with drm_clflush_sg() instead of
dma_buf_vmap(), which re-entered i915_gem_object_pin_map() on the source
object and failed igt_dmabuf_import_same_driver_smem with -EBUSY (reported
by Intel CI on v1). Foreign dma-bufs still use dma_buf_vmap() +
drm_clflush_virt_range(); wbinvd only as fallback.
- Link to v1: https://patchwork.freedesktop.org/patch/744955/?series=171760
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 55 ++++++++++++++++++----
1 file changed, 47 insertions(+), 8 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index b43d34c7d641..c798a90f1c0f 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -10,6 +10,8 @@
#include <asm/smp.h>
+#include <drm/drm_cache.h>
+
#include "gem/i915_gem_dmabuf.h"
#include "i915_drv.h"
#include "i915_gem_object.h"
@@ -249,16 +251,53 @@ static int i915_gem_object_get_pages_dmabuf(struct
drm_i915_gem_object *obj)
* DG1 is special here since it still snoops transactions even with
* CACHE_NONE. This is not the case with other HAS_SNOOP platforms. We
* might need to revisit this as we add new discrete platforms.
- *
- * XXX: Consider doing a vmap flush or something, where possible.
- * Currently we just do a heavy handed wbinvd_on_all_cpus() here since
- * the underlying sg_table might not even point to struct pages, so we
- * can't just call drm_clflush_sg or similar, like we do elsewhere in
- * the driver.
*/
if (i915_gem_object_can_bypass_llc(obj) ||
- (!HAS_LLC(i915) && !IS_DG1(i915)))
- wbinvd_on_all_cpus();
+ (!HAS_LLC(i915) && !IS_DG1(i915))) {
+ struct dma_buf *dma_buf = obj->base.import_attach->dmabuf;
+
+ if (dma_buf->ops == &i915_dmabuf_ops) {
+ struct drm_i915_gem_object *dma_obj =
+ dma_buf_to_obj(dma_buf);
+
+ /*
+ * We imported one of our own dma-bufs. The backing
+ * object is struct-page backed once migrated to SMEM,
+ * so flush it directly, the same way we flush our
+ * other objects. This also avoids re-entering the
+ * exporter through dma_buf_vmap(), which would recurse
+ * into i915_gem_object_pin_map() on the source object
+ * we already hold locked.
+ */
+ if (i915_gem_object_has_struct_page(dma_obj))
+ drm_clflush_sg(dma_obj->mm.pages);
+ else
+ wbinvd_on_all_cpus();
+ } else {
+ struct iosys_map map;
+
+ /*
+ * A foreign sg_table is not guaranteed to be backed by
+ * struct pages, so we cannot use drm_clflush_sg(). vmap
+ * the buffer and flush the virtual range instead; x86
+ * uses PIPT caches, so flushing one alias evicts the
+ * lines for every alias of the same physical pages.
+ *
+ * We already hold the dma_resv lock via the imported
+ * obj, so use the locked dma_buf_vmap() variant.
+ */
+ if (!dma_buf_vmap(dma_buf, &map)) {
+ if (!map.is_iomem)
+ drm_clflush_virt_range(map.vaddr,
+ obj->base.size);
+ else
+ wbinvd_on_all_cpus();
+ dma_buf_vunmap(dma_buf, &map);
+ } else {
+ wbinvd_on_all_cpus();
+ }
+ }
+ }
__i915_gem_object_set_pages(obj, sgt);
base-commit: 682ea2d28d18bb06f9fc663cb5ab7e80dc0e606a
--
2.43.0
