The patch set allows to register a dmabuf to an io_uring instance for
a specified file and use it with io_uring read / write requests. The
infrastructure is not tied to io_uring and there could be more users
in the future. A similar idea was attempted some years ago by Keith [1],
from where I borrowed a good number of changes, and later was brough up
by Tushar and Vishal.
It's an opt-in feature for files, and they need to implement a new
file operation to use it. Only NVMe block devices are supported in this
series. The user API is built on top of io_uring's "registered buffers",
where a dmabuf is registered in a special way, but after it can be used
as any other "registered buffer" with IORING_OP_{READ,WRITE}_FIXED
requests. It's created via a new file operation and the resulted map is
then passed through the I/O stack in a new iterator type. There is some
additional infrastructure to glue it together, count requests, manage
lifetime and implement invalidation.
Tushar, William, Phil as well as Kanchan, Anuj and Nitesh did a lot of
testing and experimentation with different devices. Some earlier
benchmarks by Anuj for IOMMU optimisations with udmabuf showed:
STRICT: before = 570 KIOPS, after = 5.01 MIOPS
LAZY: before = 1.93 MIOPS, after = 5.01 MIOPS
PASSTHROUGH: before = 5.01 MIOPS, after = 5.01 MIOPS
# Patch set structure:
- Patches 1-2 introduce internal API and infrastructure mediating
io_uring and target subsystem / devices
- Patches 3-7: block layer support + prep patches
- Patches 8-9 implement NVMe support
- Patches 10-14 add io_uring support and uapi. Patch 14 specifically
enables dma-buf support, the rest are preparation patches.
There are some liburing tests that can serve as an example:
git: https://github.com/isilence/liburing.git rw-dmabuf-tests-v4
url: https://github.com/isilence/liburing/tree/rw-dmabuf-tests-v4
[1] https://lore.kernel.org/io-uring/[email protected]/
v4: - Add sgl support from Anuj
- Move it under drivers/dma-buf/ and rename
- Fix mis-sized allocations
- Fix io_uring re-import mishandling
- Drop map before io_uring "task work"
- Move blk-mq callback to block_device_operations
- Convert bio flag to REQ_OP*
- Other small changes
v3:
https://lore.kernel.org/io-uring/[email protected]/
- Rework io_uring registration
- Move token/map infrastructure code out of blk-mq
- Simplify callbacks: remove a separate blk-mq table, which was
mostly just forwarding calls (to nvme).
- Don't skip dma sync depending on request direction
- Fix a couple of hangs
- Rename s/dma/dmabuf/
- Other small changes
v2: - Don't pass raw dma addresses, wrap it into a driver specific object
- Split into two objects: token and map
- Implement move_notify
Anuj Gupta (1):
nvme-pci: add SGL support for the dmabuf path
Pavel Begunkov (13):
dma-buf: introduce initial file I/O infrastructure
iov_iter: add iterator type for dmabuf maps
block: rename bi_bvec_done
block: always adjust bi_offset on bio_advance_iter
block: move bvec init into __bio_clone
block: introduce dma map backed bio type
block: forward init_dma_buf_io_ctx to drivers
nvme-pci: implement dma_token backed requests
io_uring/rsrc: introduce buf registration structure
io_uring/rsrc: extend buffer update
io_uring/rsrc: add uncloneable regbuf flag
io_uring/rsrc: add regbuf import flags
io_uring/rsrc: add dmabuf backed registered buffers
Documentation/block/biovecs.rst | 8 +-
block/bio.c | 30 ++-
block/blk-merge.c | 23 +-
block/blk-mq-dma.c | 2 +-
block/blk.h | 2 +-
block/fops.c | 16 ++
drivers/block/loop.c | 2 +-
drivers/block/zloop.c | 2 +-
drivers/dma-buf/Makefile | 2 +-
drivers/dma-buf/dma-buf-io.c | 275 ++++++++++++++++++++
drivers/md/dm-io-rewind.c | 16 +-
drivers/md/dm-pcache/segment.c | 4 +-
drivers/nvdimm/btt.c | 2 +-
drivers/nvme/host/core.c | 12 +
drivers/nvme/host/nvme.h | 2 +
drivers/nvme/host/pci.c | 448 +++++++++++++++++++++++++++++++-
drivers/nvme/host/tcp.c | 2 +-
fs/btrfs/misc.h | 2 +-
include/linux/bio.h | 22 +-
include/linux/blk-mq.h | 7 +
include/linux/blk_types.h | 14 +-
include/linux/blkdev.h | 2 +
include/linux/bvec.h | 17 +-
include/linux/dma-buf-io.h | 91 +++++++
include/linux/fs.h | 2 +
include/linux/io_uring_types.h | 5 +
include/linux/uio.h | 11 +
include/uapi/linux/io_uring.h | 31 ++-
io_uring/io_uring.c | 3 +-
io_uring/net.c | 4 +-
io_uring/rsrc.c | 281 +++++++++++++++++---
io_uring/rsrc.h | 45 +++-
io_uring/rw.c | 6 +-
lib/iov_iter.c | 31 ++-
net/ceph/messenger.c | 4 +-
35 files changed, 1315 insertions(+), 111 deletions(-)
create mode 100644 drivers/dma-buf/dma-buf-io.c
create mode 100644 include/linux/dma-buf-io.h
--
2.54.0
