Re: [PATCH v7 00/12] Introduce VDUSE - vDPA Device in Userspace
On Tue, May 25, 2021 at 2:48 PM Michael S. Tsirkin wrote: > > On Tue, May 25, 2021 at 02:40:57PM +0800, Jason Wang wrote: > > > > 在 2021/5/20 下午5:06, Yongji Xie 写道: > > > On Thu, May 20, 2021 at 2:06 PM Michael S. Tsirkin > > > wrote: > > > > On Mon, May 17, 2021 at 05:55:01PM +0800, Xie Yongji wrote: > > > > > This series introduces a framework, which can be used to implement > > > > > vDPA Devices in a userspace program. The work consist of two parts: > > > > > control path forwarding and data path offloading. > > > > > > > > > > In the control path, the VDUSE driver will make use of message > > > > > mechnism to forward the config operation from vdpa bus driver > > > > > to userspace. Userspace can use read()/write() to receive/reply > > > > > those control messages. > > > > > > > > > > In the data path, the core is mapping dma buffer into VDUSE > > > > > daemon's address space, which can be implemented in different ways > > > > > depending on the vdpa bus to which the vDPA device is attached. > > > > > > > > > > In virtio-vdpa case, we implements a MMU-based on-chip IOMMU driver > > > > > with > > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, > > > > > the dma > > > > > buffer is reside in a userspace memory region which can be shared to > > > > > the > > > > > VDUSE userspace processs via transferring the shmfd. > > > > > > > > > > The details and our user case is shown below: > > > > > > > > > > - > > > > > -- > > > > > |Container || QEMU(VM) | | > > > > > VDUSE daemon | > > > > > | - || --- | | > > > > > - | > > > > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA > > > > > device emulation | | block driver | | > > > > > +--- ---+ > > > > > -+--+- > > > > > | | > > > > > | | > > > > > | | > > > > > | | > > > > > +---++--+- > > > > > || block device | | vhost device || vduse > > > > > driver | | TCP/IP || > > > > > |---+ + > > > > > ---+ -+| > > > > > | | | | > > > > > || > > > > > | --+-- --+--- > > > > > ---+---|| > > > > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa > > > > > device ||| > > > > > | --+-- --+--- > > > > > ---+---|| > > > > > | | virtio bus | | > > > > > || > > > > > | ++--- | | > > > > > || > > > > > || | | > > > > > || > > > > > | --+--| | > > > > > || > > > > > | | virtio-blk device || | > > > > > || > > > > > | --+--| | > > > > > || > > > > > || | | > > > > > || > > > > > | ---+--- | | > > > > > || > > > > > | | virtio-vdpa driver | | | > > > > > || > > > > > | ---+--- | | > > > > > || > > > > > || | | > > > > >vdpa bus || > > > > > | > > > > > ---+--+---+ > > > > >|| > > > > > | > > > > >---+--- | > > > > > -| > > > > > NIC |-- > > > > > > > > > >
Re: [PATCH v7 00/12] Introduce VDUSE - vDPA Device in Userspace
On Tue, May 25, 2021 at 02:40:57PM +0800, Jason Wang wrote: > > 在 2021/5/20 下午5:06, Yongji Xie 写道: > > On Thu, May 20, 2021 at 2:06 PM Michael S. Tsirkin wrote: > > > On Mon, May 17, 2021 at 05:55:01PM +0800, Xie Yongji wrote: > > > > This series introduces a framework, which can be used to implement > > > > vDPA Devices in a userspace program. The work consist of two parts: > > > > control path forwarding and data path offloading. > > > > > > > > In the control path, the VDUSE driver will make use of message > > > > mechnism to forward the config operation from vdpa bus driver > > > > to userspace. Userspace can use read()/write() to receive/reply > > > > those control messages. > > > > > > > > In the data path, the core is mapping dma buffer into VDUSE > > > > daemon's address space, which can be implemented in different ways > > > > depending on the vdpa bus to which the vDPA device is attached. > > > > > > > > In virtio-vdpa case, we implements a MMU-based on-chip IOMMU driver with > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the > > > > dma > > > > buffer is reside in a userspace memory region which can be shared to the > > > > VDUSE userspace processs via transferring the shmfd. > > > > > > > > The details and our user case is shown below: > > > > > > > > - > > > > -- > > > > |Container || QEMU(VM) | | > > > > VDUSE daemon | > > > > | - || --- | | > > > > - | > > > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device > > > > emulation | | block driver | | > > > > +--- ---+ > > > > -+--+- > > > > | || > > > > | > > > > | || > > > > | > > > > +---++--+- > > > > || block device | | vhost device || vduse > > > > driver | | TCP/IP || > > > > |---+ + > > > > ---+ -+| > > > > | | | | > > > > || > > > > | --+-- --+--- > > > > ---+---|| > > > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa > > > > device ||| > > > > | --+-- --+--- > > > > ---+---|| > > > > | | virtio bus | | > > > > || > > > > | ++--- | | > > > > || > > > > || | | > > > > || > > > > | --+--| | > > > > || > > > > | | virtio-blk device || | > > > > || > > > > | --+--| | > > > > || > > > > || | | > > > > || > > > > | ---+--- | | > > > > || > > > > | | virtio-vdpa driver | | | > > > > || > > > > | ---+--- | | > > > > || > > > > || | | > > > > vdpa bus || > > > > | > > > > ---+--+---+ > > > >|| > > > > | > > > > ---+--- | > > > > -| > > > > NIC |-- > > > > > > > > ---+--- > > > > > > > > | > > > > > > > >
Re: [PATCH v7 00/12] Introduce VDUSE - vDPA Device in Userspace
在 2021/5/20 下午5:06, Yongji Xie 写道: On Thu, May 20, 2021 at 2:06 PM Michael S. Tsirkin wrote: On Mon, May 17, 2021 at 05:55:01PM +0800, Xie Yongji wrote: This series introduces a framework, which can be used to implement vDPA Devices in a userspace program. The work consist of two parts: control path forwarding and data path offloading. In the control path, the VDUSE driver will make use of message mechnism to forward the config operation from vdpa bus driver to userspace. Userspace can use read()/write() to receive/reply those control messages. In the data path, the core is mapping dma buffer into VDUSE daemon's address space, which can be implemented in different ways depending on the vdpa bus to which the vDPA device is attached. In virtio-vdpa case, we implements a MMU-based on-chip IOMMU driver with bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the dma buffer is reside in a userspace memory region which can be shared to the VDUSE userspace processs via transferring the shmfd. The details and our user case is shown below: - -- |Container || QEMU(VM) | | VDUSE daemon | | - || --- | | - | | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device emulation | | block driver | | +--- ---+ -+--+- | || | | || | +---++--+- || block device | | vhost device || vduse driver | | TCP/IP || |---+ +---+ -+| | | | | || | --+-- --+--- ---+--- || | | virtio-blk driver | | vhost-vdpa driver | | vdpa device | || | --+-- --+--- ---+--- || | | virtio bus | | || | ++--- | | || || | | || | --+--| | || | | virtio-blk device || | || | --+--| | || || | | || | ---+--- | | || | | virtio-vdpa driver | | | || | ---+--- | | || || | |vdpa bus || | ---+--+---+ || | ---+--- | -| NIC |-- ---+--- | -+- | Remote Storages | --- We make use of it to implement a block device connecting to our distributed storage, which can be used both in containers and VMs. Thus, we can have an unified technology stack in this two cases. To test it with null-blk: $ qemu-storage-daemon \ --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server,nowait \ --monitor chardev=charmonitor \ --blockdev driver=host_device,cache.direct=on,aio=native,filename=/dev/nullb0,node-name=disk0 \ --export
Re: Re: [PATCH v7 00/12] Introduce VDUSE - vDPA Device in Userspace
On Thu, May 20, 2021 at 2:06 PM Michael S. Tsirkin wrote: > > On Mon, May 17, 2021 at 05:55:01PM +0800, Xie Yongji wrote: > > This series introduces a framework, which can be used to implement > > vDPA Devices in a userspace program. The work consist of two parts: > > control path forwarding and data path offloading. > > > > In the control path, the VDUSE driver will make use of message > > mechnism to forward the config operation from vdpa bus driver > > to userspace. Userspace can use read()/write() to receive/reply > > those control messages. > > > > In the data path, the core is mapping dma buffer into VDUSE > > daemon's address space, which can be implemented in different ways > > depending on the vdpa bus to which the vDPA device is attached. > > > > In virtio-vdpa case, we implements a MMU-based on-chip IOMMU driver with > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the dma > > buffer is reside in a userspace memory region which can be shared to the > > VDUSE userspace processs via transferring the shmfd. > > > > The details and our user case is shown below: > > > > - > > -- > > |Container || QEMU(VM) | | > > VDUSE daemon | > > | - || --- | | > > - | > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device > > emulation | | block driver | | > > +--- ---+ > > -+--+- > > | || > > | > > | || > > | > > +---++--+- > > || block device | | vhost device || vduse driver > > | | TCP/IP || > > |---+ + > > ---+ -+| > > | | | | > > || > > | --+-- --+--- > > ---+---|| > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa device > > ||| > > | --+-- --+--- > > ---+---|| > > | | virtio bus | | > > || > > | ++--- | | > > || > > || | | > > || > > | --+--| | > > || > > | | virtio-blk device || | > > || > > | --+--| | > > || > > || | | > > || > > | ---+--- | | > > || > > | | virtio-vdpa driver | | | > > || > > | ---+--- | | > > || > > || | | > > vdpa bus || > > | > > ---+--+---+ > > || > > | > > ---+--- | > > -| > > NIC |-- > > > > ---+--- > > > > | > > > >-+- > > > >| Remote Storages | > > > >--- > > > > We make use of it to implement a block device connecting to > > our distributed storage, which can be used both in containers and > > VMs. Thus, we can have an unified technology stack in this two cases. > > > > To
Re: [PATCH v7 00/12] Introduce VDUSE - vDPA Device in Userspace
On Mon, May 17, 2021 at 05:55:01PM +0800, Xie Yongji wrote: > This series introduces a framework, which can be used to implement > vDPA Devices in a userspace program. The work consist of two parts: > control path forwarding and data path offloading. > > In the control path, the VDUSE driver will make use of message > mechnism to forward the config operation from vdpa bus driver > to userspace. Userspace can use read()/write() to receive/reply > those control messages. > > In the data path, the core is mapping dma buffer into VDUSE > daemon's address space, which can be implemented in different ways > depending on the vdpa bus to which the vDPA device is attached. > > In virtio-vdpa case, we implements a MMU-based on-chip IOMMU driver with > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the dma > buffer is reside in a userspace memory region which can be shared to the > VDUSE userspace processs via transferring the shmfd. > > The details and our user case is shown below: > > - > -- > |Container || QEMU(VM) | | > VDUSE daemon | > | - || --- | | > - | > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device > emulation | | block driver | | > +--- ---+ > -+--+- > | || > | > | || > | > +---++--+- > || block device | | vhost device || vduse driver | > | TCP/IP || > |---+ +---+ > -+| > | | | | > || > | --+-- --+--- ---+--- > || > | | virtio-blk driver | | vhost-vdpa driver | | vdpa device | > || > | --+-- --+--- ---+--- > || > | | virtio bus | | > || > | ++--- | | > || > || | | > || > | --+--| | > || > | | virtio-blk device || | > || > | --+--| | > || > || | | > || > | ---+--- | | > || > | | virtio-vdpa driver | | | > || > | ---+--- | | > || > || | |vdpa > bus || > | > ---+--+---+ > || > | >---+--- | > -| > NIC |-- > >---+--- > > | > > -+- > > | Remote Storages | > > --- > > We make use of it to implement a block device connecting to > our distributed storage, which can be used both in containers and > VMs. Thus, we can have an unified technology stack in this two cases. > > To test it with null-blk: > > $ qemu-storage-daemon \ > --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server,nowait \ > --monitor chardev=charmonitor \ > --blockdev >
[PATCH v7 00/12] Introduce VDUSE - vDPA Device in Userspace
This series introduces a framework, which can be used to implement vDPA Devices in a userspace program. The work consist of two parts: control path forwarding and data path offloading. In the control path, the VDUSE driver will make use of message mechnism to forward the config operation from vdpa bus driver to userspace. Userspace can use read()/write() to receive/reply those control messages. In the data path, the core is mapping dma buffer into VDUSE daemon's address space, which can be implemented in different ways depending on the vdpa bus to which the vDPA device is attached. In virtio-vdpa case, we implements a MMU-based on-chip IOMMU driver with bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the dma buffer is reside in a userspace memory region which can be shared to the VDUSE userspace processs via transferring the shmfd. The details and our user case is shown below: - -- |Container || QEMU(VM) | | VDUSE daemon | | - || --- | | - | | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device emulation | | block driver | | +--- ---+ -+--+- | || | | || | +---++--+- || block device | | vhost device || vduse driver | | TCP/IP || |---+ +---+ -+| | | | | || | --+-- --+--- ---+--- || | | virtio-blk driver | | vhost-vdpa driver | | vdpa device | || | --+-- --+--- ---+--- || | | virtio bus | | || | ++--- | | || || | | || | --+--| | || | | virtio-blk device || | || | --+--| | || || | | || | ---+--- | | || | | virtio-vdpa driver | | | || | ---+--- | | || || | |vdpa bus || | ---+--+---+ || | ---+--- | -| NIC |-- ---+--- | -+- | Remote Storages | --- We make use of it to implement a block device connecting to our distributed storage, which can be used both in containers and VMs. Thus, we can have an unified technology stack in this two cases. To test it with null-blk: $ qemu-storage-daemon \ --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server,nowait \ --monitor chardev=charmonitor \ --blockdev driver=host_device,cache.direct=on,aio=native,filename=/dev/nullb0,node-name=disk0 \ --export type=vduse-blk,id=test,node-name=disk0,writable=on,name=vduse-null,num-queues=16,queue-size=128 The qemu-storage-daemon can be found at https://github.com/bytedance/qemu/tree/vduse To make the
