Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
On Tue, Jan 11, 2022 at 08:57:49PM +0800, Yongji Xie wrote: > On Tue, Jan 11, 2022 at 7:54 PM Michael S. Tsirkin wrote: > > > > On Tue, Jan 11, 2022 at 11:31:37AM +0800, Yongji Xie wrote: > > > On Mon, Jan 10, 2022 at 11:44 PM Michael S. Tsirkin > > > wrote: > > > > > > > > On Mon, Jan 10, 2022 at 11:24:40PM +0800, Yongji Xie wrote: > > > > > On Mon, Jan 10, 2022 at 11:10 PM Michael S. Tsirkin > > > > > wrote: > > > > > > > > > > > > On Mon, Jan 10, 2022 at 09:54:08PM +0800, Yongji Xie wrote: > > > > > > > On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin > > > > > > > wrote: > > > > > > > > > > > > > > > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > > > > > > > > > This series introduces a framework that makes it possible to > > > > > > > > > implement > > > > > > > > > software-emulated vDPA devices in userspace. And to make the > > > > > > > > > device > > > > > > > > > emulation more secure, the emulated vDPA device's control > > > > > > > > > path is handled > > > > > > > > > in the kernel and only the data path is implemented in the > > > > > > > > > userspace. > > > > > > > > > > > > > > > > > > Since the emuldated vDPA device's control path is handled in > > > > > > > > > the kernel, > > > > > > > > > a message mechnism is introduced to make userspace be aware > > > > > > > > > of the data > > > > > > > > > path related changes. Userspace can use read()/write() to > > > > > > > > > receive/reply > > > > > > > > > the control messages. > > > > > > > > > > > > > > > > > > In the data path, the core is mapping dma buffer into VDUSE > > > > > > > > > daemon's > > > > > > > > > address space, which can be implemented in different ways > > > > > > > > > depending on > > > > > > > > > the vdpa bus to which the vDPA device is attached. > > > > > > > > > > > > > > > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB > > > > > > > > > with > > > > > > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa > > > > > > > > > case, the dma > > > > > > > > > buffer is reside in a userspace memory region which can be > > > > > > > > > shared to the > > > > > > > > > VDUSE userspace processs via transferring the shmfd. > > > > > > > > > > > > > > > > > > The details and our user case is shown below: > > > > > > > > > > > > > > > > > > - > > > > > > > > > -- > > > > > > > > > |Container || QEMU(VM) | | > > > > > > > > > VDUSE daemon | > > > > > > > > > | - || --- | | > > > > > > > > > - | > > > > > > > > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | > > > > > > > > > vDPA device emulation | | block driver | | > > > > > > > > > +--- ---+ > > > > > > > > > -+--+- > > > > > > > > > | | > > > > > > > > >| | > > > > > > > > > | | > > > > > > > > >| | > > > > > > > > > +---++--+- > > > > > > > > > || block device | | vhost device | > > > > > > > > > | vduse driver | | TCP/IP || > > > > > > > > > |---+ + > > > > > > > > > ---+ -+| > > > > > > > > > | | | > > > > > > > > > | || > > > > > > > > > | --+-- --+--- > > > > > > > > > ---+---|| > > > > > > > > > | | virtio-blk driver | | vhost-vdpa driver | > > > > > > > > > | vdpa device ||| > > > > > > > > > | --+-- --+--- > > > > > > > > > ---+---|| > > > > > > > > > | | virtio bus | > > > > > > > > > | || > > > > > > > > > | ++--- | > > > > > > > > > | || > > > > > > > > > || | > > > > > > > > > | || > > > > > > > > > | --+--| > > > > > > > > > | || > > > > > > > > > | | virtio-blk device || > > > > > > > > > | || > > > > > > > > > | --+--
Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
On Tue, Jan 11, 2022 at 7:54 PM Michael S. Tsirkin wrote: > > On Tue, Jan 11, 2022 at 11:31:37AM +0800, Yongji Xie wrote: > > On Mon, Jan 10, 2022 at 11:44 PM Michael S. Tsirkin wrote: > > > > > > On Mon, Jan 10, 2022 at 11:24:40PM +0800, Yongji Xie wrote: > > > > On Mon, Jan 10, 2022 at 11:10 PM Michael S. Tsirkin > > > > wrote: > > > > > > > > > > On Mon, Jan 10, 2022 at 09:54:08PM +0800, Yongji Xie wrote: > > > > > > On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin > > > > > > wrote: > > > > > > > > > > > > > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > > > > > > > > This series introduces a framework that makes it possible to > > > > > > > > implement > > > > > > > > software-emulated vDPA devices in userspace. And to make the > > > > > > > > device > > > > > > > > emulation more secure, the emulated vDPA device's control path > > > > > > > > is handled > > > > > > > > in the kernel and only the data path is implemented in the > > > > > > > > userspace. > > > > > > > > > > > > > > > > Since the emuldated vDPA device's control path is handled in > > > > > > > > the kernel, > > > > > > > > a message mechnism is introduced to make userspace be aware of > > > > > > > > the data > > > > > > > > path related changes. Userspace can use read()/write() to > > > > > > > > receive/reply > > > > > > > > the control messages. > > > > > > > > > > > > > > > > In the data path, the core is mapping dma buffer into VDUSE > > > > > > > > daemon's > > > > > > > > address space, which can be implemented in different ways > > > > > > > > depending on > > > > > > > > the vdpa bus to which the vDPA device is attached. > > > > > > > > > > > > > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB > > > > > > > > with > > > > > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa > > > > > > > > case, the dma > > > > > > > > buffer is reside in a userspace memory region which can be > > > > > > > > shared to the > > > > > > > > VDUSE userspace processs via transferring the shmfd. > > > > > > > > > > > > > > > > The details and our user case is shown below: > > > > > > > > > > > > > > > > - > > > > > > > > -- > > > > > > > > |Container || QEMU(VM) | | > > > > > > > > VDUSE daemon | > > > > > > > > | - || --- | | > > > > > > > > - | > > > > > > > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | > > > > > > > > vDPA device emulation | | block driver | | > > > > > > > > +--- ---+ > > > > > > > > -+--+- > > > > > > > > | | > > > > > > > > | | > > > > > > > > | | > > > > > > > > | | > > > > > > > > +---++--+- > > > > > > > > || block device | | vhost device || > > > > > > > > vduse driver | | TCP/IP || > > > > > > > > |---+ + > > > > > > > > ---+ -+| > > > > > > > > | | | > > > > > > > > | || > > > > > > > > | --+-- --+--- > > > > > > > > ---+---|| > > > > > > > > | | virtio-blk driver | | vhost-vdpa driver | | > > > > > > > > vdpa device ||| > > > > > > > > | --+-- --+--- > > > > > > > > ---+---|| > > > > > > > > | | virtio bus | > > > > > > > > | || > > > > > > > > | ++--- | > > > > > > > > | || > > > > > > > > || | > > > > > > > > | || > > > > > > > > | --+--| > > > > > > > > | || > > > > > > > > | | virtio-blk device || > > > > > > > > | || > > > > > > > > | --+--| > > > > > > > > | || > > > > > > > > || | > > > > > > > > | || > > > > > > > > |
Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
On Tue, Jan 11, 2022 at 11:31:37AM +0800, Yongji Xie wrote: > On Mon, Jan 10, 2022 at 11:44 PM Michael S. Tsirkin wrote: > > > > On Mon, Jan 10, 2022 at 11:24:40PM +0800, Yongji Xie wrote: > > > On Mon, Jan 10, 2022 at 11:10 PM Michael S. Tsirkin > > > wrote: > > > > > > > > On Mon, Jan 10, 2022 at 09:54:08PM +0800, Yongji Xie wrote: > > > > > On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin > > > > > wrote: > > > > > > > > > > > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > > > > > > > This series introduces a framework that makes it possible to > > > > > > > implement > > > > > > > software-emulated vDPA devices in userspace. And to make the > > > > > > > device > > > > > > > emulation more secure, the emulated vDPA device's control path is > > > > > > > handled > > > > > > > in the kernel and only the data path is implemented in the > > > > > > > userspace. > > > > > > > > > > > > > > Since the emuldated vDPA device's control path is handled in the > > > > > > > kernel, > > > > > > > a message mechnism is introduced to make userspace be aware of > > > > > > > the data > > > > > > > path related changes. Userspace can use read()/write() to > > > > > > > receive/reply > > > > > > > the control messages. > > > > > > > > > > > > > > In the data path, the core is mapping dma buffer into VDUSE > > > > > > > daemon's > > > > > > > address space, which can be implemented in different ways > > > > > > > depending on > > > > > > > the vdpa bus to which the vDPA device is attached. > > > > > > > > > > > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB with > > > > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa > > > > > > > case, the dma > > > > > > > buffer is reside in a userspace memory region which can be shared > > > > > > > to the > > > > > > > VDUSE userspace processs via transferring the shmfd. > > > > > > > > > > > > > > The details and our user case is shown below: > > > > > > > > > > > > > > - > > > > > > > -- > > > > > > > |Container || QEMU(VM) | | > > > > > > > VDUSE daemon | > > > > > > > | - || --- | | > > > > > > > - | > > > > > > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA > > > > > > > device emulation | | block driver | | > > > > > > > +--- ---+ > > > > > > > -+--+- > > > > > > > | | > > > > > > >| | > > > > > > > | | > > > > > > >| | > > > > > > > +---++--+- > > > > > > > || block device | | vhost device || > > > > > > > vduse driver | | TCP/IP || > > > > > > > |---+ + > > > > > > > ---+ -+| > > > > > > > | | | > > > > > > > | || > > > > > > > | --+-- --+--- > > > > > > > ---+---|| > > > > > > > | | virtio-blk driver | | vhost-vdpa driver | | > > > > > > > vdpa device ||| > > > > > > > | --+-- --+--- > > > > > > > ---+---|| > > > > > > > | | virtio bus | > > > > > > > | || > > > > > > > | ++--- | > > > > > > > | || > > > > > > > || | > > > > > > > | || > > > > > > > | --+--| > > > > > > > | || > > > > > > > | | virtio-blk device || > > > > > > > | || > > > > > > > | --+--| > > > > > > > | || > > > > > > > || | > > > > > > > | || > > > > > > > | ---+--- | > > > > > > > | || > > > > > > > | | virtio-vdpa driver | | > > > > > > > | || > > >
Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
On Mon, Jan 10, 2022 at 11:44 PM Michael S. Tsirkin wrote: > > On Mon, Jan 10, 2022 at 11:24:40PM +0800, Yongji Xie wrote: > > On Mon, Jan 10, 2022 at 11:10 PM Michael S. Tsirkin wrote: > > > > > > On Mon, Jan 10, 2022 at 09:54:08PM +0800, Yongji Xie wrote: > > > > On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin > > > > wrote: > > > > > > > > > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > > > > > > This series introduces a framework that makes it possible to > > > > > > implement > > > > > > software-emulated vDPA devices in userspace. And to make the device > > > > > > emulation more secure, the emulated vDPA device's control path is > > > > > > handled > > > > > > in the kernel and only the data path is implemented in the > > > > > > userspace. > > > > > > > > > > > > Since the emuldated vDPA device's control path is handled in the > > > > > > kernel, > > > > > > a message mechnism is introduced to make userspace be aware of the > > > > > > data > > > > > > path related changes. Userspace can use read()/write() to > > > > > > receive/reply > > > > > > the control messages. > > > > > > > > > > > > In the data path, the core is mapping dma buffer into VDUSE daemon's > > > > > > address space, which can be implemented in different ways depending > > > > > > on > > > > > > the vdpa bus to which the vDPA device is attached. > > > > > > > > > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB with > > > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, > > > > > > the dma > > > > > > buffer is reside in a userspace memory region which can be shared > > > > > > to the > > > > > > VDUSE userspace processs via transferring the shmfd. > > > > > > > > > > > > The details and our user case is shown below: > > > > > > > > > > > > - > > > > > > -- > > > > > > |Container || QEMU(VM) | | > > > > > > VDUSE daemon | > > > > > > | - || --- | | > > > > > > - | > > > > > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA > > > > > > device emulation | | block driver | | > > > > > > +--- ---+ > > > > > > -+--+- > > > > > > | | > > > > > > | | > > > > > > | | > > > > > > | | > > > > > > +---++--+- > > > > > > || block device | | vhost device || > > > > > > vduse driver | | TCP/IP || > > > > > > |---+ + > > > > > > ---+ -+| > > > > > > | | | > > > > > > | || > > > > > > | --+-- --+--- > > > > > > ---+---|| > > > > > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa > > > > > > device ||| > > > > > > | --+-- --+--- > > > > > > ---+---|| > > > > > > | | virtio bus | > > > > > > | || > > > > > > | ++--- | > > > > > > | || > > > > > > || | > > > > > > | || > > > > > > | --+--| > > > > > > | || > > > > > > | | virtio-blk device || > > > > > > | || > > > > > > | --+--| > > > > > > | || > > > > > > || | > > > > > > | || > > > > > > | ---+--- | > > > > > > | || > > > > > > | | virtio-vdpa driver | | > > > > > > | || > > > > > > | ---+--- | > > > > > > | || > > > > > > || | > > > > > > |vdpa bus || > > > > > > |
Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
On Mon, Jan 10, 2022 at 11:24:40PM +0800, Yongji Xie wrote: > On Mon, Jan 10, 2022 at 11:10 PM Michael S. Tsirkin wrote: > > > > On Mon, Jan 10, 2022 at 09:54:08PM +0800, Yongji Xie wrote: > > > On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin > > > wrote: > > > > > > > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > > > > > This series introduces a framework that makes it possible to implement > > > > > software-emulated vDPA devices in userspace. And to make the device > > > > > emulation more secure, the emulated vDPA device's control path is > > > > > handled > > > > > in the kernel and only the data path is implemented in the userspace. > > > > > > > > > > Since the emuldated vDPA device's control path is handled in the > > > > > kernel, > > > > > a message mechnism is introduced to make userspace be aware of the > > > > > data > > > > > path related changes. Userspace can use read()/write() to > > > > > receive/reply > > > > > the control messages. > > > > > > > > > > In the data path, the core is mapping dma buffer into VDUSE daemon's > > > > > address space, which can be implemented in different ways depending on > > > > > the vdpa bus to which the vDPA device is attached. > > > > > > > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB with > > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, > > > > > the dma > > > > > buffer is reside in a userspace memory region which can be shared to > > > > > the > > > > > VDUSE userspace processs via transferring the shmfd. > > > > > > > > > > The details and our user case is shown below: > > > > > > > > > > - > > > > > -- > > > > > |Container || QEMU(VM) | | > > > > > VDUSE daemon | > > > > > | - || --- | | > > > > > - | > > > > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA > > > > > device emulation | | block driver | | > > > > > +--- ---+ > > > > > -+--+- > > > > > | | > > > > > | | > > > > > | | > > > > > | | > > > > > +---++--+- > > > > > || block device | | vhost device || vduse > > > > > driver | | TCP/IP || > > > > > |---+ + > > > > > ---+ -+| > > > > > | | | | > > > > > || > > > > > | --+-- --+--- > > > > > ---+---|| > > > > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa > > > > > device ||| > > > > > | --+-- --+--- > > > > > ---+---|| > > > > > | | virtio bus | | > > > > > || > > > > > | ++--- | | > > > > > || > > > > > || | | > > > > > || > > > > > | --+--| | > > > > > || > > > > > | | virtio-blk device || | > > > > > || > > > > > | --+--| | > > > > > || > > > > > || | | > > > > > || > > > > > | ---+--- | | > > > > > || > > > > > | | virtio-vdpa driver | | | > > > > > || > > > > > | ---+--- | | > > > > > || > > > > > || | | > > > > >vdpa bus || > > > > > | > > > > > ---+--+---+ > > > > >|| > > > > > | > > > > >---+--- | > > > > >
Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
On Mon, Jan 10, 2022 at 11:10 PM Michael S. Tsirkin wrote: > > On Mon, Jan 10, 2022 at 09:54:08PM +0800, Yongji Xie wrote: > > On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin wrote: > > > > > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > > > > This series introduces a framework that makes it possible to implement > > > > software-emulated vDPA devices in userspace. And to make the device > > > > emulation more secure, the emulated vDPA device's control path is > > > > handled > > > > in the kernel and only the data path is implemented in the userspace. > > > > > > > > Since the emuldated vDPA device's control path is handled in the kernel, > > > > a message mechnism is introduced to make userspace be aware of the data > > > > path related changes. Userspace can use read()/write() to receive/reply > > > > the control messages. > > > > > > > > In the data path, the core is mapping dma buffer into VDUSE daemon's > > > > address space, which can be implemented in different ways depending on > > > > the vdpa bus to which the vDPA device is attached. > > > > > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB with > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the > > > > dma > > > > buffer is reside in a userspace memory region which can be shared to the > > > > VDUSE userspace processs via transferring the shmfd. > > > > > > > > The details and our user case is shown below: > > > > > > > > - > > > > -- > > > > |Container || QEMU(VM) | | > > > > VDUSE daemon | > > > > | - || --- | | > > > > - | > > > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device > > > > emulation | | block driver | | > > > > +--- ---+ > > > > -+--+- > > > > | || > > > > | > > > > | || > > > > | > > > > +---++--+- > > > > || block device | | vhost device || vduse > > > > driver | | TCP/IP || > > > > |---+ + > > > > ---+ -+| > > > > | | | | > > > > || > > > > | --+-- --+--- > > > > ---+---|| > > > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa > > > > device ||| > > > > | --+-- --+--- > > > > ---+---|| > > > > | | virtio bus | | > > > > || > > > > | ++--- | | > > > > || > > > > || | | > > > > || > > > > | --+--| | > > > > || > > > > | | virtio-blk device || | > > > > || > > > > | --+--| | > > > > || > > > > || | | > > > > || > > > > | ---+--- | | > > > > || > > > > | | virtio-vdpa driver | | | > > > > || > > > > | ---+--- | | > > > > || > > > > || | | > > > > vdpa bus || > > > > | > > > > ---+--+---+ > > > >|| > > > > | > > > > ---+--- | > > > > -| > > > > NIC |-- > > > > > > > > ---+--- > > > >
Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
On Mon, Jan 10, 2022 at 09:54:08PM +0800, Yongji Xie wrote: > On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin wrote: > > > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > > > This series introduces a framework that makes it possible to implement > > > software-emulated vDPA devices in userspace. And to make the device > > > emulation more secure, the emulated vDPA device's control path is handled > > > in the kernel and only the data path is implemented in the userspace. > > > > > > Since the emuldated vDPA device's control path is handled in the kernel, > > > a message mechnism is introduced to make userspace be aware of the data > > > path related changes. Userspace can use read()/write() to receive/reply > > > the control messages. > > > > > > In the data path, the core is mapping dma buffer into VDUSE daemon's > > > address space, which can be implemented in different ways depending on > > > the vdpa bus to which the vDPA device is attached. > > > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB with > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the > > > dma > > > buffer is reside in a userspace memory region which can be shared to the > > > VDUSE userspace processs via transferring the shmfd. > > > > > > The details and our user case is shown below: > > > > > > - > > > -- > > > |Container || QEMU(VM) | | > > > VDUSE daemon | > > > | - || --- | | > > > - | > > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device > > > emulation | | block driver | | > > > +--- ---+ > > > -+--+- > > > | || > > > | > > > | || > > > | > > > +---++--+- > > > || block device | | vhost device || vduse > > > driver | | TCP/IP || > > > |---+ + > > > ---+ -+| > > > | | | | > > > || > > > | --+-- --+--- > > > ---+---|| > > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa > > > device ||| > > > | --+-- --+--- > > > ---+---|| > > > | | virtio bus | | > > > || > > > | ++--- | | > > > || > > > || | | > > > || > > > | --+--| | > > > || > > > | | virtio-blk device || | > > > || > > > | --+--| | > > > || > > > || | | > > > || > > > | ---+--- | | > > > || > > > | | virtio-vdpa driver | | | > > > || > > > | ---+--- | | > > > || > > > || | | > > > vdpa bus || > > > | > > > ---+--+---+ > > >|| > > > | > > >---+--- | > > > -| > > > NIC |-- > > > > > >---+--- > > > > > > | > > > > > > -+- > > > > >
Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin wrote: > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > > This series introduces a framework that makes it possible to implement > > software-emulated vDPA devices in userspace. And to make the device > > emulation more secure, the emulated vDPA device's control path is handled > > in the kernel and only the data path is implemented in the userspace. > > > > Since the emuldated vDPA device's control path is handled in the kernel, > > a message mechnism is introduced to make userspace be aware of the data > > path related changes. Userspace can use read()/write() to receive/reply > > the control messages. > > > > In the data path, the core is mapping dma buffer into VDUSE daemon's > > address space, which can be implemented in different ways depending on > > the vdpa bus to which the vDPA device is attached. > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB with > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the dma > > buffer is reside in a userspace memory region which can be shared to the > > VDUSE userspace processs via transferring the shmfd. > > > > The details and our user case is shown below: > > > > - > > -- > > |Container || QEMU(VM) | | > > VDUSE daemon | > > | - || --- | | > > - | > > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device > > emulation | | block driver | | > > +--- ---+ > > -+--+- > > | || > > | > > | || > > | > > +---++--+- > > || block device | | vhost device || vduse driver > > | | TCP/IP || > > |---+ + > > ---+ -+| > > | | | | > > || > > | --+-- --+--- > > ---+---|| > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa device > > ||| > > | --+-- --+--- > > ---+---|| > > | | virtio bus | | > > || > > | ++--- | | > > || > > || | | > > || > > | --+--| | > > || > > | | virtio-blk device || | > > || > > | --+--| | > > || > > || | | > > || > > | ---+--- | | > > || > > | | virtio-vdpa driver | | | > > || > > | ---+--- | | > > || > > || | | > > vdpa bus || > > | > > ---+--+---+ > > || > > | > > ---+--- | > > -| > > NIC |-- > > > > ---+--- > > > > | > > > >-+- > > > >| Remote Storages | > > > >--- > > > > We make use of it to implement a block device connecting to > > our distributed storage,
Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote: > This series introduces a framework that makes it possible to implement > software-emulated vDPA devices in userspace. And to make the device > emulation more secure, the emulated vDPA device's control path is handled > in the kernel and only the data path is implemented in the userspace. > > Since the emuldated vDPA device's control path is handled in the kernel, > a message mechnism is introduced to make userspace be aware of the data > path related changes. Userspace can use read()/write() to receive/reply > the control messages. > > In the data path, the core is mapping dma buffer into VDUSE daemon's > address space, which can be implemented in different ways depending on > the vdpa bus to which the vDPA device is attached. > > In virtio-vdpa case, we implements a MMU-based software IOTLB with > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the dma > buffer is reside in a userspace memory region which can be shared to the > VDUSE userspace processs via transferring the shmfd. > > The details and our user case is shown below: > > - > -- > |Container || QEMU(VM) | | > VDUSE daemon | > | - || --- | | > - | > | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device > emulation | | block driver | | > +--- ---+ > -+--+- > | || > | > | || > | > +---++--+- > || block device | | vhost device || vduse driver | > | TCP/IP || > |---+ +---+ > -+| > | | | | > || > | --+-- --+--- ---+--- > || > | | virtio-blk driver | | vhost-vdpa driver | | vdpa device | > || > | --+-- --+--- ---+--- > || > | | virtio bus | | > || > | ++--- | | > || > || | | > || > | --+--| | > || > | | virtio-blk device || | > || > | --+--| | > || > || | | > || > | ---+--- | | > || > | | virtio-vdpa driver | | | > || > | ---+--- | | > || > || | |vdpa > bus || > | > ---+--+---+ > || > | >---+--- | > -| > NIC |-- > >---+--- > > | > > -+- > > | Remote Storages | > > --- > > We make use of it to implement a block device connecting to > our distributed storage, which can be used both in containers and > VMs. Thus, we can have an unified technology stack in this two cases. > > To test it with null-blk: > > $ qemu-storage-daemon \ > --chardev
[PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
This series introduces a framework that makes it possible to implement software-emulated vDPA devices in userspace. And to make the device emulation more secure, the emulated vDPA device's control path is handled in the kernel and only the data path is implemented in the userspace. Since the emuldated vDPA device's control path is handled in the kernel, a message mechnism is introduced to make userspace be aware of the data path related changes. Userspace can use read()/write() to receive/reply the control messages. In the data path, the core is mapping dma buffer into VDUSE daemon's address space, which can be implemented in different ways depending on the vdpa bus to which the vDPA device is attached. In virtio-vdpa case, we implements a MMU-based software IOTLB with bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the dma buffer is reside in a userspace memory region which can be shared to the VDUSE userspace processs via transferring the shmfd. The details and our user case is shown below: - -- |Container || QEMU(VM) | | VDUSE daemon | | - || --- | | - | | |dev/vdx| || |/dev/vhost-vdpa-x| | | | vDPA device emulation | | block driver | | +--- ---+ -+--+- | || | | || | +---++--+- || block device | | vhost device || vduse driver | | TCP/IP || |---+ +---+ -+| | | | | || | --+-- --+--- ---+--- || | | virtio-blk driver | | vhost-vdpa driver | | vdpa device | || | --+-- --+--- ---+--- || | | virtio bus | | || | ++--- | | || || | | || | --+--| | || | | virtio-blk device || | || | --+--| | || || | | || | ---+--- | | || | | virtio-vdpa driver | | | || | ---+--- | | || || | |vdpa bus || | ---+--+---+ || | ---+--- | -| NIC |-- ---+--- | -+- | Remote Storages | --- We make use of it to implement a block device connecting to our distributed storage, which can be used both in containers and VMs. Thus, we can have an unified technology stack in this two cases. To test it with null-blk: $ qemu-storage-daemon \ --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server,nowait \ --monitor chardev=charmonitor \ --blockdev driver=host_device,cache.direct=on,aio=native,filename=/dev/nullb0,node-name=disk0 \ --export
