Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Fri, Apr 20, 2018 at 11:52:47AM +0800, Jason Wang wrote: > > The biggest issue is that you let userspace poke at the > > device which is also allowed by the IOMMU to poke at > > kernel memory (needed for kernel driver to work). > > I don't quite get. The userspace driver could be built on top of VFIO for > sure. So kernel memory were perfectly isolated in this case. VFIO does what it can but it mostly just has the IOMMU to play with. So don't overestimate what it can do - it assumes a high level of spec compliance for protections to work. For example, ATS is enabled by default if device has it, and that treats translated requests are trusted. FLS is assumed to reset the device for when VFIO is unbound from the device. etc. > > > > Yes, maybe if device is not buggy it's all fine, but > > it's better if we do not have to trust the device > > otherwise the security picture becomes more murky. > > > > I suggested attaching a PASID to (some) queues - see my old post "using > > PASIDs to enable a safe variant of direct ring access". > > > > Then using IOMMU with VFIO to limit access through queue to corrent > > ranges of memory. > > Well userspace driver could benefit from this too. And we can even go > further by using nested IO page tables to share IOVA address space between > devices and a VM. > > Thanks Yes I suggested this separately. -- MST
Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Fri, Apr 20, 2018 at 03:50:41AM +, Liang, Cunming wrote: > > > > -Original Message- > > From: Bie, Tiwei > > Sent: Friday, April 20, 2018 11:28 AM > > To: Michael S. Tsirkin <m...@redhat.com> > > Cc: Jason Wang <jasow...@redhat.com>; alex.william...@redhat.com; > > ddut...@redhat.com; Duyck, Alexander H <alexander.h.du...@intel.com>; > > virtio-...@lists.oasis-open.org; linux-ker...@vger.kernel.org; > > k...@vger.kernel.org; virtualizat...@lists.linux-foundation.org; > > netdev@vger.kernel.org; Daly, Dan <dan.d...@intel.com>; Liang, Cunming > > <cunming.li...@intel.com>; Wang, Zhihong <zhihong.w...@intel.com>; Tan, > > Jianfeng <jianfeng....@intel.com>; Wang, Xiao W <xiao.w.w...@intel.com>; > > Tian, Kevin <kevin.t...@intel.com> > > Subject: Re: [RFC] vhost: introduce mdev based hardware vhost backend > > > > On Thu, Apr 19, 2018 at 09:40:23PM +0300, Michael S. Tsirkin wrote: > > > On Tue, Apr 10, 2018 at 03:25:45PM +0800, Jason Wang wrote: > > > > > > > One problem is that, different virtio ring compatible devices > > > > > > > may have different device interfaces. That is to say, we will > > > > > > > need different drivers in QEMU. It could be troublesome. And > > > > > > > that's what this patch trying to fix. The idea behind this > > > > > > > patch is very simple: mdev is a standard way to emulate device > > > > > > > in kernel. > > > > > > So you just move the abstraction layer from qemu to kernel, and > > > > > > you still need different drivers in kernel for different device > > > > > > interfaces of accelerators. This looks even more complex than > > > > > > leaving it in qemu. As you said, another idea is to implement > > > > > > userspace vhost backend for accelerators which seems easier and > > > > > > could co-work with other parts of qemu without inventing new type of > > messages. > > > > > I'm not quite sure. Do you think it's acceptable to add various > > > > > vendor specific hardware drivers in QEMU? > > > > > > > > > > > > > I don't object but we need to figure out the advantages of doing it > > > > in qemu too. > > > > > > > > Thanks > > > > > > To be frank kernel is exactly where device drivers belong. DPDK did > > > move them to userspace but that's merely a requirement for data path. > > > *If* you can have them in kernel that is best: > > > - update kernel and there's no need to rebuild userspace > > > - apps can be written in any language no need to maintain multiple > > > libraries or add wrappers > > > - security concerns are much smaller (ok people are trying to > > > raise the bar with IOMMUs and such, but it's already pretty > > > good even without) > > > > > > The biggest issue is that you let userspace poke at the device which > > > is also allowed by the IOMMU to poke at kernel memory (needed for > > > kernel driver to work). > > > > I think the device won't and shouldn't be allowed to poke at kernel memory. > > Its > > kernel driver needs some kernel memory to work. But the device doesn't have > > the access to them. Instead, the device only has the access to: > > > > (1) the entire memory of the VM (if vIOMMU isn't used) or > > (2) the memory belongs to the guest virtio device (if > > vIOMMU is being used). > > > > Below is the reason: > > > > For the first case, we should program the IOMMU for the hardware device > > based > > on the info in the memory table which is the entire memory of the VM. > > > > For the second case, we should program the IOMMU for the hardware device > > based on the info in the shadow page table of the vIOMMU. > > > > So the memory can be accessed by the device is limited, it should be safe > > especially for the second case. > > > > My concern is that, in this RFC, we don't program the IOMMU for the mdev > > device in the userspace via the VFIO API directly. Instead, we pass the > > memory > > table to the kernel driver via the mdev device (BAR0) and ask the driver to > > do the > > IOMMU programming. Someone may don't like it. The main reason why we don't > > program IOMMU via VFIO API in userspace directly is that, currently IOMMU > > drivers don't support mdev bus. > > > > > > > >
Re: [RFC] vhost: introduce mdev based hardware vhost backend
On 2018年04月20日 02:40, Michael S. Tsirkin wrote: On Tue, Apr 10, 2018 at 03:25:45PM +0800, Jason Wang wrote: One problem is that, different virtio ring compatible devices may have different device interfaces. That is to say, we will need different drivers in QEMU. It could be troublesome. And that's what this patch trying to fix. The idea behind this patch is very simple: mdev is a standard way to emulate device in kernel. So you just move the abstraction layer from qemu to kernel, and you still need different drivers in kernel for different device interfaces of accelerators. This looks even more complex than leaving it in qemu. As you said, another idea is to implement userspace vhost backend for accelerators which seems easier and could co-work with other parts of qemu without inventing new type of messages. I'm not quite sure. Do you think it's acceptable to add various vendor specific hardware drivers in QEMU? I don't object but we need to figure out the advantages of doing it in qemu too. Thanks To be frank kernel is exactly where device drivers belong. DPDK did move them to userspace but that's merely a requirement for data path. *If* you can have them in kernel that is best: - update kernel and there's no need to rebuild userspace Well, you still need to rebuild userspace since a new vhost backend is required which relies vhost protocol through mdev API. And I believe upgrading userspace package is considered to be more lightweight than upgrading kernel. With mdev, we're likely to repeat the story of vhost API, dealing with features/versions and inventing new API endless for new features. And you will still need to rebuild the userspace. - apps can be written in any language no need to maintain multiple libraries or add wrappers This is not a big issue consider It's not a generic network driver but a mdev driver, the only possible user is VM. - security concerns are much smaller (ok people are trying to raise the bar with IOMMUs and such, but it's already pretty good even without) Well, I think not, kernel bugs are much more serious than userspace ones. And I beg the kernel driver itself won't be small. The biggest issue is that you let userspace poke at the device which is also allowed by the IOMMU to poke at kernel memory (needed for kernel driver to work). I don't quite get. The userspace driver could be built on top of VFIO for sure. So kernel memory were perfectly isolated in this case. Yes, maybe if device is not buggy it's all fine, but it's better if we do not have to trust the device otherwise the security picture becomes more murky. I suggested attaching a PASID to (some) queues - see my old post "using PASIDs to enable a safe variant of direct ring access". Then using IOMMU with VFIO to limit access through queue to corrent ranges of memory. Well userspace driver could benefit from this too. And we can even go further by using nested IO page tables to share IOVA address space between devices and a VM. Thanks
RE: [RFC] vhost: introduce mdev based hardware vhost backend
> -Original Message- > From: Bie, Tiwei > Sent: Friday, April 20, 2018 11:28 AM > To: Michael S. Tsirkin <m...@redhat.com> > Cc: Jason Wang <jasow...@redhat.com>; alex.william...@redhat.com; > ddut...@redhat.com; Duyck, Alexander H <alexander.h.du...@intel.com>; > virtio-...@lists.oasis-open.org; linux-ker...@vger.kernel.org; > k...@vger.kernel.org; virtualizat...@lists.linux-foundation.org; > netdev@vger.kernel.org; Daly, Dan <dan.d...@intel.com>; Liang, Cunming > <cunming.li...@intel.com>; Wang, Zhihong <zhihong.w...@intel.com>; Tan, > Jianfeng <jianfeng@intel.com>; Wang, Xiao W <xiao.w.w...@intel.com>; > Tian, Kevin <kevin.t...@intel.com> > Subject: Re: [RFC] vhost: introduce mdev based hardware vhost backend > > On Thu, Apr 19, 2018 at 09:40:23PM +0300, Michael S. Tsirkin wrote: > > On Tue, Apr 10, 2018 at 03:25:45PM +0800, Jason Wang wrote: > > > > > > One problem is that, different virtio ring compatible devices > > > > > > may have different device interfaces. That is to say, we will > > > > > > need different drivers in QEMU. It could be troublesome. And > > > > > > that's what this patch trying to fix. The idea behind this > > > > > > patch is very simple: mdev is a standard way to emulate device > > > > > > in kernel. > > > > > So you just move the abstraction layer from qemu to kernel, and > > > > > you still need different drivers in kernel for different device > > > > > interfaces of accelerators. This looks even more complex than > > > > > leaving it in qemu. As you said, another idea is to implement > > > > > userspace vhost backend for accelerators which seems easier and > > > > > could co-work with other parts of qemu without inventing new type of > messages. > > > > I'm not quite sure. Do you think it's acceptable to add various > > > > vendor specific hardware drivers in QEMU? > > > > > > > > > > I don't object but we need to figure out the advantages of doing it > > > in qemu too. > > > > > > Thanks > > > > To be frank kernel is exactly where device drivers belong. DPDK did > > move them to userspace but that's merely a requirement for data path. > > *If* you can have them in kernel that is best: > > - update kernel and there's no need to rebuild userspace > > - apps can be written in any language no need to maintain multiple > > libraries or add wrappers > > - security concerns are much smaller (ok people are trying to > > raise the bar with IOMMUs and such, but it's already pretty > > good even without) > > > > The biggest issue is that you let userspace poke at the device which > > is also allowed by the IOMMU to poke at kernel memory (needed for > > kernel driver to work). > > I think the device won't and shouldn't be allowed to poke at kernel memory. > Its > kernel driver needs some kernel memory to work. But the device doesn't have > the access to them. Instead, the device only has the access to: > > (1) the entire memory of the VM (if vIOMMU isn't used) or > (2) the memory belongs to the guest virtio device (if > vIOMMU is being used). > > Below is the reason: > > For the first case, we should program the IOMMU for the hardware device based > on the info in the memory table which is the entire memory of the VM. > > For the second case, we should program the IOMMU for the hardware device > based on the info in the shadow page table of the vIOMMU. > > So the memory can be accessed by the device is limited, it should be safe > especially for the second case. > > My concern is that, in this RFC, we don't program the IOMMU for the mdev > device in the userspace via the VFIO API directly. Instead, we pass the memory > table to the kernel driver via the mdev device (BAR0) and ask the driver to > do the > IOMMU programming. Someone may don't like it. The main reason why we don't > program IOMMU via VFIO API in userspace directly is that, currently IOMMU > drivers don't support mdev bus. > > > > > Yes, maybe if device is not buggy it's all fine, but it's better if we > > do not have to trust the device otherwise the security picture becomes > > more murky. > > > > I suggested attaching a PASID to (some) queues - see my old post > > "using PASIDs to enable a safe variant of direct ring access". > Ideally we can have a device binding with normal driver in host, meanwhile support to allocate a few queues attaching with PASID on-demand. By vhost mdev transport channel, the data path ability of queues(as a device) can expose to qemu vhost adaptor as a vDPA instance. Then we can avoid VF number limitation, providing vhost data path acceleration in a small granularity. > It's pretty cool. We also have some similar ideas. > Cunming will talk more about this. > > Best regards, > Tiwei Bie > > > > > Then using IOMMU with VFIO to limit access through queue to corrent > > ranges of memory. > > > > > > -- > > MST
Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Fri, Apr 20, 2018 at 11:28:07AM +0800, Tiwei Bie wrote: > On Thu, Apr 19, 2018 at 09:40:23PM +0300, Michael S. Tsirkin wrote: > > On Tue, Apr 10, 2018 at 03:25:45PM +0800, Jason Wang wrote: > > > > > > One problem is that, different virtio ring compatible devices > > > > > > may have different device interfaces. That is to say, we will > > > > > > need different drivers in QEMU. It could be troublesome. And > > > > > > that's what this patch trying to fix. The idea behind this > > > > > > patch is very simple: mdev is a standard way to emulate device > > > > > > in kernel. > > > > > So you just move the abstraction layer from qemu to kernel, and you > > > > > still > > > > > need different drivers in kernel for different device interfaces of > > > > > accelerators. This looks even more complex than leaving it in qemu. > > > > > As you > > > > > said, another idea is to implement userspace vhost backend for > > > > > accelerators > > > > > which seems easier and could co-work with other parts of qemu without > > > > > inventing new type of messages. > > > > I'm not quite sure. Do you think it's acceptable to > > > > add various vendor specific hardware drivers in QEMU? > > > > > > > > > > I don't object but we need to figure out the advantages of doing it in > > > qemu > > > too. > > > > > > Thanks > > > > To be frank kernel is exactly where device drivers belong. DPDK did > > move them to userspace but that's merely a requirement for data path. > > *If* you can have them in kernel that is best: > > - update kernel and there's no need to rebuild userspace > > - apps can be written in any language no need to maintain multiple > > libraries or add wrappers > > - security concerns are much smaller (ok people are trying to > > raise the bar with IOMMUs and such, but it's already pretty > > good even without) > > > > The biggest issue is that you let userspace poke at the > > device which is also allowed by the IOMMU to poke at > > kernel memory (needed for kernel driver to work). > > I think the device won't and shouldn't be allowed to > poke at kernel memory. Its kernel driver needs some > kernel memory to work. But the device doesn't have > the access to them. Instead, the device only has the > access to: > > (1) the entire memory of the VM (if vIOMMU isn't used) > or > (2) the memory belongs to the guest virtio device (if > vIOMMU is being used). > > Below is the reason: > > For the first case, we should program the IOMMU for > the hardware device based on the info in the memory > table which is the entire memory of the VM. > > For the second case, we should program the IOMMU for > the hardware device based on the info in the shadow > page table of the vIOMMU. > > So the memory can be accessed by the device is limited, > it should be safe especially for the second case. > > My concern is that, in this RFC, we don't program the > IOMMU for the mdev device in the userspace via the VFIO > API directly. Instead, we pass the memory table to the > kernel driver via the mdev device (BAR0) and ask the > driver to do the IOMMU programming. Someone may don't > like it. The main reason why we don't program IOMMU via > VFIO API in userspace directly is that, currently IOMMU > drivers don't support mdev bus. But it is a pci device after all, isn't it? IOMMU drivers certainly support that ... Another issue with this approach is that internal kernel issues leak out to the interface. > > > > Yes, maybe if device is not buggy it's all fine, but > > it's better if we do not have to trust the device > > otherwise the security picture becomes more murky. > > > > I suggested attaching a PASID to (some) queues - see my old post "using > > PASIDs to enable a safe variant of direct ring access". > > It's pretty cool. We also have some similar ideas. > Cunming will talk more about this. > > Best regards, > Tiwei Bie An extra benefit to this could be that requests with PASID undergo an extra level of translation. We could use it to avoid the need for shadowing on intel. Something like this: - expose to guest a standard virtio device (no pasid support) - back it by virtio device with pasid support on the host by attaching same pasid to all queues now - guest will build 1 level of page tables we build first level page tables for requests with pasid and point the IOMMU to use the guest supplied page tables for the second level of translation. Now we do need to forward invalidations but we no longer need to set the CM bit and shadow valid entries. > > > > Then using IOMMU with VFIO to limit access through queue to corrent > > ranges of memory. > > > > > > -- > > MST
Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Thu, Apr 19, 2018 at 09:40:23PM +0300, Michael S. Tsirkin wrote: > On Tue, Apr 10, 2018 at 03:25:45PM +0800, Jason Wang wrote: > > > > > One problem is that, different virtio ring compatible devices > > > > > may have different device interfaces. That is to say, we will > > > > > need different drivers in QEMU. It could be troublesome. And > > > > > that's what this patch trying to fix. The idea behind this > > > > > patch is very simple: mdev is a standard way to emulate device > > > > > in kernel. > > > > So you just move the abstraction layer from qemu to kernel, and you > > > > still > > > > need different drivers in kernel for different device interfaces of > > > > accelerators. This looks even more complex than leaving it in qemu. As > > > > you > > > > said, another idea is to implement userspace vhost backend for > > > > accelerators > > > > which seems easier and could co-work with other parts of qemu without > > > > inventing new type of messages. > > > I'm not quite sure. Do you think it's acceptable to > > > add various vendor specific hardware drivers in QEMU? > > > > > > > I don't object but we need to figure out the advantages of doing it in qemu > > too. > > > > Thanks > > To be frank kernel is exactly where device drivers belong. DPDK did > move them to userspace but that's merely a requirement for data path. > *If* you can have them in kernel that is best: > - update kernel and there's no need to rebuild userspace > - apps can be written in any language no need to maintain multiple > libraries or add wrappers > - security concerns are much smaller (ok people are trying to > raise the bar with IOMMUs and such, but it's already pretty > good even without) > > The biggest issue is that you let userspace poke at the > device which is also allowed by the IOMMU to poke at > kernel memory (needed for kernel driver to work). I think the device won't and shouldn't be allowed to poke at kernel memory. Its kernel driver needs some kernel memory to work. But the device doesn't have the access to them. Instead, the device only has the access to: (1) the entire memory of the VM (if vIOMMU isn't used) or (2) the memory belongs to the guest virtio device (if vIOMMU is being used). Below is the reason: For the first case, we should program the IOMMU for the hardware device based on the info in the memory table which is the entire memory of the VM. For the second case, we should program the IOMMU for the hardware device based on the info in the shadow page table of the vIOMMU. So the memory can be accessed by the device is limited, it should be safe especially for the second case. My concern is that, in this RFC, we don't program the IOMMU for the mdev device in the userspace via the VFIO API directly. Instead, we pass the memory table to the kernel driver via the mdev device (BAR0) and ask the driver to do the IOMMU programming. Someone may don't like it. The main reason why we don't program IOMMU via VFIO API in userspace directly is that, currently IOMMU drivers don't support mdev bus. > > Yes, maybe if device is not buggy it's all fine, but > it's better if we do not have to trust the device > otherwise the security picture becomes more murky. > > I suggested attaching a PASID to (some) queues - see my old post "using > PASIDs to enable a safe variant of direct ring access". It's pretty cool. We also have some similar ideas. Cunming will talk more about this. Best regards, Tiwei Bie > > Then using IOMMU with VFIO to limit access through queue to corrent > ranges of memory. > > > -- > MST
Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Tue, Apr 10, 2018 at 03:25:45PM +0800, Jason Wang wrote: > > > > One problem is that, different virtio ring compatible devices > > > > may have different device interfaces. That is to say, we will > > > > need different drivers in QEMU. It could be troublesome. And > > > > that's what this patch trying to fix. The idea behind this > > > > patch is very simple: mdev is a standard way to emulate device > > > > in kernel. > > > So you just move the abstraction layer from qemu to kernel, and you still > > > need different drivers in kernel for different device interfaces of > > > accelerators. This looks even more complex than leaving it in qemu. As you > > > said, another idea is to implement userspace vhost backend for > > > accelerators > > > which seems easier and could co-work with other parts of qemu without > > > inventing new type of messages. > > I'm not quite sure. Do you think it's acceptable to > > add various vendor specific hardware drivers in QEMU? > > > > I don't object but we need to figure out the advantages of doing it in qemu > too. > > Thanks To be frank kernel is exactly where device drivers belong. DPDK did move them to userspace but that's merely a requirement for data path. *If* you can have them in kernel that is best: - update kernel and there's no need to rebuild userspace - apps can be written in any language no need to maintain multiple libraries or add wrappers - security concerns are much smaller (ok people are trying to raise the bar with IOMMUs and such, but it's already pretty good even without) The biggest issue is that you let userspace poke at the device which is also allowed by the IOMMU to poke at kernel memory (needed for kernel driver to work). Yes, maybe if device is not buggy it's all fine, but it's better if we do not have to trust the device otherwise the security picture becomes more murky. I suggested attaching a PASID to (some) queues - see my old post "using PASIDs to enable a safe variant of direct ring access". Then using IOMMU with VFIO to limit access through queue to corrent ranges of memory. -- MST
Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On 2018年04月10日 22:23, Liang, Cunming wrote: -Original Message- From: Michael S. Tsirkin [mailto:m...@redhat.com] Sent: Tuesday, April 10, 2018 9:36 PM To: Liang, Cunming<cunming.li...@intel.com> Cc: Paolo Bonzini<pbonz...@redhat.com>; Bie, Tiwei<tiwei@intel.com>; Jason Wang<jasow...@redhat.com>;alex.william...@redhat.com; ddut...@redhat.com; Duyck, Alexander H<alexander.h.du...@intel.com>; virtio-...@lists.oasis-open.org;linux-ker...@vger.kernel.org; k...@vger.kernel.org;virtualizat...@lists.linux-foundation.org; netdev@vger.kernel.org; Daly, Dan<dan.d...@intel.com>; Wang, Zhihong <zhihong.w...@intel.com>; Tan, Jianfeng<jianfeng@intel.com>; Wang, Xiao W<xiao.w.w...@intel.com> Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend On Tue, Apr 10, 2018 at 09:23:53AM +, Liang, Cunming wrote: -Original Message- From: Paolo Bonzini [mailto:pbonz...@redhat.com] Sent: Tuesday, April 10, 2018 3:52 PM To: Bie, Tiwei<tiwei@intel.com>; Jason Wang <jasow...@redhat.com> Cc:m...@redhat.com;alex.william...@redhat.com;ddut...@redhat.com; Duyck, Alexander H<alexander.h.du...@intel.com>; virtio-dev@lists.oasis- open.org;linux-ker...@vger.kernel.org; k...@vger.kernel.org;virtualizat...@lists.linux-foundation.org; netdev@vger.kernel.org; Daly, Dan<dan.d...@intel.com>; Liang, Cunming<cunming.li...@intel.com>; Wang, Zhihong <zhihong.w...@intel.com>; Tan, Jianfeng<jianfeng....@intel.com>; Wang, Xiao W<xiao.w.w...@intel.com> Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend On 10/04/2018 06:57, Tiwei Bie wrote: So you just move the abstraction layer from qemu to kernel, and you still need different drivers in kernel for different device interfaces of accelerators. This looks even more complex than leaving it in qemu. As you said, another idea is to implement userspace vhost backend for accelerators which seems easier and could co-work with other parts of qemu without inventing new type of messages. I'm not quite sure. Do you think it's acceptable to add various vendor specific hardware drivers in QEMU? I think so. We have vendor-specific quirks, and at some point there was an idea of using quirks to implement (vendor-specific) live migration support for assigned devices. Vendor-specific quirks of accessing VGA is a small portion. Other major portions are still handled by guest driver. While in this case, when saying various vendor specific drivers in QEMU, it says QEMU takes over and provides the entire user space device drivers. Some parts are even not relevant to vhost, they're basic device function enabling. Moreover, it could be different kinds of devices(network/block/...) under vhost. No matter # of vendors or # of types, total LOC is not small. The idea is to avoid introducing these extra complexity out of QEMU, keeping vhost adapter simple. As vhost protocol is de factor standard, it leverages kernel device driver to provide the diversity. Changing once in QEMU, then it supports multi-vendor devices whose drivers naturally providing kernel driver there. If QEMU is going to build a user space driver framework there, we're open mind on that, even leveraging DPDK as the underlay library. Looking forward to more others' comments from community. Steve Dependency on a kernel driver is fine IMHO. It's the dependency on a DPDK backend that makes people unhappy, since the functionality in question is setup- time only. Agreed, we don't see dependency on kernel driver is a problem. At engineering level, kernel driver is harder than userspace driver. mdev based vhost backend (this patch set) is independent with vhost-user extension patch set. In fact, there're a few vhost-user providers, DPDK librte_vhost is one of them. FD.IO/VPP and snabbswitch have their own vhost-user providers. So I can't agree on vhost-user extension patch depends on DPDK backend. But anyway, that's the topic of another mail thread. Well we can treat mdev as another kind of transport of vhost-user. And technically we can even implement a relay mdev than forward vhost-user messages to dpdk. Thanks
Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On 2018年04月10日 17:23, Liang, Cunming wrote: -Original Message- From: Paolo Bonzini [mailto:pbonz...@redhat.com] Sent: Tuesday, April 10, 2018 3:52 PM To: Bie, Tiwei <tiwei@intel.com>; Jason Wang <jasow...@redhat.com> Cc: m...@redhat.com; alex.william...@redhat.com; ddut...@redhat.com; Duyck, Alexander H <alexander.h.du...@intel.com>; virtio-dev@lists.oasis- open.org; linux-ker...@vger.kernel.org; k...@vger.kernel.org; virtualizat...@lists.linux-foundation.org; netdev@vger.kernel.org; Daly, Dan <dan.d...@intel.com>; Liang, Cunming <cunming.li...@intel.com>; Wang, Zhihong <zhihong.w...@intel.com>; Tan, Jianfeng <jianfeng@intel.com>; Wang, Xiao W <xiao.w.w...@intel.com> Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend On 10/04/2018 06:57, Tiwei Bie wrote: So you just move the abstraction layer from qemu to kernel, and you still need different drivers in kernel for different device interfaces of accelerators. This looks even more complex than leaving it in qemu. As you said, another idea is to implement userspace vhost backend for accelerators which seems easier and could co-work with other parts of qemu without inventing new type of messages. I'm not quite sure. Do you think it's acceptable to add various vendor specific hardware drivers in QEMU? I think so. We have vendor-specific quirks, and at some point there was an idea of using quirks to implement (vendor-specific) live migration support for assigned devices. Vendor-specific quirks of accessing VGA is a small portion. Other major portions are still handled by guest driver. While in this case, when saying various vendor specific drivers in QEMU, it says QEMU takes over and provides the entire user space device drivers. Some parts are even not relevant to vhost, they're basic device function enabling. Moreover, it could be different kinds of devices(network/block/...) under vhost. No matter # of vendors or # of types, total LOC is not small. The idea is to avoid introducing these extra complexity out of QEMU, keeping vhost adapter simple. As vhost protocol is de factor standard, it leverages kernel device driver to provide the diversity. Changing once in QEMU, then it supports multi-vendor devices whose drivers naturally providing kernel driver there. Let me clarify my question, it's not qemu vs kenrel but userspace vs kernel. It could be a library which could be linked to qemu. Doing it in userspace have the following obvious advantages: - attack surface was limited to userspace - easier to be maintained (compared to kernel driver) - easier to be extended without introducing new userspace/kernel interfaces - not tied to a specific operating system If we want to do it in the kernel, need to consider to unify code between mdev device driver and generic driver. For net driver, maybe we can even consider to do it on top of exist drivers. If QEMU is going to build a user space driver framework there, we're open mind on that, even leveraging DPDK as the underlay library. Looking forward to more others' comments from community. I'm doing this now by implementing vhost inside qemu IOThreads. Hope I can post RFC in few months. Thanks Steve Paolo
Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Tue, Apr 10, 2018 at 09:23:53AM +, Liang, Cunming wrote: > If QEMU is going to build a user space driver framework there, we're open > mind on that, even leveraging DPDK as the underlay library. Looking forward > to more others' comments from community. There is already an NVMe VFIO driver in QEMU (see block/nvme.c). So in principle there's no reason against userspace drivers in QEMU. Stefan signature.asc Description: PGP signature
RE: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
> From: Liang, Cunming > Sent: Tuesday, April 10, 2018 10:24 PM > [...] > > > > As others said, we do not need to go overeboard. A couple of small > vendor- > > specific quirks in qemu isn't a big deal. > > It's quite challenge to identify it's small or not, there's no uniform metric. > > It's only dependent on QEMU itself, that's the obvious benefit. Tradeoff is > to build the entire device driver. We don't object to do that in QEMU, but > wanna make sure to understand the boundary size clearly. > It might be helpful if you can post some sample code using proposed framework - then people have a clear feeling about what size is talked about here and whether it falls into the concept of 'small quirks'. Thanks Kevin
RE: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
> -Original Message- > From: Michael S. Tsirkin [mailto:m...@redhat.com] > Sent: Tuesday, April 10, 2018 9:36 PM > To: Liang, Cunming <cunming.li...@intel.com> > Cc: Paolo Bonzini <pbonz...@redhat.com>; Bie, Tiwei <tiwei@intel.com>; > Jason Wang <jasow...@redhat.com>; alex.william...@redhat.com; > ddut...@redhat.com; Duyck, Alexander H <alexander.h.du...@intel.com>; > virtio-...@lists.oasis-open.org; linux-ker...@vger.kernel.org; > k...@vger.kernel.org; virtualizat...@lists.linux-foundation.org; > netdev@vger.kernel.org; Daly, Dan <dan.d...@intel.com>; Wang, Zhihong > <zhihong.w...@intel.com>; Tan, Jianfeng <jianfeng@intel.com>; Wang, Xiao > W <xiao.w.w...@intel.com> > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost > backend > > On Tue, Apr 10, 2018 at 09:23:53AM +, Liang, Cunming wrote: > > > > > > > -Original Message- > > > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > > > Sent: Tuesday, April 10, 2018 3:52 PM > > > To: Bie, Tiwei <tiwei@intel.com>; Jason Wang > > > <jasow...@redhat.com> > > > Cc: m...@redhat.com; alex.william...@redhat.com; ddut...@redhat.com; > > > Duyck, Alexander H <alexander.h.du...@intel.com>; > > > virtio-dev@lists.oasis- open.org; linux-ker...@vger.kernel.org; > > > k...@vger.kernel.org; virtualizat...@lists.linux-foundation.org; > > > netdev@vger.kernel.org; Daly, Dan <dan.d...@intel.com>; Liang, > > > Cunming <cunming.li...@intel.com>; Wang, Zhihong > > > <zhihong.w...@intel.com>; Tan, Jianfeng <jianfeng@intel.com>; > > > Wang, Xiao W <xiao.w.w...@intel.com> > > > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based > > > hardware vhost backend > > > > > > On 10/04/2018 06:57, Tiwei Bie wrote: > > > >> So you just move the abstraction layer from qemu to kernel, and > > > >> you still need different drivers in kernel for different device > > > >> interfaces of accelerators. This looks even more complex than > > > >> leaving it in qemu. As you said, another idea is to implement > > > >> userspace vhost backend for accelerators which seems easier and > > > >> could co-work with other parts of qemu without inventing new type of > messages. > > > > > > > > I'm not quite sure. Do you think it's acceptable to add various > > > > vendor specific hardware drivers in QEMU? > > > > > > I think so. We have vendor-specific quirks, and at some point there > > > was an idea of using quirks to implement (vendor-specific) live > > > migration support for assigned devices. > > > > Vendor-specific quirks of accessing VGA is a small portion. Other major > > portions > are still handled by guest driver. > > > > While in this case, when saying various vendor specific drivers in QEMU, it > > says > QEMU takes over and provides the entire user space device drivers. Some parts > are even not relevant to vhost, they're basic device function enabling. > Moreover, > it could be different kinds of devices(network/block/...) under vhost. No > matter > # of vendors or # of types, total LOC is not small. > > > > The idea is to avoid introducing these extra complexity out of QEMU, keeping > vhost adapter simple. As vhost protocol is de factor standard, it leverages > kernel > device driver to provide the diversity. Changing once in QEMU, then it > supports > multi-vendor devices whose drivers naturally providing kernel driver there. > > > > If QEMU is going to build a user space driver framework there, we're open > > mind > on that, even leveraging DPDK as the underlay library. Looking forward to more > others' comments from community. > > > > Steve > > Dependency on a kernel driver is fine IMHO. It's the dependency on a DPDK > backend that makes people unhappy, since the functionality in question is > setup- > time only. Agreed, we don't see dependency on kernel driver is a problem. mdev based vhost backend (this patch set) is independent with vhost-user extension patch set. In fact, there're a few vhost-user providers, DPDK librte_vhost is one of them. FD.IO/VPP and snabbswitch have their own vhost-user providers. So I can't agree on vhost-user extension patch depends on DPDK backend. But anyway, that's the topic of another mail thread. > > As others said, we do not need to go overeboard. A couple of small vendor- > specific quirks in qemu isn't a big deal. It's quite challenge to identify it's small or not, there's no uniform metric. It's only dependent on QEMU itself, that's the obvious benefit. Tradeoff is to build the entire device driver. We don't object to do that in QEMU, but wanna make sure to understand the boundary size clearly. > > > > > > > Paolo
Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Tue, Apr 10, 2018 at 09:23:53AM +, Liang, Cunming wrote: > > > > -Original Message- > > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > > Sent: Tuesday, April 10, 2018 3:52 PM > > To: Bie, Tiwei <tiwei@intel.com>; Jason Wang <jasow...@redhat.com> > > Cc: m...@redhat.com; alex.william...@redhat.com; ddut...@redhat.com; > > Duyck, Alexander H <alexander.h.du...@intel.com>; virtio-dev@lists.oasis- > > open.org; linux-ker...@vger.kernel.org; k...@vger.kernel.org; > > virtualizat...@lists.linux-foundation.org; netdev@vger.kernel.org; Daly, Dan > > <dan.d...@intel.com>; Liang, Cunming <cunming.li...@intel.com>; Wang, > > Zhihong <zhihong.w...@intel.com>; Tan, Jianfeng <jianfeng@intel.com>; > > Wang, Xiao W <xiao.w.w...@intel.com> > > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware > > vhost backend > > > > On 10/04/2018 06:57, Tiwei Bie wrote: > > >> So you just move the abstraction layer from qemu to kernel, and you > > >> still need different drivers in kernel for different device > > >> interfaces of accelerators. This looks even more complex than leaving > > >> it in qemu. As you said, another idea is to implement userspace vhost > > >> backend for accelerators which seems easier and could co-work with > > >> other parts of qemu without inventing new type of messages. > > > > > > I'm not quite sure. Do you think it's acceptable to add various vendor > > > specific hardware drivers in QEMU? > > > > I think so. We have vendor-specific quirks, and at some point there was an > > idea of using quirks to implement (vendor-specific) live migration support > > for > > assigned devices. > > Vendor-specific quirks of accessing VGA is a small portion. Other major > portions are still handled by guest driver. > > While in this case, when saying various vendor specific drivers in QEMU, it > says QEMU takes over and provides the entire user space device drivers. Some > parts are even not relevant to vhost, they're basic device function enabling. > Moreover, it could be different kinds of devices(network/block/...) under > vhost. No matter # of vendors or # of types, total LOC is not small. > > The idea is to avoid introducing these extra complexity out of QEMU, keeping > vhost adapter simple. As vhost protocol is de factor standard, it leverages > kernel device driver to provide the diversity. Changing once in QEMU, then it > supports multi-vendor devices whose drivers naturally providing kernel driver > there. > > If QEMU is going to build a user space driver framework there, we're open > mind on that, even leveraging DPDK as the underlay library. Looking forward > to more others' comments from community. > > Steve Dependency on a kernel driver is fine IMHO. It's the dependency on a DPDK backend that makes people unhappy, since the functionality in question is setup-time only. As others said, we do not need to go overeboard. A couple of small vendor-specific quirks in qemu isn't a big deal. > > > > Paolo
RE: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
> -Original Message- > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > Sent: Tuesday, April 10, 2018 3:52 PM > To: Bie, Tiwei <tiwei@intel.com>; Jason Wang <jasow...@redhat.com> > Cc: m...@redhat.com; alex.william...@redhat.com; ddut...@redhat.com; > Duyck, Alexander H <alexander.h.du...@intel.com>; virtio-dev@lists.oasis- > open.org; linux-ker...@vger.kernel.org; k...@vger.kernel.org; > virtualizat...@lists.linux-foundation.org; netdev@vger.kernel.org; Daly, Dan > <dan.d...@intel.com>; Liang, Cunming <cunming.li...@intel.com>; Wang, > Zhihong <zhihong.w...@intel.com>; Tan, Jianfeng <jianfeng@intel.com>; > Wang, Xiao W <xiao.w.w...@intel.com> > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware > vhost backend > > On 10/04/2018 06:57, Tiwei Bie wrote: > >> So you just move the abstraction layer from qemu to kernel, and you > >> still need different drivers in kernel for different device > >> interfaces of accelerators. This looks even more complex than leaving > >> it in qemu. As you said, another idea is to implement userspace vhost > >> backend for accelerators which seems easier and could co-work with > >> other parts of qemu without inventing new type of messages. > > > > I'm not quite sure. Do you think it's acceptable to add various vendor > > specific hardware drivers in QEMU? > > I think so. We have vendor-specific quirks, and at some point there was an > idea of using quirks to implement (vendor-specific) live migration support for > assigned devices. Vendor-specific quirks of accessing VGA is a small portion. Other major portions are still handled by guest driver. While in this case, when saying various vendor specific drivers in QEMU, it says QEMU takes over and provides the entire user space device drivers. Some parts are even not relevant to vhost, they're basic device function enabling. Moreover, it could be different kinds of devices(network/block/...) under vhost. No matter # of vendors or # of types, total LOC is not small. The idea is to avoid introducing these extra complexity out of QEMU, keeping vhost adapter simple. As vhost protocol is de factor standard, it leverages kernel device driver to provide the diversity. Changing once in QEMU, then it supports multi-vendor devices whose drivers naturally providing kernel driver there. If QEMU is going to build a user space driver framework there, we're open mind on that, even leveraging DPDK as the underlay library. Looking forward to more others' comments from community. Steve > > Paolo
Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On 10/04/2018 06:57, Tiwei Bie wrote: >> So you just move the abstraction layer from qemu to kernel, and you still >> need different drivers in kernel for different device interfaces of >> accelerators. This looks even more complex than leaving it in qemu. As you >> said, another idea is to implement userspace vhost backend for accelerators >> which seems easier and could co-work with other parts of qemu without >> inventing new type of messages. > > I'm not quite sure. Do you think it's acceptable to > add various vendor specific hardware drivers in QEMU? I think so. We have vendor-specific quirks, and at some point there was an idea of using quirks to implement (vendor-specific) live migration support for assigned devices. Paolo
Re: [RFC] vhost: introduce mdev based hardware vhost backend
On 2018年04月10日 12:57, Tiwei Bie wrote: On Tue, Apr 10, 2018 at 10:52:52AM +0800, Jason Wang wrote: On 2018年04月02日 23:23, Tiwei Bie wrote: This patch introduces a mdev (mediated device) based hardware vhost backend. This backend is an abstraction of the various hardware vhost accelerators (potentially any device that uses virtio ring can be used as a vhost accelerator). Some generic mdev parent ops are provided for accelerator drivers to support generating mdev instances. What's this === The idea is that we can setup a virtio ring compatible device with the messages available at the vhost-backend. Originally, these messages are used to implement a software vhost backend, but now we will use these messages to setup a virtio ring compatible hardware device. Then the hardware device will be able to work with the guest virtio driver in the VM just like what the software backend does. That is to say, we can implement a hardware based vhost backend in QEMU, and any virtio ring compatible devices potentially can be used with this backend. (We also call it vDPA -- vhost Data Path Acceleration). One problem is that, different virtio ring compatible devices may have different device interfaces. That is to say, we will need different drivers in QEMU. It could be troublesome. And that's what this patch trying to fix. The idea behind this patch is very simple: mdev is a standard way to emulate device in kernel. So you just move the abstraction layer from qemu to kernel, and you still need different drivers in kernel for different device interfaces of accelerators. This looks even more complex than leaving it in qemu. As you said, another idea is to implement userspace vhost backend for accelerators which seems easier and could co-work with other parts of qemu without inventing new type of messages. I'm not quite sure. Do you think it's acceptable to add various vendor specific hardware drivers in QEMU? I don't object but we need to figure out the advantages of doing it in qemu too. Thanks
Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Tue, Apr 10, 2018 at 10:52:52AM +0800, Jason Wang wrote:
> On 2018年04月02日 23:23, Tiwei Bie wrote:
> > This patch introduces a mdev (mediated device) based hardware
> > vhost backend. This backend is an abstraction of the various
> > hardware vhost accelerators (potentially any device that uses
> > virtio ring can be used as a vhost accelerator). Some generic
> > mdev parent ops are provided for accelerator drivers to support
> > generating mdev instances.
> >
> > What's this
> > ===
> >
> > The idea is that we can setup a virtio ring compatible device
> > with the messages available at the vhost-backend. Originally,
> > these messages are used to implement a software vhost backend,
> > but now we will use these messages to setup a virtio ring
> > compatible hardware device. Then the hardware device will be
> > able to work with the guest virtio driver in the VM just like
> > what the software backend does. That is to say, we can implement
> > a hardware based vhost backend in QEMU, and any virtio ring
> > compatible devices potentially can be used with this backend.
> > (We also call it vDPA -- vhost Data Path Acceleration).
> >
> > One problem is that, different virtio ring compatible devices
> > may have different device interfaces. That is to say, we will
> > need different drivers in QEMU. It could be troublesome. And
> > that's what this patch trying to fix. The idea behind this
> > patch is very simple: mdev is a standard way to emulate device
> > in kernel.
>
> So you just move the abstraction layer from qemu to kernel, and you still
> need different drivers in kernel for different device interfaces of
> accelerators. This looks even more complex than leaving it in qemu. As you
> said, another idea is to implement userspace vhost backend for accelerators
> which seems easier and could co-work with other parts of qemu without
> inventing new type of messages.
I'm not quite sure. Do you think it's acceptable to
add various vendor specific hardware drivers in QEMU?
>
> Need careful thought here to seek a best solution here.
Yeah, definitely! :)
And your opinions would be very helpful!
>
> > So we defined a standard device based on mdev, which
> > is able to accept vhost messages. When the mdev emulation code
> > (i.e. the generic mdev parent ops provided by this patch) gets
> > vhost messages, it will parse and deliver them to accelerator
> > drivers. Drivers can use these messages to setup accelerators.
> >
> > That is to say, the generic mdev parent ops (e.g. read()/write()/
> > ioctl()/...) will be provided for accelerator drivers to register
> > accelerators as mdev parent devices. And each accelerator device
> > will support generating standard mdev instance(s).
> >
> > With this standard device interface, we will be able to just
> > develop one userspace driver to implement the hardware based
> > vhost backend in QEMU.
> >
> > Difference between vDPA and PCI passthru
> >
> >
> > The key difference between vDPA and PCI passthru is that, in
> > vDPA only the data path of the device (e.g. DMA ring, notify
> > region and queue interrupt) is pass-throughed to the VM, the
> > device control path (e.g. PCI configuration space and MMIO
> > regions) is still defined and emulated by QEMU.
> >
> > The benefits of keeping virtio device emulation in QEMU compared
> > with virtio device PCI passthru include (but not limit to):
> >
> > - consistent device interface for guest OS in the VM;
> > - max flexibility on the hardware design, especially the
> >accelerator for each vhost backend doesn't have to be a
> >full PCI device;
> > - leveraging the existing virtio live-migration framework;
> >
> > The interface of this mdev based device
> > ===
> >
> > 1. BAR0
> >
> > The MMIO region described by BAR0 is the main control
> > interface. Messages will be written to or read from
> > this region.
> >
> > The message type is determined by the `request` field
> > in message header. The message size is encoded in the
> > message header too. The message format looks like this:
> >
> > struct vhost_vfio_op {
> > __u64 request;
> > __u32 flags;
> > /* Flag values: */
> > #define VHOST_VFIO_NEED_REPLY 0x1 /* Whether need reply */
> > __u32 size;
> > union {
> > __u64 u64;
> > struct vhost_vring_state state;
> > struct vhost_vring_addr addr;
> > struct vhost_memory memory;
> > } payload;
> > };
> >
> > The existing vhost-kernel ioctl cmds are reused as
> > the message requests in above structure.
> >
> > Each message will be written to or read from this
> > region at offset 0:
> >
> > int vhost_vfio_write(struct vhost_dev *dev, struct vhost_vfio_op *op)
> > {
> > int count = VHOST_VFIO_OP_HDR_SIZE + op->size;
> > struct vhost_vfio *vfio = dev->opaque;
> > int ret;
> >
> > ret = pwrite64(vfio->device_fd, op, count,
Re: [RFC] vhost: introduce mdev based hardware vhost backend
On 2018年04月02日 23:23, Tiwei Bie wrote:
This patch introduces a mdev (mediated device) based hardware
vhost backend. This backend is an abstraction of the various
hardware vhost accelerators (potentially any device that uses
virtio ring can be used as a vhost accelerator). Some generic
mdev parent ops are provided for accelerator drivers to support
generating mdev instances.
What's this
===
The idea is that we can setup a virtio ring compatible device
with the messages available at the vhost-backend. Originally,
these messages are used to implement a software vhost backend,
but now we will use these messages to setup a virtio ring
compatible hardware device. Then the hardware device will be
able to work with the guest virtio driver in the VM just like
what the software backend does. That is to say, we can implement
a hardware based vhost backend in QEMU, and any virtio ring
compatible devices potentially can be used with this backend.
(We also call it vDPA -- vhost Data Path Acceleration).
One problem is that, different virtio ring compatible devices
may have different device interfaces. That is to say, we will
need different drivers in QEMU. It could be troublesome. And
that's what this patch trying to fix. The idea behind this
patch is very simple: mdev is a standard way to emulate device
in kernel.
So you just move the abstraction layer from qemu to kernel, and you
still need different drivers in kernel for different device interfaces
of accelerators. This looks even more complex than leaving it in qemu.
As you said, another idea is to implement userspace vhost backend for
accelerators which seems easier and could co-work with other parts of
qemu without inventing new type of messages.
Need careful thought here to seek a best solution here.
So we defined a standard device based on mdev, which
is able to accept vhost messages. When the mdev emulation code
(i.e. the generic mdev parent ops provided by this patch) gets
vhost messages, it will parse and deliver them to accelerator
drivers. Drivers can use these messages to setup accelerators.
That is to say, the generic mdev parent ops (e.g. read()/write()/
ioctl()/...) will be provided for accelerator drivers to register
accelerators as mdev parent devices. And each accelerator device
will support generating standard mdev instance(s).
With this standard device interface, we will be able to just
develop one userspace driver to implement the hardware based
vhost backend in QEMU.
Difference between vDPA and PCI passthru
The key difference between vDPA and PCI passthru is that, in
vDPA only the data path of the device (e.g. DMA ring, notify
region and queue interrupt) is pass-throughed to the VM, the
device control path (e.g. PCI configuration space and MMIO
regions) is still defined and emulated by QEMU.
The benefits of keeping virtio device emulation in QEMU compared
with virtio device PCI passthru include (but not limit to):
- consistent device interface for guest OS in the VM;
- max flexibility on the hardware design, especially the
accelerator for each vhost backend doesn't have to be a
full PCI device;
- leveraging the existing virtio live-migration framework;
The interface of this mdev based device
===
1. BAR0
The MMIO region described by BAR0 is the main control
interface. Messages will be written to or read from
this region.
The message type is determined by the `request` field
in message header. The message size is encoded in the
message header too. The message format looks like this:
struct vhost_vfio_op {
__u64 request;
__u32 flags;
/* Flag values: */
#define VHOST_VFIO_NEED_REPLY 0x1 /* Whether need reply */
__u32 size;
union {
__u64 u64;
struct vhost_vring_state state;
struct vhost_vring_addr addr;
struct vhost_memory memory;
} payload;
};
The existing vhost-kernel ioctl cmds are reused as
the message requests in above structure.
Each message will be written to or read from this
region at offset 0:
int vhost_vfio_write(struct vhost_dev *dev, struct vhost_vfio_op *op)
{
int count = VHOST_VFIO_OP_HDR_SIZE + op->size;
struct vhost_vfio *vfio = dev->opaque;
int ret;
ret = pwrite64(vfio->device_fd, op, count, vfio->bar0_offset);
if (ret != count)
return -1;
return 0;
}
int vhost_vfio_read(struct vhost_dev *dev, struct vhost_vfio_op *op)
{
int count = VHOST_VFIO_OP_HDR_SIZE + op->size;
struct vhost_vfio *vfio = dev->opaque;
uint64_t request = op->request;
int ret;
ret = pread64(vfio->device_fd, op, count, vfio->bar0_offset);
if (ret != count || request != op->request)
return -1;
return 0;
}
It's quite straightforward to set things to the device.
Just need to write the message to
[RFC] vhost: introduce mdev based hardware vhost backend
This patch introduces a mdev (mediated device) based hardware
vhost backend. This backend is an abstraction of the various
hardware vhost accelerators (potentially any device that uses
virtio ring can be used as a vhost accelerator). Some generic
mdev parent ops are provided for accelerator drivers to support
generating mdev instances.
What's this
===
The idea is that we can setup a virtio ring compatible device
with the messages available at the vhost-backend. Originally,
these messages are used to implement a software vhost backend,
but now we will use these messages to setup a virtio ring
compatible hardware device. Then the hardware device will be
able to work with the guest virtio driver in the VM just like
what the software backend does. That is to say, we can implement
a hardware based vhost backend in QEMU, and any virtio ring
compatible devices potentially can be used with this backend.
(We also call it vDPA -- vhost Data Path Acceleration).
One problem is that, different virtio ring compatible devices
may have different device interfaces. That is to say, we will
need different drivers in QEMU. It could be troublesome. And
that's what this patch trying to fix. The idea behind this
patch is very simple: mdev is a standard way to emulate device
in kernel. So we defined a standard device based on mdev, which
is able to accept vhost messages. When the mdev emulation code
(i.e. the generic mdev parent ops provided by this patch) gets
vhost messages, it will parse and deliver them to accelerator
drivers. Drivers can use these messages to setup accelerators.
That is to say, the generic mdev parent ops (e.g. read()/write()/
ioctl()/...) will be provided for accelerator drivers to register
accelerators as mdev parent devices. And each accelerator device
will support generating standard mdev instance(s).
With this standard device interface, we will be able to just
develop one userspace driver to implement the hardware based
vhost backend in QEMU.
Difference between vDPA and PCI passthru
The key difference between vDPA and PCI passthru is that, in
vDPA only the data path of the device (e.g. DMA ring, notify
region and queue interrupt) is pass-throughed to the VM, the
device control path (e.g. PCI configuration space and MMIO
regions) is still defined and emulated by QEMU.
The benefits of keeping virtio device emulation in QEMU compared
with virtio device PCI passthru include (but not limit to):
- consistent device interface for guest OS in the VM;
- max flexibility on the hardware design, especially the
accelerator for each vhost backend doesn't have to be a
full PCI device;
- leveraging the existing virtio live-migration framework;
The interface of this mdev based device
===
1. BAR0
The MMIO region described by BAR0 is the main control
interface. Messages will be written to or read from
this region.
The message type is determined by the `request` field
in message header. The message size is encoded in the
message header too. The message format looks like this:
struct vhost_vfio_op {
__u64 request;
__u32 flags;
/* Flag values: */
#define VHOST_VFIO_NEED_REPLY 0x1 /* Whether need reply */
__u32 size;
union {
__u64 u64;
struct vhost_vring_state state;
struct vhost_vring_addr addr;
struct vhost_memory memory;
} payload;
};
The existing vhost-kernel ioctl cmds are reused as
the message requests in above structure.
Each message will be written to or read from this
region at offset 0:
int vhost_vfio_write(struct vhost_dev *dev, struct vhost_vfio_op *op)
{
int count = VHOST_VFIO_OP_HDR_SIZE + op->size;
struct vhost_vfio *vfio = dev->opaque;
int ret;
ret = pwrite64(vfio->device_fd, op, count, vfio->bar0_offset);
if (ret != count)
return -1;
return 0;
}
int vhost_vfio_read(struct vhost_dev *dev, struct vhost_vfio_op *op)
{
int count = VHOST_VFIO_OP_HDR_SIZE + op->size;
struct vhost_vfio *vfio = dev->opaque;
uint64_t request = op->request;
int ret;
ret = pread64(vfio->device_fd, op, count, vfio->bar0_offset);
if (ret != count || request != op->request)
return -1;
return 0;
}
It's quite straightforward to set things to the device.
Just need to write the message to device directly:
int vhost_vfio_set_features(struct vhost_dev *dev, uint64_t features)
{
struct vhost_vfio_op op;
op.request = VHOST_SET_FEATURES;
op.flags = 0;
op.size = sizeof(features);
op.payload.u64 = features;
return vhost_vfio_write(dev, );
}
To get things from the device, two steps are needed.
Take VHOST_GET_FEATURE as an example:
int vhost_vfio_get_features(struct vhost_dev *dev, uint64_t *features)
{
struct vhost_vfio_op op;
