Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On 2018年04月10日 22:23, Liang, Cunming wrote: -Original Message- From: Michael S. Tsirkin [mailto:m...@redhat.com] Sent: Tuesday, April 10, 2018 9:36 PM To: Liang, Cunming Cc: Paolo Bonzini; Bie, Tiwei; Jason Wang;alex.william...@redhat.com; ddut...@redhat.com; Duyck, Alexander H; virtio-dev@lists.oasis-open.org;linux-ker...@vger.kernel.org; k...@vger.kernel.org;virtualizat...@lists.linux-foundation.org; net...@vger.kernel.org; Daly, Dan; Wang, Zhihong ; Tan, Jianfeng; Wang, Xiao W Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend On Tue, Apr 10, 2018 at 09:23:53AM +, Liang, Cunming wrote: -Original Message- From: Paolo Bonzini [mailto:pbonz...@redhat.com] Sent: Tuesday, April 10, 2018 3:52 PM To: Bie, Tiwei; Jason Wang Cc:m...@redhat.com;alex.william...@redhat.com;ddut...@redhat.com; Duyck, Alexander H; virtio-dev@lists.oasis- open.org;linux-ker...@vger.kernel.org; k...@vger.kernel.org;virtualizat...@lists.linux-foundation.org; net...@vger.kernel.org; Daly, Dan; Liang, Cunming; Wang, Zhihong ; Tan, Jianfeng; Wang, Xiao W Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend On 10/04/2018 06:57, Tiwei Bie wrote: So you just move the abstraction layer from qemu to kernel, and you still need different drivers in kernel for different device interfaces of accelerators. This looks even more complex than leaving it in qemu. As you said, another idea is to implement userspace vhost backend for accelerators which seems easier and could co-work with other parts of qemu without inventing new type of messages. I'm not quite sure. Do you think it's acceptable to add various vendor specific hardware drivers in QEMU? I think so. We have vendor-specific quirks, and at some point there was an idea of using quirks to implement (vendor-specific) live migration support for assigned devices. Vendor-specific quirks of accessing VGA is a small portion. Other major portions are still handled by guest driver. While in this case, when saying various vendor specific drivers in QEMU, it says QEMU takes over and provides the entire user space device drivers. Some parts are even not relevant to vhost, they're basic device function enabling. Moreover, it could be different kinds of devices(network/block/...) under vhost. No matter # of vendors or # of types, total LOC is not small. The idea is to avoid introducing these extra complexity out of QEMU, keeping vhost adapter simple. As vhost protocol is de factor standard, it leverages kernel device driver to provide the diversity. Changing once in QEMU, then it supports multi-vendor devices whose drivers naturally providing kernel driver there. If QEMU is going to build a user space driver framework there, we're open mind on that, even leveraging DPDK as the underlay library. Looking forward to more others' comments from community. Steve Dependency on a kernel driver is fine IMHO. It's the dependency on a DPDK backend that makes people unhappy, since the functionality in question is setup- time only. Agreed, we don't see dependency on kernel driver is a problem. At engineering level, kernel driver is harder than userspace driver. mdev based vhost backend (this patch set) is independent with vhost-user extension patch set. In fact, there're a few vhost-user providers, DPDK librte_vhost is one of them. FD.IO/VPP and snabbswitch have their own vhost-user providers. So I can't agree on vhost-user extension patch depends on DPDK backend. But anyway, that's the topic of another mail thread. Well we can treat mdev as another kind of transport of vhost-user. And technically we can even implement a relay mdev than forward vhost-user messages to dpdk. Thanks - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On 2018年04月10日 17:23, Liang, Cunming wrote: -Original Message- From: Paolo Bonzini [mailto:pbonz...@redhat.com] Sent: Tuesday, April 10, 2018 3:52 PM To: Bie, Tiwei ; Jason Wang Cc: m...@redhat.com; alex.william...@redhat.com; ddut...@redhat.com; Duyck, Alexander H ; virtio-dev@lists.oasis- open.org; linux-ker...@vger.kernel.org; k...@vger.kernel.org; virtualizat...@lists.linux-foundation.org; net...@vger.kernel.org; Daly, Dan ; Liang, Cunming ; Wang, Zhihong ; Tan, Jianfeng ; Wang, Xiao W Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend On 10/04/2018 06:57, Tiwei Bie wrote: So you just move the abstraction layer from qemu to kernel, and you still need different drivers in kernel for different device interfaces of accelerators. This looks even more complex than leaving it in qemu. As you said, another idea is to implement userspace vhost backend for accelerators which seems easier and could co-work with other parts of qemu without inventing new type of messages. I'm not quite sure. Do you think it's acceptable to add various vendor specific hardware drivers in QEMU? I think so. We have vendor-specific quirks, and at some point there was an idea of using quirks to implement (vendor-specific) live migration support for assigned devices. Vendor-specific quirks of accessing VGA is a small portion. Other major portions are still handled by guest driver. While in this case, when saying various vendor specific drivers in QEMU, it says QEMU takes over and provides the entire user space device drivers. Some parts are even not relevant to vhost, they're basic device function enabling. Moreover, it could be different kinds of devices(network/block/...) under vhost. No matter # of vendors or # of types, total LOC is not small. The idea is to avoid introducing these extra complexity out of QEMU, keeping vhost adapter simple. As vhost protocol is de factor standard, it leverages kernel device driver to provide the diversity. Changing once in QEMU, then it supports multi-vendor devices whose drivers naturally providing kernel driver there. Let me clarify my question, it's not qemu vs kenrel but userspace vs kernel. It could be a library which could be linked to qemu. Doing it in userspace have the following obvious advantages: - attack surface was limited to userspace - easier to be maintained (compared to kernel driver) - easier to be extended without introducing new userspace/kernel interfaces - not tied to a specific operating system If we want to do it in the kernel, need to consider to unify code between mdev device driver and generic driver. For net driver, maybe we can even consider to do it on top of exist drivers. If QEMU is going to build a user space driver framework there, we're open mind on that, even leveraging DPDK as the underlay library. Looking forward to more others' comments from community. I'm doing this now by implementing vhost inside qemu IOThreads. Hope I can post RFC in few months. Thanks Steve Paolo - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Tue, Apr 10, 2018 at 09:23:53AM +, Liang, Cunming wrote: > If QEMU is going to build a user space driver framework there, we're open > mind on that, even leveraging DPDK as the underlay library. Looking forward > to more others' comments from community. There is already an NVMe VFIO driver in QEMU (see block/nvme.c). So in principle there's no reason against userspace drivers in QEMU. Stefan signature.asc Description: PGP signature
RE: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
> From: Liang, Cunming > Sent: Tuesday, April 10, 2018 10:24 PM > [...] > > > > As others said, we do not need to go overeboard. A couple of small > vendor- > > specific quirks in qemu isn't a big deal. > > It's quite challenge to identify it's small or not, there's no uniform metric. > > It's only dependent on QEMU itself, that's the obvious benefit. Tradeoff is > to build the entire device driver. We don't object to do that in QEMU, but > wanna make sure to understand the boundary size clearly. > It might be helpful if you can post some sample code using proposed framework - then people have a clear feeling about what size is talked about here and whether it falls into the concept of 'small quirks'. Thanks Kevin - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
RE: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
> -Original Message- > From: Michael S. Tsirkin [mailto:m...@redhat.com] > Sent: Tuesday, April 10, 2018 9:36 PM > To: Liang, Cunming > Cc: Paolo Bonzini ; Bie, Tiwei ; > Jason Wang ; alex.william...@redhat.com; > ddut...@redhat.com; Duyck, Alexander H ; > virtio-dev@lists.oasis-open.org; linux-ker...@vger.kernel.org; > k...@vger.kernel.org; virtualizat...@lists.linux-foundation.org; > net...@vger.kernel.org; Daly, Dan ; Wang, Zhihong > ; Tan, Jianfeng ; Wang, Xiao > W > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost > backend > > On Tue, Apr 10, 2018 at 09:23:53AM +, Liang, Cunming wrote: > > > > > > > -Original Message- > > > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > > > Sent: Tuesday, April 10, 2018 3:52 PM > > > To: Bie, Tiwei ; Jason Wang > > > > > > Cc: m...@redhat.com; alex.william...@redhat.com; ddut...@redhat.com; > > > Duyck, Alexander H ; > > > virtio-dev@lists.oasis- open.org; linux-ker...@vger.kernel.org; > > > k...@vger.kernel.org; virtualizat...@lists.linux-foundation.org; > > > net...@vger.kernel.org; Daly, Dan ; Liang, > > > Cunming ; Wang, Zhihong > > > ; Tan, Jianfeng ; > > > Wang, Xiao W > > > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based > > > hardware vhost backend > > > > > > On 10/04/2018 06:57, Tiwei Bie wrote: > > > >> So you just move the abstraction layer from qemu to kernel, and > > > >> you still need different drivers in kernel for different device > > > >> interfaces of accelerators. This looks even more complex than > > > >> leaving it in qemu. As you said, another idea is to implement > > > >> userspace vhost backend for accelerators which seems easier and > > > >> could co-work with other parts of qemu without inventing new type of > messages. > > > > > > > > I'm not quite sure. Do you think it's acceptable to add various > > > > vendor specific hardware drivers in QEMU? > > > > > > I think so. We have vendor-specific quirks, and at some point there > > > was an idea of using quirks to implement (vendor-specific) live > > > migration support for assigned devices. > > > > Vendor-specific quirks of accessing VGA is a small portion. Other major > > portions > are still handled by guest driver. > > > > While in this case, when saying various vendor specific drivers in QEMU, it > > says > QEMU takes over and provides the entire user space device drivers. Some parts > are even not relevant to vhost, they're basic device function enabling. > Moreover, > it could be different kinds of devices(network/block/...) under vhost. No > matter > # of vendors or # of types, total LOC is not small. > > > > The idea is to avoid introducing these extra complexity out of QEMU, keeping > vhost adapter simple. As vhost protocol is de factor standard, it leverages > kernel > device driver to provide the diversity. Changing once in QEMU, then it > supports > multi-vendor devices whose drivers naturally providing kernel driver there. > > > > If QEMU is going to build a user space driver framework there, we're open > > mind > on that, even leveraging DPDK as the underlay library. Looking forward to more > others' comments from community. > > > > Steve > > Dependency on a kernel driver is fine IMHO. It's the dependency on a DPDK > backend that makes people unhappy, since the functionality in question is > setup- > time only. Agreed, we don't see dependency on kernel driver is a problem. mdev based vhost backend (this patch set) is independent with vhost-user extension patch set. In fact, there're a few vhost-user providers, DPDK librte_vhost is one of them. FD.IO/VPP and snabbswitch have their own vhost-user providers. So I can't agree on vhost-user extension patch depends on DPDK backend. But anyway, that's the topic of another mail thread. > > As others said, we do not need to go overeboard. A couple of small vendor- > specific quirks in qemu isn't a big deal. It's quite challenge to identify it's small or not, there's no uniform metric. It's only dependent on QEMU itself, that's the obvious benefit. Tradeoff is to build the entire device driver. We don't object to do that in QEMU, but wanna make sure to understand the boundary size clearly. > > > > > > > Paolo - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On Tue, Apr 10, 2018 at 09:23:53AM +, Liang, Cunming wrote: > > > > -Original Message- > > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > > Sent: Tuesday, April 10, 2018 3:52 PM > > To: Bie, Tiwei ; Jason Wang > > Cc: m...@redhat.com; alex.william...@redhat.com; ddut...@redhat.com; > > Duyck, Alexander H ; virtio-dev@lists.oasis- > > open.org; linux-ker...@vger.kernel.org; k...@vger.kernel.org; > > virtualizat...@lists.linux-foundation.org; net...@vger.kernel.org; Daly, Dan > > ; Liang, Cunming ; Wang, > > Zhihong ; Tan, Jianfeng ; > > Wang, Xiao W > > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware > > vhost backend > > > > On 10/04/2018 06:57, Tiwei Bie wrote: > > >> So you just move the abstraction layer from qemu to kernel, and you > > >> still need different drivers in kernel for different device > > >> interfaces of accelerators. This looks even more complex than leaving > > >> it in qemu. As you said, another idea is to implement userspace vhost > > >> backend for accelerators which seems easier and could co-work with > > >> other parts of qemu without inventing new type of messages. > > > > > > I'm not quite sure. Do you think it's acceptable to add various vendor > > > specific hardware drivers in QEMU? > > > > I think so. We have vendor-specific quirks, and at some point there was an > > idea of using quirks to implement (vendor-specific) live migration support > > for > > assigned devices. > > Vendor-specific quirks of accessing VGA is a small portion. Other major > portions are still handled by guest driver. > > While in this case, when saying various vendor specific drivers in QEMU, it > says QEMU takes over and provides the entire user space device drivers. Some > parts are even not relevant to vhost, they're basic device function enabling. > Moreover, it could be different kinds of devices(network/block/...) under > vhost. No matter # of vendors or # of types, total LOC is not small. > > The idea is to avoid introducing these extra complexity out of QEMU, keeping > vhost adapter simple. As vhost protocol is de factor standard, it leverages > kernel device driver to provide the diversity. Changing once in QEMU, then it > supports multi-vendor devices whose drivers naturally providing kernel driver > there. > > If QEMU is going to build a user space driver framework there, we're open > mind on that, even leveraging DPDK as the underlay library. Looking forward > to more others' comments from community. > > Steve Dependency on a kernel driver is fine IMHO. It's the dependency on a DPDK backend that makes people unhappy, since the functionality in question is setup-time only. As others said, we do not need to go overeboard. A couple of small vendor-specific quirks in qemu isn't a big deal. > > > > Paolo - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
RE: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
> -Original Message- > From: Paolo Bonzini [mailto:pbonz...@redhat.com] > Sent: Tuesday, April 10, 2018 3:52 PM > To: Bie, Tiwei ; Jason Wang > Cc: m...@redhat.com; alex.william...@redhat.com; ddut...@redhat.com; > Duyck, Alexander H ; virtio-dev@lists.oasis- > open.org; linux-ker...@vger.kernel.org; k...@vger.kernel.org; > virtualizat...@lists.linux-foundation.org; net...@vger.kernel.org; Daly, Dan > ; Liang, Cunming ; Wang, > Zhihong ; Tan, Jianfeng ; > Wang, Xiao W > Subject: Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware > vhost backend > > On 10/04/2018 06:57, Tiwei Bie wrote: > >> So you just move the abstraction layer from qemu to kernel, and you > >> still need different drivers in kernel for different device > >> interfaces of accelerators. This looks even more complex than leaving > >> it in qemu. As you said, another idea is to implement userspace vhost > >> backend for accelerators which seems easier and could co-work with > >> other parts of qemu without inventing new type of messages. > > > > I'm not quite sure. Do you think it's acceptable to add various vendor > > specific hardware drivers in QEMU? > > I think so. We have vendor-specific quirks, and at some point there was an > idea of using quirks to implement (vendor-specific) live migration support for > assigned devices. Vendor-specific quirks of accessing VGA is a small portion. Other major portions are still handled by guest driver. While in this case, when saying various vendor specific drivers in QEMU, it says QEMU takes over and provides the entire user space device drivers. Some parts are even not relevant to vhost, they're basic device function enabling. Moreover, it could be different kinds of devices(network/block/...) under vhost. No matter # of vendors or # of types, total LOC is not small. The idea is to avoid introducing these extra complexity out of QEMU, keeping vhost adapter simple. As vhost protocol is de factor standard, it leverages kernel device driver to provide the diversity. Changing once in QEMU, then it supports multi-vendor devices whose drivers naturally providing kernel driver there. If QEMU is going to build a user space driver framework there, we're open mind on that, even leveraging DPDK as the underlay library. Looking forward to more others' comments from community. Steve > > Paolo
Re: [virtio-dev] Re: [RFC] vhost: introduce mdev based hardware vhost backend
On 10/04/2018 06:57, Tiwei Bie wrote: >> So you just move the abstraction layer from qemu to kernel, and you still >> need different drivers in kernel for different device interfaces of >> accelerators. This looks even more complex than leaving it in qemu. As you >> said, another idea is to implement userspace vhost backend for accelerators >> which seems easier and could co-work with other parts of qemu without >> inventing new type of messages. > > I'm not quite sure. Do you think it's acceptable to > add various vendor specific hardware drivers in QEMU? I think so. We have vendor-specific quirks, and at some point there was an idea of using quirks to implement (vendor-specific) live migration support for assigned devices. Paolo - To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org
