Re: [Qemu-devel] [PATCH 4/6] vhost-user: support registering external host notifiers
On Thu, May 24, 2018 at 07:15:24PM +0300, Michael S. Tsirkin wrote:
> On Fri, May 25, 2018 at 12:05:50AM +0800, Tiwei Bie wrote:
> > On Thu, May 24, 2018 at 06:49:47PM +0300, Michael S. Tsirkin wrote:
> > > On Thu, May 24, 2018 at 06:33:34PM +0800, Tiwei Bie wrote:
> > > > This patch introduces VHOST_USER_PROTOCOL_F_HOST_NOTIFIER.
> > > > With this feature negotiated, vhost-user backend can register
> > > > memory region based host notifiers. And it will allow the guest
> > > > driver in the VM to notify the hardware accelerator at the
> > > > vhost-user backend directly.
> > > >
> > > > Signed-off-by: Tiwei Bie
> > >
> > > So maybe we don't need a new VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD?
> > > Let's check VHOST_USER_PROTOCOL_F_HOST_NOTIFIER instead?
> >
> > Yeah. I think it would be the best choice!
> >
> > If this feature (HOST NOTIFIER) is negotiated, it means
> > the QEMU is able to safely receive (the expected number
> > of) file descriptors.
> >
> > >
> > > > ---
> > [...]
> > > > +typedef struct VhostUserHostNotifier {
> > > > +MemoryRegion mr;
> > > > +void *addr;
> > > > +bool set;
> > > > +} VhostUserHostNotifier;
> > > >
> > > > typedef struct VhostUserState {
> > > > CharBackend *chr;
> > > > +VhostUserHostNotifier notifier[VIRTIO_QUEUE_MAX];
> > > > } VhostUserState;
> > >
> > > So this notifier is per-queue. Can't we maintain it in
> > > NetVhostUserState then?
> >
> > This notifier is per virtio-queue. But NetVhostUserState
> > is per net-queue-pair.
> >
> > And ideally, I think this structure shouldn't be visible
> > to net/vhost-user, vhost-user-scsi, vhost-user-crypto, etc.
> >
> > Best regards,
> > Tiwei Bie
>
> I wonder whether we can create a new per vq structure.
Maybe something like:
typedef struct VhostUserStatePerVQ {
VhostUserHostNotifier notifier;
} VhostUserStatePerVQ;
typedef struct VhostUserState {
CharBackend *chr;
VhostUserStatePerVQ queue[VIRTIO_QUEUE_MAX];
} VhostUserState;
Best regards,
Tiwei Bie
Re: [Qemu-devel] [PATCH 4/6] vhost-user: support registering external host notifiers
On Fri, May 25, 2018 at 12:05:50AM +0800, Tiwei Bie wrote:
> On Thu, May 24, 2018 at 06:49:47PM +0300, Michael S. Tsirkin wrote:
> > On Thu, May 24, 2018 at 06:33:34PM +0800, Tiwei Bie wrote:
> > > This patch introduces VHOST_USER_PROTOCOL_F_HOST_NOTIFIER.
> > > With this feature negotiated, vhost-user backend can register
> > > memory region based host notifiers. And it will allow the guest
> > > driver in the VM to notify the hardware accelerator at the
> > > vhost-user backend directly.
> > >
> > > Signed-off-by: Tiwei Bie
> >
> > So maybe we don't need a new VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD?
> > Let's check VHOST_USER_PROTOCOL_F_HOST_NOTIFIER instead?
>
> Yeah. I think it would be the best choice!
>
> If this feature (HOST NOTIFIER) is negotiated, it means
> the QEMU is able to safely receive (the expected number
> of) file descriptors.
>
> >
> > > ---
> [...]
> > > +typedef struct VhostUserHostNotifier {
> > > +MemoryRegion mr;
> > > +void *addr;
> > > +bool set;
> > > +} VhostUserHostNotifier;
> > >
> > > typedef struct VhostUserState {
> > > CharBackend *chr;
> > > +VhostUserHostNotifier notifier[VIRTIO_QUEUE_MAX];
> > > } VhostUserState;
> >
> > So this notifier is per-queue. Can't we maintain it in
> > NetVhostUserState then?
>
> This notifier is per virtio-queue. But NetVhostUserState
> is per net-queue-pair.
>
> And ideally, I think this structure shouldn't be visible
> to net/vhost-user, vhost-user-scsi, vhost-user-crypto, etc.
>
> Best regards,
> Tiwei Bie
I wonder whether we can create a new per vq structure.
--
MST
Re: [Qemu-devel] [PATCH 4/6] vhost-user: support registering external host notifiers
On Thu, May 24, 2018 at 06:49:47PM +0300, Michael S. Tsirkin wrote:
> On Thu, May 24, 2018 at 06:33:34PM +0800, Tiwei Bie wrote:
> > This patch introduces VHOST_USER_PROTOCOL_F_HOST_NOTIFIER.
> > With this feature negotiated, vhost-user backend can register
> > memory region based host notifiers. And it will allow the guest
> > driver in the VM to notify the hardware accelerator at the
> > vhost-user backend directly.
> >
> > Signed-off-by: Tiwei Bie
>
> So maybe we don't need a new VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD?
> Let's check VHOST_USER_PROTOCOL_F_HOST_NOTIFIER instead?
Yeah. I think it would be the best choice!
If this feature (HOST NOTIFIER) is negotiated, it means
the QEMU is able to safely receive (the expected number
of) file descriptors.
>
> > ---
[...]
> > +typedef struct VhostUserHostNotifier {
> > +MemoryRegion mr;
> > +void *addr;
> > +bool set;
> > +} VhostUserHostNotifier;
> >
> > typedef struct VhostUserState {
> > CharBackend *chr;
> > +VhostUserHostNotifier notifier[VIRTIO_QUEUE_MAX];
> > } VhostUserState;
>
> So this notifier is per-queue. Can't we maintain it in
> NetVhostUserState then?
This notifier is per virtio-queue. But NetVhostUserState
is per net-queue-pair.
And ideally, I think this structure shouldn't be visible
to net/vhost-user, vhost-user-scsi, vhost-user-crypto, etc.
Best regards,
Tiwei Bie
Re: [Qemu-devel] [PATCH 4/6] vhost-user: support registering external host notifiers
On Thu, May 24, 2018 at 06:33:34PM +0800, Tiwei Bie wrote:
> This patch introduces VHOST_USER_PROTOCOL_F_HOST_NOTIFIER.
> With this feature negotiated, vhost-user backend can register
> memory region based host notifiers. And it will allow the guest
> driver in the VM to notify the hardware accelerator at the
> vhost-user backend directly.
>
> Signed-off-by: Tiwei Bie
So maybe we don't need a new VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD?
Let's check VHOST_USER_PROTOCOL_F_HOST_NOTIFIER instead?
> ---
> docs/interop/vhost-user.txt| 33 ++
> hw/virtio/vhost-user.c | 113 +
> include/hw/virtio/vhost-user.h | 8 +++
> 3 files changed, 154 insertions(+)
>
> diff --git a/docs/interop/vhost-user.txt b/docs/interop/vhost-user.txt
> index 682a683eb4..d51fd58242 100644
> --- a/docs/interop/vhost-user.txt
> +++ b/docs/interop/vhost-user.txt
> @@ -132,6 +132,16 @@ Depending on the request type, payload can be:
> Payload: Size bytes array holding the contents of the virtio
> device's configuration space
>
> + * Vring area description
> + ---
> + | u64 | size | offset |
> + ---
> +
> + u64: a 64-bit integer contains vring index and flags
> + Size: a 64-bit size of this area
> + Offset: a 64-bit offset of this area from the start of the
> + supplied file descriptor
> +
> In QEMU the vhost-user message is implemented with the following struct:
>
> typedef struct VhostUserMsg {
> @@ -146,6 +156,7 @@ typedef struct VhostUserMsg {
> VhostUserLog log;
> struct vhost_iotlb_msg iotlb;
> VhostUserConfig config;
> +VhostUserVringArea area;
> };
> } QEMU_PACKED VhostUserMsg;
>
> @@ -385,6 +396,7 @@ Protocol features
> #define VHOST_USER_PROTOCOL_F_PAGEFAULT 8
> #define VHOST_USER_PROTOCOL_F_CONFIG 9
> #define VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD 10
> +#define VHOST_USER_PROTOCOL_F_HOST_NOTIFIER 11
>
> Master message types
>
> @@ -782,6 +794,27 @@ Slave message types
> the VHOST_USER_NEED_REPLY flag, master must respond with zero when
> operation is successfully completed, or non-zero otherwise.
>
> + * VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG
> +
> + Id: 3
> + Equivalent ioctl: N/A
> + Slave payload: vring area description
> + Master payload: N/A
> +
> + Sets host notifier for a specified queue. The queue index is contained
> + in the u64 field of the vring area description. The host notifier is
> + described by the file descriptor (typically it's a VFIO device fd)
> which
> + is passed as ancillary data and the size (which is mmap size and should
> + be the same as host page size) and offset (which is mmap offset)
> carried
> + in the vring area description. QEMU can mmap the file descriptor based
> + on the size and offset to get a memory range. Registering a host
> notifier
> + means mapping this memory range to the VM as the specified queue's
> notify
> + MMIO region. Slave sends this request to tell QEMU to de-register the
> + existing notifier if any and register the new notifier if the request
> is
> + sent with a file descriptor.
> + This request should be sent only when
> VHOST_USER_PROTOCOL_F_HOST_NOTIFIER
> + protocol feature has been successfully negotiated.
> +
> VHOST_USER_PROTOCOL_F_REPLY_ACK:
> ---
> The original vhost-user specification only demands replies for certain
> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> index 70b9610c87..b041343632 100644
> --- a/hw/virtio/vhost-user.c
> +++ b/hw/virtio/vhost-user.c
> @@ -13,6 +13,7 @@
> #include "hw/virtio/vhost.h"
> #include "hw/virtio/vhost-user.h"
> #include "hw/virtio/vhost-backend.h"
> +#include "hw/virtio/virtio.h"
> #include "hw/virtio/virtio-net.h"
> #include "chardev/char-fe.h"
> #include "sysemu/kvm.h"
> @@ -50,6 +51,7 @@ enum VhostUserProtocolFeature {
> VHOST_USER_PROTOCOL_F_PAGEFAULT = 8,
> VHOST_USER_PROTOCOL_F_CONFIG = 9,
> VHOST_USER_PROTOCOL_F_SLAVE_SEND_FD = 10,
> +VHOST_USER_PROTOCOL_F_HOST_NOTIFIER = 11,
> VHOST_USER_PROTOCOL_F_MAX
> };
>
> @@ -94,6 +96,7 @@ typedef enum VhostUserSlaveRequest {
> VHOST_USER_SLAVE_NONE = 0,
> VHOST_USER_SLAVE_IOTLB_MSG = 1,
> VHOST_USER_SLAVE_CONFIG_CHANGE_MSG = 2,
> +VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG = 3,
> VHOST_USER_SLAVE_MAX
> } VhostUserSlaveRequest;
>
> @@ -138,6 +141,12 @@ static VhostUserConfig c __attribute__ ((unused));
> + sizeof(c.size) \
> + sizeof(c.flags))
>
> +typedef struct VhostUserVringArea {
> +uint64_t u64;
> +uint64_t size;
> +uint64_t offset;
> +} VhostUserVringArea;
> +
> typedef struct {
> VhostUserRequest request;
>
> @@ -159,6 +168,7 @@ typedef union {
> stru
