Re: [PATCH RFC 2/2] vhost: support urgent descriptors
On Mon, Sep 22, 2014 at 11:30:23AM +0800, Jason Wang wrote: On 09/20/2014 06:00 PM, Paolo Bonzini wrote: Il 19/09/2014 09:10, Jason Wang ha scritto: -if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { +if (vq-urgent || !vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { So the urgent descriptor only work when event index was not enabled? This seems suboptimal, we may still want to benefit from event index even if urgent descriptor is used. Looks like we need return true here when vq-urgent is true? Its ||, not . Without event index, all descriptors are treated as urgent. Paolo The problem is if vq-urgent is true, the patch checks VRING_AVAIL_F_NO_INTERRUPT bit. This bit were set unconditionally in virtqueue_enable_cb() regardless of event index feature and cleared unconditionally in virtqueue_disable_cb(). The reverse actually, right? So virtqueue_enable_cb() was used to not only publish a new event index but also enable the urgent descriptor. And virtqueue_disable_cb() disabled all interrupts including the urgent descriptor. Guest won't get urgent interrupts by just adding virtqueue_add_outbuf_urgent() since what it needs is to enable and disable interrupt for !urgent descriptor. Right, we want a new API that advances event index but does not set VRING_AVAIL_F_NO_INTERRUPT. IMO still want to set VRING_AVAIL_F_NO_INTERRUPT when handling tx interrupts, to avoid interrupt storms. Btw, not sure urgent is a suitable name, since interrupt is often slow in kvm guest. And in fact virtio-net will probably use urgent descriptor for those packets (e.g stream packets who can be delayed a little bit to batch more bytes from userspace) who was not urgent compared to other packets. Yes but we are asking for an interrupt before event index is reached because something is waiting for the packet to be transmitted. I couldn't come up with a better name. -- MST -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC 2/2] vhost: support urgent descriptors
On 09/22/2014 02:55 PM, Michael S. Tsirkin wrote: On Mon, Sep 22, 2014 at 11:30:23AM +0800, Jason Wang wrote: On 09/20/2014 06:00 PM, Paolo Bonzini wrote: Il 19/09/2014 09:10, Jason Wang ha scritto: -if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { +if (vq-urgent || !vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { So the urgent descriptor only work when event index was not enabled? This seems suboptimal, we may still want to benefit from event index even if urgent descriptor is used. Looks like we need return true here when vq-urgent is true? Its ||, not . Without event index, all descriptors are treated as urgent. Paolo The problem is if vq-urgent is true, the patch checks VRING_AVAIL_F_NO_INTERRUPT bit. This bit were set unconditionally in virtqueue_enable_cb() regardless of event index feature and cleared unconditionally in virtqueue_disable_cb(). The reverse actually, right? Ah, right. So virtqueue_enable_cb() was used to not only publish a new event index but also enable the urgent descriptor. And virtqueue_disable_cb() disabled all interrupts including the urgent descriptor. Guest won't get urgent interrupts by just adding virtqueue_add_outbuf_urgent() since what it needs is to enable and disable interrupt for !urgent descriptor. Right, we want a new API that advances event index but does not set VRING_AVAIL_F_NO_INTERRUPT. IMO still want to set VRING_AVAIL_F_NO_INTERRUPT when handling tx interrupts, to avoid interrupt storms. I see, so urgent descriptor needs to be disabled in this case. But vhost parts need a little big changes, we could not just check VRING_AVAIL_F_NO_INTERRUPT when vq-urgent is true. If event index is enabled, we still need to check used event to make sure the current tx delayed interrupt works. But just re-using VRING_AVAIL_F_NO_INTERRUPT for urgent descriptor may not work in some case. I see codes of virtqueue_get_buf() that may breaks this: /* If we expect an interrupt for the next entry, tell host * by writing event index and flush out the write before * the read in the next get_buf call. */ if (!(vq-vring.avail-flags VRING_AVAIL_F_NO_INTERRUPT)) { vring_used_event(vq-vring) = vq-last_used_idx; virtio_mb(vq-weak_barriers); } Consider if only urgent descriptor is enabled, this will publish used event which in fact enable lots of unnecessary interrupt. In fact I don't quite understand how the above lines is used. Virtio-net stop the queue before enable the tx interrupt in start_xmit(), so the above lines will not run at all. Btw, not sure urgent is a suitable name, since interrupt is often slow in kvm guest. And in fact virtio-net will probably use urgent descriptor for those packets (e.g stream packets who can be delayed a little bit to batch more bytes from userspace) who was not urgent compared to other packets. Yes but we are asking for an interrupt before event index is reached because something is waiting for the packet to be transmitted. I couldn't come up with a better name. Ok. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC 2/2] vhost: support urgent descriptors
On Mon, Sep 22, 2014 at 05:55:23PM +0800, Jason Wang wrote: On 09/22/2014 02:55 PM, Michael S. Tsirkin wrote: On Mon, Sep 22, 2014 at 11:30:23AM +0800, Jason Wang wrote: On 09/20/2014 06:00 PM, Paolo Bonzini wrote: Il 19/09/2014 09:10, Jason Wang ha scritto: - if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { + if (vq-urgent || !vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { So the urgent descriptor only work when event index was not enabled? This seems suboptimal, we may still want to benefit from event index even if urgent descriptor is used. Looks like we need return true here when vq-urgent is true? Its ||, not . Without event index, all descriptors are treated as urgent. Paolo The problem is if vq-urgent is true, the patch checks VRING_AVAIL_F_NO_INTERRUPT bit. This bit were set unconditionally in virtqueue_enable_cb() regardless of event index feature and cleared unconditionally in virtqueue_disable_cb(). The reverse actually, right? Ah, right. So virtqueue_enable_cb() was used to not only publish a new event index but also enable the urgent descriptor. And virtqueue_disable_cb() disabled all interrupts including the urgent descriptor. Guest won't get urgent interrupts by just adding virtqueue_add_outbuf_urgent() since what it needs is to enable and disable interrupt for !urgent descriptor. Right, we want a new API that advances event index but does not set VRING_AVAIL_F_NO_INTERRUPT. IMO still want to set VRING_AVAIL_F_NO_INTERRUPT when handling tx interrupts, to avoid interrupt storms. I see, so urgent descriptor needs to be disabled in this case. But vhost parts need a little big changes, we could not just check VRING_AVAIL_F_NO_INTERRUPT when vq-urgent is true. If event index is enabled, we still need to check used event to make sure the current tx delayed interrupt works. But just re-using VRING_AVAIL_F_NO_INTERRUPT for urgent descriptor may not work in some case. I see codes of virtqueue_get_buf() that may breaks this: /* If we expect an interrupt for the next entry, tell host * by writing event index and flush out the write before * the read in the next get_buf call. */ if (!(vq-vring.avail-flags VRING_AVAIL_F_NO_INTERRUPT)) { vring_used_event(vq-vring) = vq-last_used_idx; virtio_mb(vq-weak_barriers); } Right, we need to change this path to use a private flag. Consider if only urgent descriptor is enabled, this will publish used event which in fact enable lots of unnecessary interrupt. In fact I don't quite understand how the above lines is used. Virtio-net stop the queue before enable the tx interrupt in start_xmit(), so the above lines will not run at all. True - this code is for drivers which process VQ without disabling interrupts first. No rule says this is not allowed. Btw, not sure urgent is a suitable name, since interrupt is often slow in kvm guest. And in fact virtio-net will probably use urgent descriptor for those packets (e.g stream packets who can be delayed a little bit to batch more bytes from userspace) who was not urgent compared to other packets. Yes but we are asking for an interrupt before event index is reached because something is waiting for the packet to be transmitted. I couldn't come up with a better name. Ok. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC 2/2] vhost: support urgent descriptors
On 09/20/2014 06:00 PM, Paolo Bonzini wrote: Il 19/09/2014 09:10, Jason Wang ha scritto: - if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { + if (vq-urgent || !vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { So the urgent descriptor only work when event index was not enabled? This seems suboptimal, we may still want to benefit from event index even if urgent descriptor is used. Looks like we need return true here when vq-urgent is true? Its ||, not . Without event index, all descriptors are treated as urgent. Paolo The problem is if vq-urgent is true, the patch checks VRING_AVAIL_F_NO_INTERRUPT bit. This bit were set unconditionally in virtqueue_enable_cb() regardless of event index feature and cleared unconditionally in virtqueue_disable_cb(). So virtqueue_enable_cb() was used to not only publish a new event index but also enable the urgent descriptor. And virtqueue_disable_cb() disabled all interrupts including the urgent descriptor. Guest won't get urgent interrupts by just adding virtqueue_add_outbuf_urgent() since what it needs is to enable and disable interrupt for !urgent descriptor. Btw, not sure urgent is a suitable name, since interrupt is often slow in kvm guest. And in fact virtio-net will probably use urgent descriptor for those packets (e.g stream packets who can be delayed a little bit to batch more bytes from userspace) who was not urgent compared to other packets. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC 2/2] vhost: support urgent descriptors
Il 19/09/2014 09:10, Jason Wang ha scritto: - if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { + if (vq-urgent || !vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { So the urgent descriptor only work when event index was not enabled? This seems suboptimal, we may still want to benefit from event index even if urgent descriptor is used. Looks like we need return true here when vq-urgent is true? Its ||, not . Without event index, all descriptors are treated as urgent. Paolo -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC 2/2] vhost: support urgent descriptors
On 07/01/2014 06:49 PM, Michael S. Tsirkin wrote: Signed-off-by: Michael S. Tsirkin m...@redhat.com --- drivers/vhost/vhost.h | 19 +-- drivers/vhost/net.c | 30 +- drivers/vhost/scsi.c | 23 +++ drivers/vhost/test.c | 5 +++-- drivers/vhost/vhost.c | 23 --- 5 files changed, 68 insertions(+), 32 deletions(-) diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 3eda654..61ca542 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h [...] EXPORT_SYMBOL_GPL(vhost_add_used_n); @@ -1433,12 +1439,13 @@ static bool vhost_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) unlikely(vq-avail_idx == vq-last_avail_idx)) return true; - if (!vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { + if (vq-urgent || !vhost_has_feature(vq, VIRTIO_RING_F_EVENT_IDX)) { So the urgent descriptor only work when event index was not enabled? This seems suboptimal, we may still want to benefit from event index even if urgent descriptor is used. Looks like we need return true here when vq-urgent is true? Another question is whether or not we need to do this a new flag. Technically we can do it purely in guest side through event index, this can help to eliminate both the changes in host and a new feature bit. __u16 flags; if (__get_user(flags, vq-avail-flags)) { vq_err(vq, Failed to get flags); return true; } + vq-urgent = false; return !(flags VRING_AVAIL_F_NO_INTERRUPT); } old = vq-signalled_used; @@ -1468,9 +1475,10 @@ EXPORT_SYMBOL_GPL(vhost_signal); /* And here's the combo meal deal. Supersize me! */ void vhost_add_used_and_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq, +bool urgent, unsigned int head, int len) { - vhost_add_used(vq, head, len); + vhost_add_used(vq, urgent, head, len); vhost_signal(dev, vq); } EXPORT_SYMBOL_GPL(vhost_add_used_and_signal); @@ -1478,9 +1486,10 @@ EXPORT_SYMBOL_GPL(vhost_add_used_and_signal); /* multi-buffer version of vhost_add_used_and_signal */ void vhost_add_used_and_signal_n(struct vhost_dev *dev, struct vhost_virtqueue *vq, + bool urgent, struct vring_used_elem *heads, unsigned count) { - vhost_add_used_n(vq, heads, count); + vhost_add_used_n(vq, urgent, heads, count); vhost_signal(dev, vq); } EXPORT_SYMBOL_GPL(vhost_add_used_and_signal_n); -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH RFC 2/2] vhost: support urgent descriptors
On 07/01/2014 06:49 PM, Michael S. Tsirkin wrote: Signed-off-by: Michael S. Tsirkin m...@redhat.com --- drivers/vhost/vhost.h | 19 +-- drivers/vhost/net.c | 30 +- drivers/vhost/scsi.c | 23 +++ drivers/vhost/test.c | 5 +++-- drivers/vhost/vhost.c | 23 --- 5 files changed, 68 insertions(+), 32 deletions(-) diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 3eda654..61ca542 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -96,6 +96,9 @@ struct vhost_virtqueue { /* Last used index value we have signalled on */ bool signalled_used_valid; + /* Urgent descriptor was used */ + bool urgent; + /* Log writes to used structure. */ bool log_used; u64 log_addr; @@ -138,20 +141,24 @@ long vhost_vring_ioctl(struct vhost_dev *d, int ioctl, void __user *argp); int vhost_vq_access_ok(struct vhost_virtqueue *vq); int vhost_log_access_ok(struct vhost_dev *); -int vhost_get_vq_desc(struct vhost_virtqueue *, +int vhost_get_vq_desc(struct vhost_virtqueue *, bool *urgent, struct iovec iov[], unsigned int iov_count, unsigned int *out_num, unsigned int *in_num, struct vhost_log *log, unsigned int *log_num); void vhost_discard_vq_desc(struct vhost_virtqueue *, int n); int vhost_init_used(struct vhost_virtqueue *); -int vhost_add_used(struct vhost_virtqueue *, unsigned int head, int len); -int vhost_add_used_n(struct vhost_virtqueue *, struct vring_used_elem *heads, - unsigned count); -void vhost_add_used_and_signal(struct vhost_dev *, struct vhost_virtqueue *, +int vhost_add_used(struct vhost_virtqueue *, bool urgent, +unsigned int head, int len); +int vhost_add_used_n(struct vhost_virtqueue *, bool urgent, + struct vring_used_elem *heads, unsigned count); +void vhost_add_used_and_signal(struct vhost_dev *, +struct vhost_virtqueue *, +bool urgent, unsigned int id, int len); void vhost_add_used_and_signal_n(struct vhost_dev *, struct vhost_virtqueue *, -struct vring_used_elem *heads, unsigned count); + bool urgent, + struct vring_used_elem *heads, unsigned count); void vhost_signal(struct vhost_dev *, struct vhost_virtqueue *); void vhost_disable_notify(struct vhost_dev *, struct vhost_virtqueue *); bool vhost_enable_notify(struct vhost_dev *, struct vhost_virtqueue *); diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 8dae2f7..5f0567f 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -48,9 +48,11 @@ MODULE_PARM_DESC(experimental_zcopytx, Enable Zero Copy TX; * status internally; used for zerocopy tx only. */ /* Lower device DMA failed */ -#define VHOST_DMA_FAILED_LEN 3 +#define VHOST_DMA_FAILED_LEN 4 /* Lower device DMA done */ -#define VHOST_DMA_DONE_LEN 2 +#define VHOST_DMA_DONE_LEN 3 +/* Lower device DMA in progress, urgent bit set */ +#define VHOST_DMA_URGENT 2 /* Lower device DMA in progress */ #define VHOST_DMA_IN_PROGRESS1 /* Buffer unused */ @@ -284,11 +286,13 @@ static void vhost_zerocopy_signal_used(struct vhost_net *net, container_of(vq, struct vhost_net_virtqueue, vq); int i, add; int j = 0; + bool urgent = false; for (i = nvq-done_idx; i != nvq-upend_idx; i = (i + 1) % UIO_MAXIOV) { if (vq-heads[i].len == VHOST_DMA_FAILED_LEN) vhost_net_tx_err(net); if (VHOST_DMA_IS_DONE(vq-heads[i].len)) { + urgent = urgent || vq-heads[i].len == VHOST_DMA_URGENT; A problem is len was either VHOST_DMA_DONE_LEN or VHOST_DMA_FAILED_LEN here. Looks like we need another new VHOST_DMA_DONE_LEN_URGENT and change the len to this value if it was an urgent descriptor in zerocopy callback. vq-heads[i].len = VHOST_DMA_CLEAR_LEN; ++j; } else @@ -296,7 +300,7 @@ static void vhost_zerocopy_signal_used(struct vhost_net *net, } while (j) { add = min(UIO_MAXIOV - nvq-done_idx, j); - vhost_add_used_and_signal_n(vq-dev, vq, + vhost_add_used_and_signal_n(vq-dev, vq, urgent, vq-heads[nvq-done_idx], add); nvq-done_idx = (nvq-done_idx + add) % UIO_MAXIOV; j -= add; @@ -363,6 +367,7 @@ static void handle_tx(struct vhost_net *net) zcopy = nvq-ubufs; for (;;) { + bool urgent; /* Release DMAs done buffers first */ if (zcopy) vhost_zerocopy_signal_used(net, vq); @@ -374,7 +379,7 @@ static void
[PATCH RFC 2/2] vhost: support urgent descriptors
Signed-off-by: Michael S. Tsirkin m...@redhat.com --- drivers/vhost/vhost.h | 19 +-- drivers/vhost/net.c | 30 +- drivers/vhost/scsi.c | 23 +++ drivers/vhost/test.c | 5 +++-- drivers/vhost/vhost.c | 23 --- 5 files changed, 68 insertions(+), 32 deletions(-) diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 3eda654..61ca542 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -96,6 +96,9 @@ struct vhost_virtqueue { /* Last used index value we have signalled on */ bool signalled_used_valid; + /* Urgent descriptor was used */ + bool urgent; + /* Log writes to used structure. */ bool log_used; u64 log_addr; @@ -138,20 +141,24 @@ long vhost_vring_ioctl(struct vhost_dev *d, int ioctl, void __user *argp); int vhost_vq_access_ok(struct vhost_virtqueue *vq); int vhost_log_access_ok(struct vhost_dev *); -int vhost_get_vq_desc(struct vhost_virtqueue *, +int vhost_get_vq_desc(struct vhost_virtqueue *, bool *urgent, struct iovec iov[], unsigned int iov_count, unsigned int *out_num, unsigned int *in_num, struct vhost_log *log, unsigned int *log_num); void vhost_discard_vq_desc(struct vhost_virtqueue *, int n); int vhost_init_used(struct vhost_virtqueue *); -int vhost_add_used(struct vhost_virtqueue *, unsigned int head, int len); -int vhost_add_used_n(struct vhost_virtqueue *, struct vring_used_elem *heads, -unsigned count); -void vhost_add_used_and_signal(struct vhost_dev *, struct vhost_virtqueue *, +int vhost_add_used(struct vhost_virtqueue *, bool urgent, + unsigned int head, int len); +int vhost_add_used_n(struct vhost_virtqueue *, bool urgent, +struct vring_used_elem *heads, unsigned count); +void vhost_add_used_and_signal(struct vhost_dev *, + struct vhost_virtqueue *, + bool urgent, unsigned int id, int len); void vhost_add_used_and_signal_n(struct vhost_dev *, struct vhost_virtqueue *, - struct vring_used_elem *heads, unsigned count); +bool urgent, +struct vring_used_elem *heads, unsigned count); void vhost_signal(struct vhost_dev *, struct vhost_virtqueue *); void vhost_disable_notify(struct vhost_dev *, struct vhost_virtqueue *); bool vhost_enable_notify(struct vhost_dev *, struct vhost_virtqueue *); diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 8dae2f7..5f0567f 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -48,9 +48,11 @@ MODULE_PARM_DESC(experimental_zcopytx, Enable Zero Copy TX; * status internally; used for zerocopy tx only. */ /* Lower device DMA failed */ -#define VHOST_DMA_FAILED_LEN 3 +#define VHOST_DMA_FAILED_LEN 4 /* Lower device DMA done */ -#define VHOST_DMA_DONE_LEN 2 +#define VHOST_DMA_DONE_LEN 3 +/* Lower device DMA in progress, urgent bit set */ +#define VHOST_DMA_URGENT 2 /* Lower device DMA in progress */ #define VHOST_DMA_IN_PROGRESS 1 /* Buffer unused */ @@ -284,11 +286,13 @@ static void vhost_zerocopy_signal_used(struct vhost_net *net, container_of(vq, struct vhost_net_virtqueue, vq); int i, add; int j = 0; + bool urgent = false; for (i = nvq-done_idx; i != nvq-upend_idx; i = (i + 1) % UIO_MAXIOV) { if (vq-heads[i].len == VHOST_DMA_FAILED_LEN) vhost_net_tx_err(net); if (VHOST_DMA_IS_DONE(vq-heads[i].len)) { + urgent = urgent || vq-heads[i].len == VHOST_DMA_URGENT; vq-heads[i].len = VHOST_DMA_CLEAR_LEN; ++j; } else @@ -296,7 +300,7 @@ static void vhost_zerocopy_signal_used(struct vhost_net *net, } while (j) { add = min(UIO_MAXIOV - nvq-done_idx, j); - vhost_add_used_and_signal_n(vq-dev, vq, + vhost_add_used_and_signal_n(vq-dev, vq, urgent, vq-heads[nvq-done_idx], add); nvq-done_idx = (nvq-done_idx + add) % UIO_MAXIOV; j -= add; @@ -363,6 +367,7 @@ static void handle_tx(struct vhost_net *net) zcopy = nvq-ubufs; for (;;) { + bool urgent; /* Release DMAs done buffers first */ if (zcopy) vhost_zerocopy_signal_used(net, vq); @@ -374,7 +379,7 @@ static void handle_tx(struct vhost_net *net) % UIO_MAXIOV == nvq-done_idx)) break; - head = vhost_get_vq_desc(vq, vq-iov, + head = vhost_get_vq_desc(vq, urgent, vq-iov,