On Fri, Dec 01, 2023 at 01:40:41PM +0300, Arseniy Krasnov wrote:
>
>
> On 01.12.2023 12:48, Stefano Garzarella wrote:
> > On Fri, Dec 01, 2023 at 11:35:56AM +0300, Arseniy Krasnov wrote:
> >>
> >>
> >> On 01.12.2023 11:27, Stefano Garzarella wrote:
> >>> On Thu, Nov 30, 2023 at 12:40:43PM -0500, Michael S. Tsirkin wrote:
> >>>> On Thu, Nov 30, 2023 at 03:11:19PM +0100, Stefano Garzarella wrote:
> >>>>> On Thu, Nov 30, 2023 at 08:58:58AM -0500, Michael S. Tsirkin wrote:
> >>>>> > On Thu, Nov 30, 2023 at 04:43:34PM +0300, Arseniy Krasnov wrote:
> >>>>> > >
> >>>>> > >
> >>>>> > > On 30.11.2023 16:42, Michael S. Tsirkin wrote:
> >>>>> > > > On Thu, Nov 30, 2023 at 04:08:39PM +0300, Arseniy Krasnov wrote:
> >>>>> > > >> Send credit update message when SO_RCVLOWAT is updated and it is
> >>>>> > > >> bigger
> >>>>> > > >> than number of bytes in rx queue. It is needed, because 'poll()'
> >>>>> > > >> will
> >>>>> > > >> wait until number of bytes in rx queue will be not smaller than
> >>>>> > > >> SO_RCVLOWAT, so kick sender to send more data. Otherwise mutual
> >>>>> > > >> hungup
> >>>>> > > >> for tx/rx is possible: sender waits for free space and receiver
> >>>>> > > >> is
> >>>>> > > >> waiting data in 'poll()'.
> >>>>> > > >>
> >>>>> > > >> Signed-off-by: Arseniy Krasnov <avkras...@salutedevices.com>
> >>>>> > > >> ---
> >>>>> > > >> Changelog:
> >>>>> > > >> v1 -> v2:
> >>>>> > > >> * Update commit message by removing 'This patch adds XXX'
> >>>>> > > >>manner.
> >>>>> > > >> * Do not initialize 'send_update' variable - set it directly
> >>>>> > > >>during
> >>>>> > > >> first usage.
> >>>>> > > >> v3 -> v4:
> >>>>> > > >> * Fit comment in 'virtio_transport_notify_set_rcvlowat()' to
> >>>>> > > >>80 chars.
> >>>>> > > >> v4 -> v5:
> >>>>> > > >> * Do not change callbacks order in transport structures.
> >>>>> > > >>
> >>>>> > > >> drivers/vhost/vsock.c | 1 +
> >>>>> > > >> include/linux/virtio_vsock.h | 1 +
> >>>>> > > >> net/vmw_vsock/virtio_transport.c | 1 +
> >>>>> > > >> net/vmw_vsock/virtio_transport_common.c | 27
> >>>>> > > >>+++++++++++++++++++++++++
> >>>>> > > >> net/vmw_vsock/vsock_loopback.c | 1 +
> >>>>> > > >> 5 files changed, 31 insertions(+)
> >>>>> > > >>
> >>>>> > > >> diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
> >>>>> > > >> index f75731396b7e..4146f80db8ac 100644
> >>>>> > > >> --- a/drivers/vhost/vsock.c
> >>>>> > > >> +++ b/drivers/vhost/vsock.c
> >>>>> > > >> @@ -451,6 +451,7 @@ static struct virtio_transport
> >>>>> > > >> vhost_transport = {
> >>>>> > > >> .notify_buffer_size =
> >>>>> > > >>virtio_transport_notify_buffer_size,
> >>>>> > > >>
> >>>>> > > >> .read_skb = virtio_transport_read_skb,
> >>>>> > > >> + .notify_set_rcvlowat =
> >>>>> > > >> virtio_transport_notify_set_rcvlowat
> >>>>> > > >> },
> >>>>> > > >>
> >>>>> > > >> .send_pkt = vhost_transport_send_pkt,
> >>>>> > > >> diff --git a/include/linux/virtio_vsock.h
> >>>>> > > >> b/include/linux/virtio_vsock.h
> >>>>> > > >> index ebb3ce63d64d..c82089dee0c8 100644
> >>>>> > > >> --- a/include/linux/virtio_vsock.h
> >>>>> > > >> +++ b/include/linux/virtio_vsock.h
> >>>>> > > >> @@ -256,4 +256,5 @@ void virtio_transport_put_credit(struct
> >>>>> > > >> virtio_vsock_sock *vvs, u32 credit);
> >>>>> > > >> void virtio_transport_deliver_tap_pkt(struct sk_buff *skb);
> >>>>> > > >> int virtio_transport_purge_skbs(void *vsk, struct sk_buff_head
> >>>>> > > >>*list);
> >>>>> > > >> int virtio_transport_read_skb(struct vsock_sock *vsk,
> >>>>> > > >>skb_read_actor_t read_actor);
> >>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock
> >>>>> > > >> *vsk, int val);
> >>>>> > > >> #endif /* _LINUX_VIRTIO_VSOCK_H */
> >>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport.c
> >>>>> > > >> b/net/vmw_vsock/virtio_transport.c
> >>>>> > > >> index af5bab1acee1..8007593a3a93 100644
> >>>>> > > >> --- a/net/vmw_vsock/virtio_transport.c
> >>>>> > > >> +++ b/net/vmw_vsock/virtio_transport.c
> >>>>> > > >> @@ -539,6 +539,7 @@ static struct virtio_transport
> >>>>> > > >> virtio_transport = {
> >>>>> > > >> .notify_buffer_size =
> >>>>> > > >>virtio_transport_notify_buffer_size,
> >>>>> > > >>
> >>>>> > > >> .read_skb = virtio_transport_read_skb,
> >>>>> > > >> + .notify_set_rcvlowat =
> >>>>> > > >> virtio_transport_notify_set_rcvlowat
> >>>>> > > >> },
> >>>>> > > >>
> >>>>> > > >> .send_pkt = virtio_transport_send_pkt,
> >>>>> > > >> diff --git a/net/vmw_vsock/virtio_transport_common.c
> >>>>> > > >> b/net/vmw_vsock/virtio_transport_common.c
> >>>>> > > >> index f6dc896bf44c..1cb556ad4597 100644
> >>>>> > > >> --- a/net/vmw_vsock/virtio_transport_common.c
> >>>>> > > >> +++ b/net/vmw_vsock/virtio_transport_common.c
> >>>>> > > >> @@ -1684,6 +1684,33 @@ int virtio_transport_read_skb(struct
> >>>>> > > >> vsock_sock *vsk, skb_read_actor_t recv_acto
> >>>>> > > >> }
> >>>>> > > >> EXPORT_SYMBOL_GPL(virtio_transport_read_skb);
> >>>>> > > >>
> >>>>> > > >> +int virtio_transport_notify_set_rcvlowat(struct vsock_sock *vsk,
> >>>>> > > >> int val)
> >>>>> > > >> +{
> >>>>> > > >> + struct virtio_vsock_sock *vvs = vsk->trans;
> >>>>> > > >> + bool send_update;
> >>>>> > > >> +
> >>>>> > > >> + spin_lock_bh(&vvs->rx_lock);
> >>>>> > > >> +
> >>>>> > > >> + /* If number of available bytes is less than new
> >>>>> > > >> SO_RCVLOWAT value,
> >>>>> > > >> + * kick sender to send more data, because sender may sleep
> >>>>> > > >> in
> >>>>> > > >> its
> >>>>> > > >> + * 'send()' syscall waiting for enough space at our side.
> >>>>> > > >> + */
> >>>>> > > >> + send_update = vvs->rx_bytes < val;
> >>>>> > > >> +
> >>>>> > > >> + spin_unlock_bh(&vvs->rx_lock);
> >>>>> > > >> +
> >>>>> > > >> + if (send_update) {
> >>>>> > > >> + int err;
> >>>>> > > >> +
> >>>>> > > >> + err = virtio_transport_send_credit_update(vsk);
> >>>>> > > >> + if (err < 0)
> >>>>> > > >> + return err;
> >>>>> > > >> + }
> >>>>> > > >> +
> >>>>> > > >> + return 0;
> >>>>> > > >> +}
> >>>>> > > >
> >>>>> > > >
> >>>>> > > > I find it strange that this will send a credit update
> >>>>> > > > even if nothing changed since this was called previously.
> >>>>> > > > I'm not sure whether this is a problem protocol-wise,
> >>>>> > > > but it certainly was not envisioned when the protocol was
> >>>>> > > > built. WDYT?
> >>>>> > >
> >>>>> > > >From virtio spec I found:
> >>>>> > >
> >>>>> > > It is also valid to send a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet
> >>>>> > > without previously receiving a
> >>>>> > > VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows communicating
> >>>>> > > updates any time a change
> >>>>> > > in buffer space occurs.
> >>>>> > > So I guess there is no limitations to send such type of packet,
> >>>>> > > e.g. it is not
> >>>>> > > required to be a reply for some another packet. Please, correct me
> >>>>> > > if im wrong.
> >>>>> > >
> >>>>> > > Thanks, Arseniy
> >>>>> >
> >>>>> >
> >>>>> > Absolutely. My point was different - with this patch it is possible
> >>>>> > that you are not adding any credits at all since the previous
> >>>>> > VIRTIO_VSOCK_OP_CREDIT_UPDATE.
> >>>>>
> >>>>> I think the problem we're solving here is that since as an optimization
> >>>>> we
> >>>>> avoid sending the update for every byte we consume, but we put a
> >>>>> threshold,
> >>>>> then we make sure we update the peer.
> >>>>>
> >>>>> A credit update contains a snapshot and sending it the same as the
> >>>>> previous
> >>>>> one should not create any problem.
> >>>>
> >>>> Well it consumes a buffer on the other side.
> >>>
> >>> Sure, but we are already speculating by not updating the other side when
> >>> we consume bytes before a certain threshold. This already avoids to
> >>> consume many buffers.
> >>>
> >>> Here we're only sending it once, when the user sets RCVLOWAT, so
> >>> basically I expect it won't affect performance.
> >>
> >> Moreover I think in practice setting RCVLOWAT is rare case, while this
> >> patch
> >> fixes real problem I guess
> >>
> >>
> >>>
> >>>>
> >>>>> My doubt now is that we only do this when we set RCVLOWAT , should we
> >>>>> also
> >>>>> do something when we consume bytes to avoid the optimization we have?
> >>>>>
> >>>>> Stefano
> >>>>
> >>>> Isn't this why we have credit request?
> >>>
> >>> Yep, but in practice we never use it. It would also consume 2 buffers,
> >>> one at the transmitter and one at the receiver.
> >>>
> >>> However I agree that maybe we should start using it before we decide not
> >>> to send any more data.
> >>>
> >>> To be compatible with older devices, though, I think for now we also
> >>> need to send a credit update when the bytes in the receive queue are
> >>> less than RCVLOWAT, as Arseniy proposed in the other series.
> >>
> >> Looks like (in theory of course), that credit request is considered to be
> >> paired with credit update. While current usage of credit update is
> >> something
> >> like ACK packet in TCP, e.g. telling peer that we are ready to receive more
> >> data.
> >
> > I don't honestly know what the original author's choice was, but I think we
> > reduce latency this way.
>
> Ah I see,ok
>
> >
> > Effectively though, if we never send any credit update when we consume
> > bytes and always leave it up to the transmitter to ask for an update before
> > transmission, we save even more buffer than the optimization we have, but
> > maybe the latency would grow a lot.
>
> I think:
> 1) Way where sender must request current credit status before sending packet
> requires rework of kernel part, and for me this approach is not
> so clear than current simple implementation (send RW, reply with
> CREDIT_UPDATE if needed).
> 2) In theory yes, we need one more buffer for such CREDIT_UPDATE, but in
> practice I don't know how big is this trouble.
>
> Thanks, Arseniy
I just worry that yes, normal users will only call RCVLOWAT once,
but a bad user might call it many times causing a ton of
credit updates. This is why I feel it's prudent to at least
keep track of last credit update and if nothing changed
do not resend it on a repeated RCVLOWAT.
> >
> > Stefano
> >
