On Thu, Dec 29, 2022 at 11:49 AM Xuan Zhuo <xuanz...@linux.alibaba.com> wrote:
>
> On Thu, 29 Dec 2022 11:22:13 +0800, Jason Wang <jasow...@redhat.com> wrote:
> > On Thu, Dec 29, 2022 at 10:10 AM Xuan Zhuo <xuanz...@linux.alibaba.com>
> > wrote:
> > >
> > > On Wed, 28 Dec 2022 19:41:13 +0800, Jason Wang <jasow...@redhat.com>
> > > wrote:
> > > > On Wed, Dec 28, 2022 at 4:34 PM Xuan Zhuo <xuanz...@linux.alibaba.com>
> > > > wrote:
> > > > >
> > > > > On Tue, 27 Dec 2022 01:58:22 -0500, "Michael S. Tsirkin"
> > > > > <m...@redhat.com> wrote:
> > > > > > On Tue, Dec 27, 2022 at 12:33:53PM +0800, Jason Wang wrote:
> > > > > > > On Tue, Dec 27, 2022 at 10:25 AM Xuan Zhuo
> > > > > > > <xuanz...@linux.alibaba.com> wrote:
> > > > > > > >
> > > > > > > > On Mon, 26 Dec 2022 15:49:08 +0800, Jason Wang
> > > > > > > > <jasow...@redhat.com> wrote:
> > > > > > > > > We used to busy waiting on the cvq command this tends to be
> > > > > > > > > problematic since:
> > > > > > > > >
> > > > > > > > > 1) CPU could wait for ever on a buggy/malicous device
> > > > > > > > > 2) There's no wait to terminate the process that triggers the
> > > > > > > > > cvq
> > > > > > > > > command
> > > > > > > > >
> > > > > > > > > So this patch switch to use virtqueue_wait_for_used() to
> > > > > > > > > sleep with a
> > > > > > > > > timeout (1s) instead of busy polling for the cvq command
> > > > > > > > > forever. This
> > > > > > > >
> > > > > > > > I don't think that a fixed 1S is a good choice.
> > > > > > >
> > > > > > > Well, it could be tweaked to be a little bit longer.
> > > > > > >
> > > > > > > One way, as discussed, is to let the device advertise a timeout
> > > > > > > then
> > > > > > > the driver can validate if it's valid and use that timeout. But it
> > > > > > > needs extension to the spec.
> > > > > >
> > > > > > Controlling timeout from device is a good idea, e.g. hardware
> > > > > > devices
> > > > > > would benefit from a shorter timeout, hypervisor devices from a
> > > > > > longer
> > > > > > timeout or no timeout.
> > > > >
> > > > > Yes. That is good.
> > > > >
> > > > > Before introducing this feature, I personally like to use "wait",
> > > > > rather than
> > > > > define a timeout.
> > > >
> > > > Note that the driver still needs to validate what device advertises to
> > > > avoid infinite wait.
> > >
> > > Sorry, I didn't understand what you mean.
> >
> > I meant the interface needs to carefully designed to
> >
> > 1) avoid device to advertise a infinite (or very long) timeout
> > 2) driver need to have its own max timeout regardless what device advertises
>
>
> I see.
>
> As far as I know, different operations will take different time.
> For example, the queues are initialized one by one when performing
> VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET commands. If the number of queues is large,
> then
> this time will be very long.
I see. This is the case even for the software backends.
>
> So we should set different timeouts for different commands.
Probably but it would result in a very complex interface, the device
can just choose to advertise the maximum timeout of all the commands
in this case. As discussed, I think we can start a very long timeout.
Is 1 minutes sufficient in this case?
Thanks
>
> Thanks.
>
> >
> > Thanks
> >
> > >
> > > Thanks.
> > >
> > > >
> > > > Thanks
> > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > > Some of the DPUs are very
> > > > > > > > lazy for cvq handle.
> > > > > > >
> > > > > > > Such design needs to be revisited, cvq (control path) should have
> > > > > > > a
> > > > > > > better priority or QOS than datapath.
> > > > > >
> > > > > > Spec says nothing about this, so driver can't assume this either.
> > > > > >
> > > > > > > > In particular, we will also directly break the device.
> > > > > > >
> > > > > > > It's kind of hardening for malicious devices.
> > > > > >
> > > > > > ATM no amount of hardening can prevent a malicious hypervisor from
> > > > > > blocking the guest. Recovering when a hardware device is broken
> > > > > > would be
> > > > > > nice but I think if we do bother then we should try harder to
> > > > > > recover,
> > > > > > such as by driving device reset.
> > > > > >
> > > > > >
> > > > > > Also, does your patch break surprise removal? There's no callback
> > > > > > in this case ATM.
> > > > > >
> > > > > > > >
> > > > > > > > I think it is necessary to add a Virtio-Net parameter to allow
> > > > > > > > users to define
> > > > > > > > this timeout by themselves. Although I don't think this is a
> > > > > > > > good way.
> > > > > > >
> > > > > > > Very hard and unfriendly to the end users.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > >
> > > > > > > > > gives the scheduler a breath and can let the process can
> > > > > > > > > respond to
> > > > > > > > > asignal. If the device doesn't respond in the timeout, break
> > > > > > > > > the
> > > > > > > > > device.
> > > > > > > > >
> > > > > > > > > Signed-off-by: Jason Wang <jasow...@redhat.com>
> > > > > > > > > ---
> > > > > > > > > Changes since V1:
> > > > > > > > > - break the device when timeout
> > > > > > > > > - get buffer manually since the virtio core check more_used()
> > > > > > > > > instead
> > > > > > > > > ---
> > > > > > > > > drivers/net/virtio_net.c | 24 ++++++++++++++++--------
> > > > > > > > > 1 file changed, 16 insertions(+), 8 deletions(-)
> > > > > > > > >
> > > > > > > > > diff --git a/drivers/net/virtio_net.c
> > > > > > > > > b/drivers/net/virtio_net.c
> > > > > > > > > index efd9dd55828b..6a2ea64cfcb5 100644
> > > > > > > > > --- a/drivers/net/virtio_net.c
> > > > > > > > > +++ b/drivers/net/virtio_net.c
> > > > > > > > > @@ -405,6 +405,7 @@ static void disable_rx_mode_work(struct
> > > > > > > > > virtnet_info *vi)
> > > > > > > > > vi->rx_mode_work_enabled = false;
> > > > > > > > > spin_unlock_bh(&vi->rx_mode_lock);
> > > > > > > > >
> > > > > > > > > + virtqueue_wake_up(vi->cvq);
> > > > > > > > > flush_work(&vi->rx_mode_work);
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > @@ -1497,6 +1498,11 @@ static bool try_fill_recv(struct
> > > > > > > > > virtnet_info *vi, struct receive_queue *rq,
> > > > > > > > > return !oom;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > +static void virtnet_cvq_done(struct virtqueue *cvq)
> > > > > > > > > +{
> > > > > > > > > + virtqueue_wake_up(cvq);
> > > > > > > > > +}
> > > > > > > > > +
> > > > > > > > > static void skb_recv_done(struct virtqueue *rvq)
> > > > > > > > > {
> > > > > > > > > struct virtnet_info *vi = rvq->vdev->priv;
> > > > > > > > > @@ -1984,6 +1990,8 @@ static int virtnet_tx_resize(struct
> > > > > > > > > virtnet_info *vi,
> > > > > > > > > return err;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > +static int virtnet_close(struct net_device *dev);
> > > > > > > > > +
> > > > > > > > > /*
> > > > > > > > > * Send command via the control virtqueue and check status.
> > > > > > > > > Commands
> > > > > > > > > * supported by the hypervisor, as indicated by feature
> > > > > > > > > bits, should
> > > > > > > > > @@ -2026,14 +2034,14 @@ static bool
> > > > > > > > > virtnet_send_command(struct virtnet_info *vi, u8 class, u8
> > > > > > > > > cmd,
> > > > > > > > > if (unlikely(!virtqueue_kick(vi->cvq)))
> > > > > > > > > return vi->ctrl->status == VIRTIO_NET_OK;
> > > > > > > > >
> > > > > > > > > - /* Spin for a response, the kick causes an ioport
> > > > > > > > > write, trapping
> > > > > > > > > - * into the hypervisor, so the request should be
> > > > > > > > > handled immediately.
> > > > > > > > > - */
> > > > > > > > > - while (!virtqueue_get_buf(vi->cvq, &tmp) &&
> > > > > > > > > - !virtqueue_is_broken(vi->cvq))
> > > > > > > > > - cpu_relax();
> > > > > > > > > + if (virtqueue_wait_for_used(vi->cvq)) {
> > > > > > > > > + virtqueue_get_buf(vi->cvq, &tmp);
> > > > > > > > > + return vi->ctrl->status == VIRTIO_NET_OK;
> > > > > > > > > + }
> > > > > > > > >
> > > > > > > > > - return vi->ctrl->status == VIRTIO_NET_OK;
> > > > > > > > > + netdev_err(vi->dev, "CVQ command timeout, break the
> > > > > > > > > virtio device.");
> > > > > > > > > + virtio_break_device(vi->vdev);
> > > > > > > > > + return VIRTIO_NET_ERR;
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > static int virtnet_set_mac_address(struct net_device *dev,
> > > > > > > > > void *p)
> > > > > > > > > @@ -3526,7 +3534,7 @@ static int virtnet_find_vqs(struct
> > > > > > > > > virtnet_info *vi)
> > > > > > > > >
> > > > > > > > > /* Parameters for control virtqueue, if any */
> > > > > > > > > if (vi->has_cvq) {
> > > > > > > > > - callbacks[total_vqs - 1] = NULL;
> > > > > > > > > + callbacks[total_vqs - 1] = virtnet_cvq_done;
> > > > > > > > > names[total_vqs - 1] = "control";
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > 2.25.1
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > Virtualization mailing list
> > > > > > > > > Virtualization@lists.linux-foundation.org
> > > > > > > > > https://lists.linuxfoundation.org/mailman/listinfo/virtualization
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
