Re: [PATCH v23 2/2] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ

[Wei Wang]

On 01/24/2018 01:01 PM, Michael S. Tsirkin wrote:

On Wed, Jan 24, 2018 at 10:50:27AM +0800, Wei Wang wrote:
This will not DTRT in all cases. It's quite possible
that host does not need the kick when ring is half full but
does need it later when ring is full.
You can kick at ring half full as optimization but you absolutely
still must kick on ring full. Something like:

if (vq->num_free == virtqueue_get_vring_size(vq) / 2 ||
vq->num_free <= 2)


Right. Would "if (vq->num_free < virtqueue_get_vring_size(vq) / 2" be 
better?



Best,
Wei
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH v23 2/2] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ

[Michael S. Tsirkin]

On Wed, Jan 24, 2018 at 10:50:27AM +0800, Wei Wang wrote:
> Negotiation of the VIRTIO_BALLOON_F_FREE_PAGE_VQ feature indicates the
> support of reporting hints of guest free pages to host via virtio-balloon.
> 
> Host requests the guest to report free pages by sending a new cmd
> id to the guest via the free_page_report_cmd_id configuration register.
> 
> When the guest starts to report, the first element added to the free page
> vq is the cmd id given by host. When the guest finishes the reporting
> of all the free pages, VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_ID is added
> to the vq to tell host that the reporting is done. Host may also requests
> the guest to stop the reporting in advance by sending the stop cmd id to
> the guest via the configuration register.
> 
> Signed-off-by: Wei Wang 
> Signed-off-by: Liang Li 
> Cc: Michael S. Tsirkin 
> Cc: Michal Hocko 
> ---
>  drivers/virtio/virtio_balloon.c | 228 
> ++--
>  include/uapi/linux/virtio_balloon.h |   6 +
>  2 files changed, 201 insertions(+), 33 deletions(-)
> 
> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
> index a1fb52c..d038f4a 100644
> --- a/drivers/virtio/virtio_balloon.c
> +++ b/drivers/virtio/virtio_balloon.c
> @@ -51,9 +51,21 @@ MODULE_PARM_DESC(oom_pages, "pages to free on OOM");
>  static struct vfsmount *balloon_mnt;
>  #endif
>  
> +/* The number of virtqueues supported by virtio-balloon */
> +#define VIRTIO_BALLOON_VQ_NUM4
> +#define VIRTIO_BALLOON_VQ_ID_INFLATE 0
> +#define VIRTIO_BALLOON_VQ_ID_DEFLATE 1
> +#define VIRTIO_BALLOON_VQ_ID_STATS   2
> +#define VIRTIO_BALLOON_VQ_ID_FREE_PAGE   3
> +
>  struct virtio_balloon {
>   struct virtio_device *vdev;
> - struct virtqueue *inflate_vq, *deflate_vq, *stats_vq;
> + struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq;
> +
> + /* Balloon's own wq for cpu-intensive work items */
> + struct workqueue_struct *balloon_wq;
> + /* The free page reporting work item submitted to the balloon wq */
> + struct work_struct report_free_page_work;
>  
>   /* The balloon servicing is delegated to a freezable workqueue. */
>   struct work_struct update_balloon_stats_work;
> @@ -63,6 +75,13 @@ struct virtio_balloon {
>   spinlock_t stop_update_lock;
>   bool stop_update;
>  
> + /* Start to report free pages */
> + bool report_free_page;
> + /* Stores the cmd id given by host to start the free page reporting */
> + __virtio32 start_cmd_id;
> + /* Stores STOP_ID as a sign to tell host that the reporting is done */
> + __virtio32 stop_cmd_id;
> +
>   /* Waiting for host to ack the pages we released. */
>   wait_queue_head_t acked;
>  
> @@ -281,6 +300,56 @@ static unsigned int update_balloon_stats(struct 
> virtio_balloon *vb)
>   return idx;
>  }
>  
> +static void add_one_sg(struct virtqueue *vq, unsigned long pfn, uint32_t len)
> +{
> + struct scatterlist sg;
> + unsigned int unused;
> + int err;
> +
> + sg_init_table(, 1);
> + sg_set_page(, pfn_to_page(pfn), len, 0);
> +
> + /* Detach all the used buffers from the vq */
> + while (virtqueue_get_buf(vq, ))
> + ;
> +
> + /*
> +  * Since this is an optimization feature, losing a couple of free
> +  * pages to report isn't important. We simply return without adding
> +  * the page if the vq is full.
> +  * We are adding one entry each time, which essentially results in no
> +  * memory allocation, so the GFP_KERNEL flag below can be ignored.
> +  * There is always one entry reserved for the cmd id to use.
> +  */
> + if (vq->num_free > 1) {
> + err = virtqueue_add_inbuf(vq, , 1, vq, GFP_KERNEL);
> + /*
> +  * This is expected to never fail, because there is always an
> +  * entry available on the vq.
> +  */
> + BUG_ON(err);
> + }
> +
> + if (vq->num_free == virtqueue_get_vring_size(vq) / 2)
> + virtqueue_kick(vq);

This will not DTRT in all cases. It's quite possible
that host does not need the kick when ring is half full but
does need it later when ring is full.
You can kick at ring half full as optimization but you absolutely
still must kick on ring full. Something like:

if (vq->num_free == virtqueue_get_vring_size(vq) / 2 ||
vq->num_free <= 2)

> +}
> +
> +static void send_cmd_id(struct virtqueue *vq, __virtio32 *cmd_id)
> +{
> + struct scatterlist sg;
> + int err;
> +
> + sg_init_one(, cmd_id, sizeof(*cmd_id));
> +
> + err = virtqueue_add_outbuf(vq, , 1, vq, GFP_KERNEL);
> + /*
> +  * This is expected to never fail, because there is always an
> +  * entry reserved for the cmd id.
> +  */
> + BUG_ON(err);
> + virtqueue_kick(vq);

Actually add can fail if device becomes broken. 

Re: [virtio-dev] Re: [PATCH v22 2/3] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ

[Michael S. Tsirkin]

On Wed, Jan 24, 2018 at 11:18:40AM +0800, Wei Wang wrote:
> On 01/22/2018 07:25 PM, Wei Wang wrote:
> > On 01/19/2018 08:39 PM, Michael S. Tsirkin wrote:
> > > On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote:
> > > > On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote:
> > > > > On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote:
> > > > > 
> > > > > > +vb->start_cmd_id = cmd_id;
> > > > > > +queue_work(vb->balloon_wq, >report_free_page_work);
> > > > > It seems that if a command was already queued (with a different id),
> > > > > this will result in new command id being sent to host twice,
> > > > > which will
> > > > > likely confuse the host.
> > > > I think that case won't happen, because
> > > > - the host sends a cmd id to the guest via the config, while the
> > > > guest acks
> > > > back the received cmd id via the virtqueue;
> > > > - the guest ack back a cmd id only when a new cmd id is received
> > > > from the
> > > > host, that is the above check:
> > > > 
> > > >  if (cmd_id != vb->start_cmd_id) { --> the driver only queues the
> > > > reporting work only when a new cmd id is received
> > > >  /*
> > > >   * Host requests to start the reporting
> > > > by sending a
> > > >   * new cmd id.
> > > >   */
> > > >  WRITE_ONCE(vb->report_free_page, true);
> > > >  vb->start_cmd_id = cmd_id;
> > > >  queue_work(vb->balloon_wq,
> > > > >report_free_page_work);
> > > >  }
> > > > 
> > > > So the same cmd id wouldn't queue the reporting work twice.
> > > > 
> > > Like this:
> > > 
> > > vb->start_cmd_id = cmd_id;
> > > queue_work(vb->balloon_wq, >report_free_page_work);
> > > 
> > > command id changes
> > > 
> > > vb->start_cmd_id = cmd_id;
> > > 
> > > work executes
> > > 
> > > queue_work(vb->balloon_wq, >report_free_page_work);
> > > 
> > > work executes again
> > > 
> > 
> > If we think about the whole working flow, I think this case couldn't
> > happen:
> > 
> > 1) device send cmd_id=1 to driver;
> > 2) driver receives cmd_id=1 in the config and acks cmd_id=1 to the
> > device via the vq;
> > 3) device revives cmd_id=1;
> > 4) device wants to stop the reporting by sending cmd_id=STOP;
> > 5) driver receives cmd_id=STOP from the config, and acks cmd_id=STOP to
> > the device via the vq;
> > 6) device sends cmd_id=2 to driver;
> > ...
> > 
> > cmd_id=2 won't come after cmd_id=1, there will be a STOP cmd in between
> > them (STOP won't queue the work).
> > 
> > How about defining the correct device behavior in the spec:
> > The device Should NOT send a second cmd id to the driver until a STOP
> > cmd ack for the previous cmd id has been received from the guest.
> 
> 
> Thanks for the comments, and I adopted most of them in the new posted v23
> patches. The above discussion is the one that I haven't included. If you
> could still see issues in the above analysis, please let me know. Thanks.
> 
> Best,
> Wei
> 
> 
>

Yes, I think you should just fix the race in the driver.

-- 
MST
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [virtio-dev] Re: [PATCH v22 2/3] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ

[Michael S. Tsirkin]

On Mon, Jan 22, 2018 at 07:25:45PM +0800, Wei Wang wrote:
> On 01/19/2018 08:39 PM, Michael S. Tsirkin wrote:
> > On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote:
> > > On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote:
> > > > On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote:
> > > > 
> > > > > + vb->start_cmd_id = cmd_id;
> > > > > + queue_work(vb->balloon_wq, >report_free_page_work);
> > > > It seems that if a command was already queued (with a different id),
> > > > this will result in new command id being sent to host twice, which will
> > > > likely confuse the host.
> > > I think that case won't happen, because
> > > - the host sends a cmd id to the guest via the config, while the guest 
> > > acks
> > > back the received cmd id via the virtqueue;
> > > - the guest ack back a cmd id only when a new cmd id is received from the
> > > host, that is the above check:
> > > 
> > >  if (cmd_id != vb->start_cmd_id) { --> the driver only queues the
> > > reporting work only when a new cmd id is received
> > >  /*
> > >   * Host requests to start the reporting by 
> > > sending a
> > >   * new cmd id.
> > >   */
> > >  WRITE_ONCE(vb->report_free_page, true);
> > >  vb->start_cmd_id = cmd_id;
> > >  queue_work(vb->balloon_wq,
> > > >report_free_page_work);
> > >  }
> > > 
> > > So the same cmd id wouldn't queue the reporting work twice.
> > > 
> > Like this:
> > 
> > vb->start_cmd_id = cmd_id;
> > queue_work(vb->balloon_wq, >report_free_page_work);
> > 
> > command id changes
> > 
> > vb->start_cmd_id = cmd_id;
> > 
> > work executes
> > 
> > queue_work(vb->balloon_wq, >report_free_page_work);
> > 
> > work executes again
> > 
> 
> If we think about the whole working flow, I think this case couldn't happen:
> 
> 1) device send cmd_id=1 to driver;
> 2) driver receives cmd_id=1 in the config and acks cmd_id=1 to the device
> via the vq;
> 3) device revives cmd_id=1;
> 4) device wants to stop the reporting by sending cmd_id=STOP;
> 5) driver receives cmd_id=STOP from the config, and acks cmd_id=STOP to the
> device via the vq;
> 6) device sends cmd_id=2 to driver;
> ...
> 
> cmd_id=2 won't come after cmd_id=1, there will be a STOP cmd in between them
> (STOP won't queue the work).
> 
> How about defining the correct device behavior in the spec:
> The device Should NOT send a second cmd id to the driver until a STOP cmd
> ack for the previous cmd id has been received from the guest.
> 
> 
> Best,
> Wei

I think we should just fix races in the driver rather than introduce
random restrictions in the device.

If device wants to start a new sequence, it should be able to
do just that without a complicated back and forth with several
roundtrips through the driver.

-- 
MST
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [virtio-dev] Re: [PATCH v22 2/3] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ

[Wei Wang]

On 01/22/2018 07:25 PM, Wei Wang wrote:

On 01/19/2018 08:39 PM, Michael S. Tsirkin wrote:

On Fri, Jan 19, 2018 at 11:44:21AM +0800, Wei Wang wrote:

On 01/18/2018 12:44 AM, Michael S. Tsirkin wrote:

On Wed, Jan 17, 2018 at 01:10:11PM +0800, Wei Wang wrote:


+vb->start_cmd_id = cmd_id;
+queue_work(vb->balloon_wq, >report_free_page_work);

It seems that if a command was already queued (with a different id),
this will result in new command id being sent to host twice, which 
will

likely confuse the host.

I think that case won't happen, because
- the host sends a cmd id to the guest via the config, while the 
guest acks

back the received cmd id via the virtqueue;
- the guest ack back a cmd id only when a new cmd id is received 
from the

host, that is the above check:

 if (cmd_id != vb->start_cmd_id) { --> the driver only queues the
reporting work only when a new cmd id is received
 /*
  * Host requests to start the reporting by 
sending a

  * new cmd id.
  */
 WRITE_ONCE(vb->report_free_page, true);
 vb->start_cmd_id = cmd_id;
 queue_work(vb->balloon_wq,
>report_free_page_work);
 }

So the same cmd id wouldn't queue the reporting work twice.


Like this:

vb->start_cmd_id = cmd_id;
queue_work(vb->balloon_wq, >report_free_page_work);

command id changes

vb->start_cmd_id = cmd_id;

work executes

queue_work(vb->balloon_wq, >report_free_page_work);

work executes again



If we think about the whole working flow, I think this case couldn't 
happen:


1) device send cmd_id=1 to driver;
2) driver receives cmd_id=1 in the config and acks cmd_id=1 to the 
device via the vq;

3) device revives cmd_id=1;
4) device wants to stop the reporting by sending cmd_id=STOP;
5) driver receives cmd_id=STOP from the config, and acks cmd_id=STOP 
to the device via the vq;

6) device sends cmd_id=2 to driver;
...

cmd_id=2 won't come after cmd_id=1, there will be a STOP cmd in 
between them (STOP won't queue the work).


How about defining the correct device behavior in the spec:
The device Should NOT send a second cmd id to the driver until a STOP 
cmd ack for the previous cmd id has been received from the guest.



Thanks for the comments, and I adopted most of them in the new posted 
v23 patches. The above discussion is the one that I haven't included. If 
you could still see issues in the above analysis, please let me know. 
Thanks.


Best,
Wei




___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH v23 2/2] virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ

[Wei Wang]

Negotiation of the VIRTIO_BALLOON_F_FREE_PAGE_VQ feature indicates the
support of reporting hints of guest free pages to host via virtio-balloon.

Host requests the guest to report free pages by sending a new cmd
id to the guest via the free_page_report_cmd_id configuration register.

When the guest starts to report, the first element added to the free page
vq is the cmd id given by host. When the guest finishes the reporting
of all the free pages, VIRTIO_BALLOON_FREE_PAGE_REPORT_STOP_ID is added
to the vq to tell host that the reporting is done. Host may also requests
the guest to stop the reporting in advance by sending the stop cmd id to
the guest via the configuration register.

Signed-off-by: Wei Wang 
Signed-off-by: Liang Li 
Cc: Michael S. Tsirkin 
Cc: Michal Hocko 
---
 drivers/virtio/virtio_balloon.c | 228 ++--
 include/uapi/linux/virtio_balloon.h |   6 +
 2 files changed, 201 insertions(+), 33 deletions(-)

diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index a1fb52c..d038f4a 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -51,9 +51,21 @@ MODULE_PARM_DESC(oom_pages, "pages to free on OOM");
 static struct vfsmount *balloon_mnt;
 #endif
 
+/* The number of virtqueues supported by virtio-balloon */
+#define VIRTIO_BALLOON_VQ_NUM  4
+#define VIRTIO_BALLOON_VQ_ID_INFLATE   0
+#define VIRTIO_BALLOON_VQ_ID_DEFLATE   1
+#define VIRTIO_BALLOON_VQ_ID_STATS 2
+#define VIRTIO_BALLOON_VQ_ID_FREE_PAGE 3
+
 struct virtio_balloon {
struct virtio_device *vdev;
-   struct virtqueue *inflate_vq, *deflate_vq, *stats_vq;
+   struct virtqueue *inflate_vq, *deflate_vq, *stats_vq, *free_page_vq;
+
+   /* Balloon's own wq for cpu-intensive work items */
+   struct workqueue_struct *balloon_wq;
+   /* The free page reporting work item submitted to the balloon wq */
+   struct work_struct report_free_page_work;
 
/* The balloon servicing is delegated to a freezable workqueue. */
struct work_struct update_balloon_stats_work;
@@ -63,6 +75,13 @@ struct virtio_balloon {
spinlock_t stop_update_lock;
bool stop_update;
 
+   /* Start to report free pages */
+   bool report_free_page;
+   /* Stores the cmd id given by host to start the free page reporting */
+   __virtio32 start_cmd_id;
+   /* Stores STOP_ID as a sign to tell host that the reporting is done */
+   __virtio32 stop_cmd_id;
+
/* Waiting for host to ack the pages we released. */
wait_queue_head_t acked;
 
@@ -281,6 +300,56 @@ static unsigned int update_balloon_stats(struct 
virtio_balloon *vb)
return idx;
 }
 
+static void add_one_sg(struct virtqueue *vq, unsigned long pfn, uint32_t len)
+{
+   struct scatterlist sg;
+   unsigned int unused;
+   int err;
+
+   sg_init_table(, 1);
+   sg_set_page(, pfn_to_page(pfn), len, 0);
+
+   /* Detach all the used buffers from the vq */
+   while (virtqueue_get_buf(vq, ))
+   ;
+
+   /*
+* Since this is an optimization feature, losing a couple of free
+* pages to report isn't important. We simply return without adding
+* the page if the vq is full.
+* We are adding one entry each time, which essentially results in no
+* memory allocation, so the GFP_KERNEL flag below can be ignored.
+* There is always one entry reserved for the cmd id to use.
+*/
+   if (vq->num_free > 1) {
+   err = virtqueue_add_inbuf(vq, , 1, vq, GFP_KERNEL);
+   /*
+* This is expected to never fail, because there is always an
+* entry available on the vq.
+*/
+   BUG_ON(err);
+   }
+
+   if (vq->num_free == virtqueue_get_vring_size(vq) / 2)
+   virtqueue_kick(vq);
+}
+
+static void send_cmd_id(struct virtqueue *vq, __virtio32 *cmd_id)
+{
+   struct scatterlist sg;
+   int err;
+
+   sg_init_one(, cmd_id, sizeof(*cmd_id));
+
+   err = virtqueue_add_outbuf(vq, , 1, vq, GFP_KERNEL);
+   /*
+* This is expected to never fail, because there is always an
+* entry reserved for the cmd id.
+*/
+   BUG_ON(err);
+   virtqueue_kick(vq);
+}
+
 /*
  * While most virtqueues communicate guest-initiated requests to the 
hypervisor,
  * the stats queue operates in reverse.  The driver initializes the virtqueue
@@ -316,17 +385,6 @@ static void stats_handle_request(struct virtio_balloon *vb)
virtqueue_kick(vq);
 }
 
-static void virtballoon_changed(struct virtio_device *vdev)
-{
-   struct virtio_balloon *vb = vdev->priv;
-   unsigned long flags;
-
-   spin_lock_irqsave(>stop_update_lock, flags);
-   if (!vb->stop_update)
-   queue_work(system_freezable_wq, 

[PATCH v23 0/2] Virtio-balloon: support free page reporting

[Wei Wang]

This patch series is separated from the previous "Virtio-balloon
Enhancement" series. The new feature, VIRTIO_BALLOON_F_FREE_PAGE_VQ,  
implemented by this series enables the virtio-balloon driver to report
hints of guest free pages to the host. It can be used to accelerate live
migration of VMs. Here is an introduction of this usage:

Live migration needs to transfer the VM's memory from the source machine
to the destination round by round. For the 1st round, all the VM's memory
is transferred. From the 2nd round, only the pieces of memory that were
written by the guest (after the 1st round) are transferred. One method
that is popularly used by the hypervisor to track which part of memory is
written is to write-protect all the guest memory.

The second feature enables the optimization of the 1st round memory
transfer - the hypervisor can skip the transfer of guest free pages in the
1st round. It is not concerned that the memory pages are used after they
are given to the hypervisor as a hint of the free pages, because they will
be tracked by the hypervisor and transferred in the next round if they are
used and written.

ChangeLog:
v22->v23:
- change to kick the device when the vq is half-way full;
- open-code batch_free_page_sg into add_one_sg;
- change cmd_id from "uint32_t" to "__virtio32";
- reserver one entry in the vq for teh driver to send cmd_id, instead
  of busywaiting for an available entry;
- add "stop_update" check before queue_work for prudence purpose for
  now, will have a separate patch to discuss this flag check later;
- init_vqs: change to put some variables on stack to have simpler
  implementation;
- add destroy_workqueue(vb->balloon_wq);

v21->v22:
- add_one_sg: some code and comment re-arrangement
- send_cmd_id: handle a cornercase

For previous ChangeLog, please reference
https://lwn.net/Articles/743660/

Wei Wang (2):
  mm: support reporting free page blocks
  virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_VQ

 drivers/virtio/virtio_balloon.c | 228 ++--
 include/linux/mm.h  |   6 +
 include/uapi/linux/virtio_balloon.h |   6 +
 mm/page_alloc.c |  91 ++
 4 files changed, 298 insertions(+), 33 deletions(-)

-- 
2.7.4

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH v23 1/2] mm: support reporting free page blocks

[Wei Wang]

This patch adds support to walk through the free page blocks in the
system and report them via a callback function. Some page blocks may
leave the free list after zone->lock is released, so it is the caller's
responsibility to either detect or prevent the use of such pages.

One use example of this patch is to accelerate live migration by skipping
the transfer of free pages reported from the guest. A popular method used
by the hypervisor to track which part of memory is written during live
migration is to write-protect all the guest memory. So, those pages that
are reported as free pages but are written after the report function
returns will be captured by the hypervisor, and they will be added to the
next round of memory transfer.

Signed-off-by: Wei Wang 
Signed-off-by: Liang Li 
Cc: Michal Hocko 
Cc: Michael S. Tsirkin 
Acked-by: Michal Hocko 
---
 include/linux/mm.h |  6 
 mm/page_alloc.c| 91 ++
 2 files changed, 97 insertions(+)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index ea818ff..b3077dd 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1938,6 +1938,12 @@ extern void free_area_init_node(int nid, unsigned long * 
zones_size,
unsigned long zone_start_pfn, unsigned long *zholes_size);
 extern void free_initmem(void);
 
+extern void walk_free_mem_block(void *opaque,
+   int min_order,
+   bool (*report_pfn_range)(void *opaque,
+unsigned long pfn,
+unsigned long num));
+
 /*
  * Free reserved pages within range [PAGE_ALIGN(start), end & PAGE_MASK)
  * into the buddy system. The freed pages will be poisoned with pattern
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 76c9688..705de22 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4899,6 +4899,97 @@ void show_free_areas(unsigned int filter, nodemask_t 
*nodemask)
show_swap_cache_info();
 }
 
+/*
+ * Walk through a free page list and report the found pfn range via the
+ * callback.
+ *
+ * Return false if the callback requests to stop reporting. Otherwise,
+ * return true.
+ */
+static bool walk_free_page_list(void *opaque,
+   struct zone *zone,
+   int order,
+   enum migratetype mt,
+   bool (*report_pfn_range)(void *,
+unsigned long,
+unsigned long))
+{
+   struct page *page;
+   struct list_head *list;
+   unsigned long pfn, flags;
+   bool ret;
+
+   spin_lock_irqsave(>lock, flags);
+   list = >free_area[order].free_list[mt];
+   list_for_each_entry(page, list, lru) {
+   pfn = page_to_pfn(page);
+   ret = report_pfn_range(opaque, pfn, 1 << order);
+   if (!ret)
+   break;
+   }
+   spin_unlock_irqrestore(>lock, flags);
+
+   return ret;
+}
+
+/**
+ * walk_free_mem_block - Walk through the free page blocks in the system
+ * @opaque: the context passed from the caller
+ * @min_order: the minimum order of free lists to check
+ * @report_pfn_range: the callback to report the pfn range of the free pages
+ *
+ * If the callback returns false, stop iterating the list of free page blocks.
+ * Otherwise, continue to report.
+ *
+ * Please note that there are no locking guarantees for the callback and
+ * that the reported pfn range might be freed or disappear after the
+ * callback returns so the caller has to be very careful how it is used.
+ *
+ * The callback itself must not sleep or perform any operations which would
+ * require any memory allocations directly (not even GFP_NOWAIT/GFP_ATOMIC)
+ * or via any lock dependency. It is generally advisable to implement
+ * the callback as simple as possible and defer any heavy lifting to a
+ * different context.
+ *
+ * There is no guarantee that each free range will be reported only once
+ * during one walk_free_mem_block invocation.
+ *
+ * pfn_to_page on the given range is strongly discouraged and if there is
+ * an absolute need for that make sure to contact MM people to discuss
+ * potential problems.
+ *
+ * The function itself might sleep so it cannot be called from atomic
+ * contexts.
+ *
+ * In general low orders tend to be very volatile and so it makes more
+ * sense to query larger ones first for various optimizations which like
+ * ballooning etc... This will reduce the overhead as well.
+ */
+void walk_free_mem_block(void *opaque,
+int min_order,
+bool (*report_pfn_range)(void *opaque,
+ unsigned long pfn,

Re: [virtio-dev] [RFC PATCH net-next v2 1/2] virtio_net: Introduce VIRTIO_NET_F_BACKUP feature bit

[Michael S. Tsirkin]

On Mon, Jan 22, 2018 at 07:36:32PM -0800, Alexander Duyck wrote:
> On Mon, Jan 22, 2018 at 6:04 PM, Michael S. Tsirkin  wrote:
> > On Mon, Jan 22, 2018 at 05:34:37PM -0800, Samudrala, Sridhar wrote:
> >> On 1/22/2018 4:05 PM, Michael S. Tsirkin wrote:
> >> > On Mon, Jan 22, 2018 at 03:27:40PM -0800, Samudrala, Sridhar wrote:
> >> > > > > You could probably
> >> > > > > even handle the Tx queue selection via a simple eBPF program and 
> >> > > > > map
> >> > > > > since the input for whatever is used to select Tx should be pretty
> >> > > > > simple, destination MAC, source NUMA node, etc, and the data-set
> >> > > > > shouldn't be too large.
> >> > > > That sounds interesting. A separate device might make this kind of 
> >> > > > setup
> >> > > > a bit easier.  Sridhar, did you look into creating a separate device 
> >> > > > for
> >> > > > the virtual bond device at all?  It does not have to be in a separate
> >> > > > module, that kind of refactoring can come later, but once we commit 
> >> > > > to
> >> > > > using the same single device as virtio, we can't change that.
> >> > > No. I haven't looked into creating a separate device. If we are going 
> >> > > to
> >> > > create a new
> >> > > device, i guess it has to be of a new device type with its own driver.
> >> > Well not necessarily - just a separate netdev ops.
> >> > Kind of like e.g. vlans share a driver with the main driver.
> >>
> >> Not sure what you meant by vlans sharing a driver with the main driver.
> >> IIUC, vlans are supported via 802.1q driver and  creates a netdev of type
> >> 'vlan'
> >> with vlan_netdev_ops
> >
> > But nothing prevents a single module from registering
> > multiple ops.
> 
> Just to clarify, it seems like what you are suggesting is just adding
> the "master" as a separate set of netdev ops or netdevice and to have
> virtio spawn two network devices, one slave and one master, if the
> BACKUP bit is set. Do I have that right?

Yes, that was my idea.

> I am good with the code still living in the virtio driver and
> consolidation with other similar implementations and further
> improvement could probably happen later as part of some refactor.
> 
> Thanks.
> 
> - Alex

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [RFC PATCH net-next v2 2/2] virtio_net: Extend virtio to use VF datapath when available

[Michael S. Tsirkin]

On Tue, Jan 23, 2018 at 12:24:47PM -0800, Siwei Liu wrote:
> On Mon, Jan 22, 2018 at 1:41 PM, Michael S. Tsirkin  wrote:
> > On Mon, Jan 22, 2018 at 12:27:14PM -0800, Siwei Liu wrote:
> >> First off, as mentioned in another thread, the model of stacking up
> >> virt-bond functionality over virtio seems a wrong direction to me.
> >> Essentially the migration process would need to carry over all guest
> >> side configurations previously done on the VF/PT and get them moved to
> >> the new device being it virtio or VF/PT.
> >
> > I might be wrong but I don't see why we should worry about this usecase.
> > Whoever has a bond configured already has working config for migration.
> > We are trying to help people who don't, not convert existig users.
> 
> That has been placed in the view of cloud providers that the imported
> images from the store must be able to run unmodified thus no
> additional setup script is allowed (just as Stephen mentioned in
> another mail). Cloud users don't care about live migration themselves
> but the providers are required to implement such automation mechanism
> to make this process transparent if at all possible. The user does not
> care about the device underneath being VF or not, but they do care
> about consistency all across and the resulting performance
> acceleration in making VF the prefered datapath. It is not quite
> peculiar user cases but IMHO *any* approach proposed for live
> migration should be able to persist the state including network config
> e.g. as simple as MTU. Actually this requirement has nothing to do
> with virtio but our target users are live migration agnostic, being it
> tracking DMA through dirty pages, using virtio as the helper, or
> whatsoever, the goal of persisting configs across remains same.

So the patching being discussed here will mostly do exactly that if your
original config was simply a single virtio net device.


What kind of configs do your users have right now?


> >
> >> Without the help of a new
> >> upper layer bond driver that enslaves virtio and VF/PT devices
> >> underneath, virtio will be overloaded with too much specifics being a
> >> VF/PT backup in the future.
> >
> > So this paragraph already includes at least two conflicting
> > proposals. On the one hand you want a separate device for
> > the virtual bond, on the other you are saying a separate
> > driver.
> 
> Just to be crystal clear: separate virtual bond device (netdev ops,
> not necessarily bus device) for VM migration specifically with a
> separate driver.

Okay, but note that any config someone had on a virtio device won't
propagate to that bond.

> >
> > Further, the reason to have a separate *driver* was that
> > some people wanted to share code with netvsc - and that
> > one does not create a separate device, which you can't
> > change without breaking existing configs.
> 
> I'm not sure I understand this statement. netvsc is already another
> netdev being created than the enslaved VF netdev, why it bothers?

Because it shipped, so userspace ABI is frozen.  You can't really add a
netdevice and enslave an existing one without a risk of breaking some
userspace configs.


> In
> the Azure case, the stock image to be imported does not bind to a
> specific driver but only MAC address.

I'll let netvsc developers decide this, on the surface I don't think
it's reasonable to assume everyone only binds to a MAC.


> And people just deal with the
> new virt-bond netdev rather than the underlying virtio and VF. And
> both these two underlying netdevs should be made invisible to prevent
> userspace script from getting them misconfigured IMHO.
> 
> A separate driver was for code sharing for sure, only just netvsc but
> could be other para-virtual devices floating around: any PV can serve
> as the side channel and the backup path for VF/PT. Once we get the new
> driver working atop virtio we may define ops and/or protocol needed to
> talk to various other PV frontend that may implement the side channel
> of its own for datapath switching (e.g. virtio is one of them, Xen PV
> frontend can be another). I just don't like to limit the function to
> virtio only and we have to duplicate code then it starts to scatter
> around all over the places.
> 
> I understand right now we start it as simple so it may just be fine
> that the initial development activities center around virtio. However,
> from cloud provider/vendor perspective I don't see the proposed scheme
> limits to virtio only. Any other PV driver which has the plan to
> support the same scheme can benefit. The point is that we shouldn't be
> limiting the scheme to virtio specifics so early which is hard to have
> it promoted to a common driver once we get there.

The whole idea has been floating around for years. It would always
get being drowned in this kind of "lets try to cover all use-cases"
discussions, and never make progress.
So let's see some working code merged. If it works fine for virtio
and 

Re: [RFC PATCH net-next v2 2/2] virtio_net: Extend virtio to use VF datapath when available

[Siwei Liu]

On Mon, Jan 22, 2018 at 1:41 PM, Michael S. Tsirkin  wrote:
> On Mon, Jan 22, 2018 at 12:27:14PM -0800, Siwei Liu wrote:
>> First off, as mentioned in another thread, the model of stacking up
>> virt-bond functionality over virtio seems a wrong direction to me.
>> Essentially the migration process would need to carry over all guest
>> side configurations previously done on the VF/PT and get them moved to
>> the new device being it virtio or VF/PT.
>
> I might be wrong but I don't see why we should worry about this usecase.
> Whoever has a bond configured already has working config for migration.
> We are trying to help people who don't, not convert existig users.

That has been placed in the view of cloud providers that the imported
images from the store must be able to run unmodified thus no
additional setup script is allowed (just as Stephen mentioned in
another mail). Cloud users don't care about live migration themselves
but the providers are required to implement such automation mechanism
to make this process transparent if at all possible. The user does not
care about the device underneath being VF or not, but they do care
about consistency all across and the resulting performance
acceleration in making VF the prefered datapath. It is not quite
peculiar user cases but IMHO *any* approach proposed for live
migration should be able to persist the state including network config
e.g. as simple as MTU. Actually this requirement has nothing to do
with virtio but our target users are live migration agnostic, being it
tracking DMA through dirty pages, using virtio as the helper, or
whatsoever, the goal of persisting configs across remains same.

>
>> Without the help of a new
>> upper layer bond driver that enslaves virtio and VF/PT devices
>> underneath, virtio will be overloaded with too much specifics being a
>> VF/PT backup in the future.
>
> So this paragraph already includes at least two conflicting
> proposals. On the one hand you want a separate device for
> the virtual bond, on the other you are saying a separate
> driver.

Just to be crystal clear: separate virtual bond device (netdev ops,
not necessarily bus device) for VM migration specifically with a
separate driver.

>
> Further, the reason to have a separate *driver* was that
> some people wanted to share code with netvsc - and that
> one does not create a separate device, which you can't
> change without breaking existing configs.

I'm not sure I understand this statement. netvsc is already another
netdev being created than the enslaved VF netdev, why it bothers? In
the Azure case, the stock image to be imported does not bind to a
specific driver but only MAC address. And people just deal with the
new virt-bond netdev rather than the underlying virtio and VF. And
both these two underlying netdevs should be made invisible to prevent
userspace script from getting them misconfigured IMHO.

A separate driver was for code sharing for sure, only just netvsc but
could be other para-virtual devices floating around: any PV can serve
as the side channel and the backup path for VF/PT. Once we get the new
driver working atop virtio we may define ops and/or protocol needed to
talk to various other PV frontend that may implement the side channel
of its own for datapath switching (e.g. virtio is one of them, Xen PV
frontend can be another). I just don't like to limit the function to
virtio only and we have to duplicate code then it starts to scatter
around all over the places.

I understand right now we start it as simple so it may just be fine
that the initial development activities center around virtio. However,
from cloud provider/vendor perspective I don't see the proposed scheme
limits to virtio only. Any other PV driver which has the plan to
support the same scheme can benefit. The point is that we shouldn't be
limiting the scheme to virtio specifics so early which is hard to have
it promoted to a common driver once we get there.

>
> So some people want a fully userspace-configurable switchdev, and that
> already exists at some level, and maybe it makes sense to add more
> features for performance.
>
> But the point was that some host configurations are very simple,
> and it probably makes sense to pass this information to the guest
> and have guest act on it directly. Let's not conflate the two.

It may be fine to push some of the configurations from host but that
perhaps doesn't cover all the cases: how is it possible for the host
to save all network states and configs done by the guest before
migration. Some of the configs might come from future guest which is
unknown to host. Anyhow the bottom line is that the guest must be able
to act on those configuration request changes automatically without
involving users intervention.

Regards,
-Siwei

>
> --
> MST
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org

Re: [virtio-dev] [RFC PATCH net-next v2 1/2] virtio_net: Introduce VIRTIO_NET_F_BACKUP feature bit

[Jakub Kicinski]

On Tue, 23 Jan 2018 03:23:57 +0200, Michael S. Tsirkin wrote:
> > > > b. next-gen silicon may be able to disguise as virtio device, and the
> > > >loop check in virtio driver will prevent the legitimate bond it such
> > > >case.  AFAIU that's one of the goals of next-gen virtio spec as 
> > > > well.
> > > 
> > > In fact we have a virtio feature bit for the fallback.
> > > So this part does not depend on how software in guest works
> > > and does not need software solutions.  
> > 
> > You mean in the new spec?  Nice.  Still I think people will try to
> > implement the old one too given sufficiently capable HW.  
> 
> Existing HW won't have the BACKUP feature so the new functionality
> won't be activated. So no problem I think.

I meant code that compares of netdev_ops, e.g.:

+   /* Skip our own events */
+   if (event_dev->netdev_ops == _netdev)
+   return NOTIFY_DONE;

Would be an obstacle to bonding virtio_nets.  But that's just one of
the thoughts, perhaps of disputable value.  Anyway, separate netdev and
netdev_ops will solve this, and I think we agree that's preferable :)
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [RFC PATCH net-next v2 2/2] virtio_net: Extend virtio to use VF datapath when available

[Samudrala, Sridhar]

On 1/23/2018 2:33 AM, Jason Wang wrote:



On 2018年01月12日 13:58, Sridhar Samudrala wrote:
  static netdev_tx_t start_xmit(struct sk_buff *skb, struct 
net_device *dev)

  {
  struct virtnet_info *vi = netdev_priv(dev);
  int qnum = skb_get_queue_mapping(skb);
  struct send_queue *sq = >sq[qnum];
+    struct net_device *vf_netdev;
  int err;
  struct netdev_queue *txq = netdev_get_tx_queue(dev, qnum);
  bool kick = !skb->xmit_more;
  bool use_napi = sq->napi.weight;
  +    /* If VF is present and up then redirect packets
+ * called with rcu_read_lock_bh
+ */
+    vf_netdev = rcu_dereference_bh(vi->vf_netdev);
+    if (vf_netdev && netif_running(vf_netdev) &&
+    !netpoll_tx_running(dev) &&
+    is_unicast_ether_addr(eth_hdr(skb)->h_dest))
+    return virtnet_vf_xmit(dev, vf_netdev, skb);
+


A question here.

If I read the code correctly, all features were validated against 
virtio instead VF before transmitting. This assumes VF's feature is a 
superset of virtio, does this really work? Do we need to sanitize the 
feature before joining? (e.g at last NETIF_R_GSO_ROBUST needs to be 
removed).


Actually, virtnet_vf_xmit() calls dev_queue_xmit() after updating 
skb->dev to vf netdev.
So the features get validated against VF features and the right tx queue 
is selected

before the real transmit.

Thanks
Sridhar



___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH net 2/2] vhost: do not try to access device IOTLB when not initialized

[Michael S. Tsirkin]

On Tue, Jan 23, 2018 at 05:27:26PM +0800, Jason Wang wrote:
> The code will try to access dev->iotlb when processing
> VHOST_IOTLB_INVALIDATE even if it was not initialized which may lead
> to NULL pointer dereference. Fixes this by check dev->iotlb before.
> 
> Fixes: 6b1e6cc7855b0 ("vhost: new device IOTLB API")
> Signed-off-by: Jason Wang 

Acked-by: Michael S. Tsirkin 

> ---
>  drivers/vhost/vhost.c | 4 
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 549771a..5727b18 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -1015,6 +1015,10 @@ static int vhost_process_iotlb_msg(struct vhost_dev 
> *dev,
>   vhost_iotlb_notify_vq(dev, msg);
>   break;
>   case VHOST_IOTLB_INVALIDATE:
> + if (!dev->iotlb) {
> + ret = -EFAULT;
> + break;
> + }
>   vhost_vq_meta_reset(dev);
>   vhost_del_umem_range(dev->iotlb, msg->iova,
>msg->iova + msg->size - 1);
> -- 
> 2.7.4
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH net 1/2] vhost: use mutex_lock_nested() in vhost_dev_lock_vqs()

[Michael S. Tsirkin]

On Tue, Jan 23, 2018 at 05:27:25PM +0800, Jason Wang wrote:
> We used to call mutex_lock() in vhost_dev_lock_vqs() which tries to
> hold mutexes of all virtqueues. This may confuse lockdep to report a
> possible deadlock because of trying to hold locks belong to same
> class. Switch to use mutex_lock_nested() to avoid false positive.
> 
> Fixes: 6b1e6cc7855b0 ("vhost: new device IOTLB API")
> Reported-by: syzbot+dbb7c1161485e61b0...@syzkaller.appspotmail.com
> Signed-off-by: Jason Wang 

Acked-by: Michael S. Tsirkin 

> ---
>  drivers/vhost/vhost.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 33ac2b1..549771a 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -904,7 +904,7 @@ static void vhost_dev_lock_vqs(struct vhost_dev *d)
>  {
>   int i = 0;
>   for (i = 0; i < d->nvqs; ++i)
> - mutex_lock(>vqs[i]->mutex);
> + mutex_lock_nested(>vqs[i]->mutex, i);
>  }
>  
>  static void vhost_dev_unlock_vqs(struct vhost_dev *d)
> -- 
> 2.7.4
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [RFC PATCH net-next v2 2/2] virtio_net: Extend virtio to use VF datapath when available

[Jason Wang]

On 2018年01月12日 13:58, Sridhar Samudrala wrote:

  static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
  {
struct virtnet_info *vi = netdev_priv(dev);
int qnum = skb_get_queue_mapping(skb);
struct send_queue *sq = >sq[qnum];
+   struct net_device *vf_netdev;
int err;
struct netdev_queue *txq = netdev_get_tx_queue(dev, qnum);
bool kick = !skb->xmit_more;
bool use_napi = sq->napi.weight;
  
+	/* If VF is present and up then redirect packets

+* called with rcu_read_lock_bh
+*/
+   vf_netdev = rcu_dereference_bh(vi->vf_netdev);
+   if (vf_netdev && netif_running(vf_netdev) &&
+   !netpoll_tx_running(dev) &&
+   is_unicast_ether_addr(eth_hdr(skb)->h_dest))
+   return virtnet_vf_xmit(dev, vf_netdev, skb);
+


A question here.

If I read the code correctly, all features were validated against virtio 
instead VF before transmitting. This assumes VF's feature is a superset 
of virtio, does this really work? Do we need to sanitize the feature 
before joining? (e.g at last NETIF_R_GSO_ROBUST needs to be removed).


Thanks
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH net 1/2] vhost: use mutex_lock_nested() in vhost_dev_lock_vqs()

[Jason Wang]

We used to call mutex_lock() in vhost_dev_lock_vqs() which tries to
hold mutexes of all virtqueues. This may confuse lockdep to report a
possible deadlock because of trying to hold locks belong to same
class. Switch to use mutex_lock_nested() to avoid false positive.

Fixes: 6b1e6cc7855b0 ("vhost: new device IOTLB API")
Reported-by: syzbot+dbb7c1161485e61b0...@syzkaller.appspotmail.com
Signed-off-by: Jason Wang 
---
 drivers/vhost/vhost.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 33ac2b1..549771a 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -904,7 +904,7 @@ static void vhost_dev_lock_vqs(struct vhost_dev *d)
 {
int i = 0;
for (i = 0; i < d->nvqs; ++i)
-   mutex_lock(>vqs[i]->mutex);
+   mutex_lock_nested(>vqs[i]->mutex, i);
 }
 
 static void vhost_dev_unlock_vqs(struct vhost_dev *d)
-- 
2.7.4

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH net 2/2] vhost: do not try to access device IOTLB when not initialized

[Jason Wang]

The code will try to access dev->iotlb when processing
VHOST_IOTLB_INVALIDATE even if it was not initialized which may lead
to NULL pointer dereference. Fixes this by check dev->iotlb before.

Fixes: 6b1e6cc7855b0 ("vhost: new device IOTLB API")
Signed-off-by: Jason Wang 
---
 drivers/vhost/vhost.c | 4 
 1 file changed, 4 insertions(+)

diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 549771a..5727b18 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -1015,6 +1015,10 @@ static int vhost_process_iotlb_msg(struct vhost_dev *dev,
vhost_iotlb_notify_vq(dev, msg);
break;
case VHOST_IOTLB_INVALIDATE:
+   if (!dev->iotlb) {
+   ret = -EFAULT;
+   break;
+   }
vhost_vq_meta_reset(dev);
vhost_del_umem_range(dev->iotlb, msg->iova,
 msg->iova + msg->size - 1);
-- 
2.7.4

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Ping Re: [PATCH] virtio: make VIRTIO a menuconfig to ease disabling it all

[Vincent Legoll]

On 1/23/18, Michael Ellerman  wrote:
> This has been broken in linux-next for ~6 weeks now, can we please merge
> this and get it fixed.

Added Stephen Rothwell to cc

-- 
Vincent Legoll
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization