Re: [PATCH] vhost: introduce vDPA based backend

[Tiwei Bie]

On Thu, Jan 30, 2020 at 09:12:57PM -0800, Randy Dunlap wrote:
> On 1/30/20 7:56 PM, Randy Dunlap wrote:
> > Hi,
> > 
> > On 1/30/20 7:36 PM, Tiwei Bie wrote:
> >> diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig
> >> index f21c45aa5e07..13e6a94d0243 100644
> >> --- a/drivers/vhost/Kconfig
> >> +++ b/drivers/vhost/Kconfig
> >> @@ -34,6 +34,18 @@ config VHOST_VSOCK
> >>To compile this driver as a module, choose M here: the module will be 
> >> called
> >>vhost_vsock.
> >>  
> >> +config VHOST_VDPA
> >> +  tristate "Vhost driver for vDPA based backend"
> 
> oops, missed this one:
>  vDPA-based

Will fix. Thanks!

> 
> >> +  depends on EVENTFD && VDPA
> >> +  select VHOST
> >> +  default n
> >> +  ---help---
> >> +  This kernel module can be loaded in host kernel to accelerate
> >> +  guest virtio devices with the vDPA based backends.
> > 
> >   vDPA-based
> > 
> >> +
> >> +  To compile this driver as a module, choose M here: the module
> >> +  will be called vhost_vdpa.
> >> +
> > 
> > The preferred Kconfig style nowadays is
> > (a) use "help" instead of "---help---"
> > (b) indent the help text with one tab + 2 spaces
> > 
> > and don't use "default n" since that is already the default.
> > 
> >>  config VHOST
> >>tristate
> >>  depends on VHOST_IOTLB
> > 
> > thanks.
> > 
> 
> 
> -- 
> ~Randy
> 
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] vhost: introduce vDPA based backend

[Tiwei Bie]

On Thu, Jan 30, 2020 at 07:56:43PM -0800, Randy Dunlap wrote:
> Hi,
> 
> On 1/30/20 7:36 PM, Tiwei Bie wrote:
> > diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig
> > index f21c45aa5e07..13e6a94d0243 100644
> > --- a/drivers/vhost/Kconfig
> > +++ b/drivers/vhost/Kconfig
> > @@ -34,6 +34,18 @@ config VHOST_VSOCK
> > To compile this driver as a module, choose M here: the module will be 
> > called
> > vhost_vsock.
> >  
> > +config VHOST_VDPA
> > +   tristate "Vhost driver for vDPA based backend"
> > +   depends on EVENTFD && VDPA
> > +   select VHOST
> > +   default n
> > +   ---help---
> > +   This kernel module can be loaded in host kernel to accelerate
> > +   guest virtio devices with the vDPA based backends.
> 
> vDPA-based

Will fix this and other similar ones in the patch. Thanks!

> 
> > +
> > +   To compile this driver as a module, choose M here: the module
> > +   will be called vhost_vdpa.
> > +
> 
> The preferred Kconfig style nowadays is
> (a) use "help" instead of "---help---"
> (b) indent the help text with one tab + 2 spaces
> 
> and don't use "default n" since that is already the default.

Will fix in the next version.

Thanks,
Tiwei

> 
> >  config VHOST
> > tristate
> >  depends on VHOST_IOTLB
> 
> thanks.
> -- 
> ~Randy
> 
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] vhost: introduce vDPA based backend

[Randy Dunlap]

On 1/30/20 7:56 PM, Randy Dunlap wrote:
> Hi,
> 
> On 1/30/20 7:36 PM, Tiwei Bie wrote:
>> diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig
>> index f21c45aa5e07..13e6a94d0243 100644
>> --- a/drivers/vhost/Kconfig
>> +++ b/drivers/vhost/Kconfig
>> @@ -34,6 +34,18 @@ config VHOST_VSOCK
>>  To compile this driver as a module, choose M here: the module will be 
>> called
>>  vhost_vsock.
>>  
>> +config VHOST_VDPA
>> +tristate "Vhost driver for vDPA based backend"

oops, missed this one:
   vDPA-based

>> +depends on EVENTFD && VDPA
>> +select VHOST
>> +default n
>> +---help---
>> +This kernel module can be loaded in host kernel to accelerate
>> +guest virtio devices with the vDPA based backends.
> 
> vDPA-based
> 
>> +
>> +To compile this driver as a module, choose M here: the module
>> +will be called vhost_vdpa.
>> +
> 
> The preferred Kconfig style nowadays is
> (a) use "help" instead of "---help---"
> (b) indent the help text with one tab + 2 spaces
> 
> and don't use "default n" since that is already the default.
> 
>>  config VHOST
>>  tristate
>>  depends on VHOST_IOTLB
> 
> thanks.
> 


-- 
~Randy

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] vhost: introduce vDPA based backend

[Randy Dunlap]

Hi,

On 1/30/20 7:36 PM, Tiwei Bie wrote:
> diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig
> index f21c45aa5e07..13e6a94d0243 100644
> --- a/drivers/vhost/Kconfig
> +++ b/drivers/vhost/Kconfig
> @@ -34,6 +34,18 @@ config VHOST_VSOCK
>   To compile this driver as a module, choose M here: the module will be 
> called
>   vhost_vsock.
>  
> +config VHOST_VDPA
> + tristate "Vhost driver for vDPA based backend"
> + depends on EVENTFD && VDPA
> + select VHOST
> + default n
> + ---help---
> + This kernel module can be loaded in host kernel to accelerate
> + guest virtio devices with the vDPA based backends.

  vDPA-based

> +
> + To compile this driver as a module, choose M here: the module
> + will be called vhost_vdpa.
> +

The preferred Kconfig style nowadays is
(a) use "help" instead of "---help---"
(b) indent the help text with one tab + 2 spaces

and don't use "default n" since that is already the default.

>  config VHOST
>   tristate
>  depends on VHOST_IOTLB

thanks.
-- 
~Randy

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH] vhost: introduce vDPA based backend

[Tiwei Bie]

This patch introduces a vDPA based vhost backend. This
backend is built on top of the same interface defined
in virtio-vDPA and provides a generic vhost interface
for userspace to accelerate the virtio devices in guest.

This backend is implemented as a vDPA device driver on
top of the same ops used in virtio-vDPA. It will create
char device entry named vhost-vdpa/$vdpa_device_index
for userspace to use. Userspace can use vhost ioctls on
top of this char device to setup the backend.

Signed-off-by: Tiwei Bie 
---
This patch depends on below series:
https://lkml.org/lkml/2020/1/16/353

Please note that _SET_MEM_TABLE isn't fully supported yet.
Comments would be appreciated!

Changes since last patch (https://lkml.org/lkml/2019/11/18/1068)
- Switch to the vDPA bus;
- Switch to vhost's own chardev;

 drivers/vhost/Kconfig|  12 +
 drivers/vhost/Makefile   |   3 +
 drivers/vhost/vdpa.c | 705 +++
 include/uapi/linux/vhost.h   |  21 +
 include/uapi/linux/vhost_types.h |   8 +
 5 files changed, 749 insertions(+)
 create mode 100644 drivers/vhost/vdpa.c

diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig
index f21c45aa5e07..13e6a94d0243 100644
--- a/drivers/vhost/Kconfig
+++ b/drivers/vhost/Kconfig
@@ -34,6 +34,18 @@ config VHOST_VSOCK
To compile this driver as a module, choose M here: the module will be 
called
vhost_vsock.
 
+config VHOST_VDPA
+   tristate "Vhost driver for vDPA based backend"
+   depends on EVENTFD && VDPA
+   select VHOST
+   default n
+   ---help---
+   This kernel module can be loaded in host kernel to accelerate
+   guest virtio devices with the vDPA based backends.
+
+   To compile this driver as a module, choose M here: the module
+   will be called vhost_vdpa.
+
 config VHOST
tristate
 depends on VHOST_IOTLB
diff --git a/drivers/vhost/Makefile b/drivers/vhost/Makefile
index df99756fbb26..a65e9f4a2c0a 100644
--- a/drivers/vhost/Makefile
+++ b/drivers/vhost/Makefile
@@ -10,6 +10,9 @@ vhost_vsock-y := vsock.o
 
 obj-$(CONFIG_VHOST_RING) += vringh.o
 
+obj-$(CONFIG_VHOST_VDPA) += vhost_vdpa.o
+vhost_vdpa-y := vdpa.o
+
 obj-$(CONFIG_VHOST)+= vhost.o
 
 obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o
diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
new file mode 100644
index ..631d994d37ac
--- /dev/null
+++ b/drivers/vhost/vdpa.c
@@ -0,0 +1,705 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018-2020 Intel Corporation.
+ *
+ * Author: Tiwei Bie 
+ *
+ * Thanks to Jason Wang and Michael S. Tsirkin for the valuable
+ * comments and suggestions.  And thanks to Cunming Liang and
+ * Zhihong Wang for all their supports.
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "vhost.h"
+
+enum {
+   VHOST_VDPA_FEATURES =
+   (1ULL << VIRTIO_F_NOTIFY_ON_EMPTY) |
+   (1ULL << VIRTIO_F_ANY_LAYOUT) |
+   (1ULL << VIRTIO_F_VERSION_1) |
+   (1ULL << VIRTIO_F_IOMMU_PLATFORM) |
+   (1ULL << VIRTIO_F_RING_PACKED) |
+   (1ULL << VIRTIO_F_ORDER_PLATFORM) |
+   (1ULL << VIRTIO_RING_F_INDIRECT_DESC) |
+   (1ULL << VIRTIO_RING_F_EVENT_IDX),
+
+   VHOST_VDPA_NET_FEATURES = VHOST_VDPA_FEATURES |
+   (1ULL << VIRTIO_NET_F_CSUM) |
+   (1ULL << VIRTIO_NET_F_GUEST_CSUM) |
+   (1ULL << VIRTIO_NET_F_MTU) |
+   (1ULL << VIRTIO_NET_F_MAC) |
+   (1ULL << VIRTIO_NET_F_GUEST_TSO4) |
+   (1ULL << VIRTIO_NET_F_GUEST_TSO6) |
+   (1ULL << VIRTIO_NET_F_GUEST_ECN) |
+   (1ULL << VIRTIO_NET_F_GUEST_UFO) |
+   (1ULL << VIRTIO_NET_F_HOST_TSO4) |
+   (1ULL << VIRTIO_NET_F_HOST_TSO6) |
+   (1ULL << VIRTIO_NET_F_HOST_ECN) |
+   (1ULL << VIRTIO_NET_F_HOST_UFO) |
+   (1ULL << VIRTIO_NET_F_MRG_RXBUF) |
+   (1ULL << VIRTIO_NET_F_STATUS) |
+   (1ULL << VIRTIO_NET_F_SPEED_DUPLEX),
+};
+
+/* Currently, only network backend w/o multiqueue is supported. */
+#define VHOST_VDPA_VQ_MAX  2
+
+struct vhost_vdpa {
+   /* The lock is to protect this structure. */
+   struct mutex mutex;
+   struct vhost_dev vdev;
+   struct vhost_virtqueue *vqs;
+   struct vdpa_device *vdpa;
+   struct device *dev;
+   atomic_t opened;
+   int nvqs;
+   int virtio_id;
+   int minor;
+};
+
+static struct {
+   /* The lock is to protect this structure. */
+   struct mutex mutex;
+   struct class *class;
+   struct idr idr;
+   struct cdev cdev;
+   dev_t devt;
+   wait_queue_head_t release_q;
+} vhost_vdpa;
+
+static const u64 vhost_vdpa_features[] = {
+   [VIRTIO_ID_NET] = VHOST_VDPA_NET_FEATURES,
+};
+
+static int vhost_vdpa_alloc_minor(struct vhost_vdpa *v)
+{
+   return 

Re: Balloon pressuring page cache

[Tyler Sanderson via Virtualization]

On Thu, Jan 30, 2020 at 7:03 AM David Hildenbrand  wrote:

> On 29.01.20 20:11, Tyler Sanderson wrote:
> >
> >
> > On Wed, Jan 29, 2020 at 2:31 AM David Hildenbrand  > > wrote:
> >
> > On 29.01.20 01:22, Tyler Sanderson via Virtualization wrote:
> > > A primary advantage of virtio balloon over other memory reclaim
> > > mechanisms is that it can pressure the guest's page cache into
> > shrinking.
> > >
> > > However, since the balloon driver changed to using the shrinker API
> > >
> > <
> https://github.com/torvalds/linux/commit/71994620bb25a8b109388fefa9e99a28e355255a#diff-fd202acf694d9eba19c8c64da3e480c9
> > this
> > > use case has become a bit more tricky. I'm wondering what the
> intended
> > > device implementation is.
> > >
> > > When inflating the balloon against page cache (i.e. no free memory
> > > remains) vmscan.c will both shrink page cache, but also invoke the
> > > shrinkers -- including the balloon's shrinker. So the balloon
> driver
> > > allocates memory which requires reclaim, vmscan gets this memory by
> > > shrinking the balloon, and then the driver adds the memory back to
> the
> > > balloon. Basically a busy no-op.
> > >
> > > If file IO is ongoing during this balloon inflation then the page
> > cache
> > > could be growing which further puts "back pressure" on the balloon
> > > trying to inflate. In testing I've seen periods of > 45 seconds
> where
> > > balloon inflation makes no net forward progress.
> > >
> > > This wasn't a problem before the change to the shrinker API since
> > forced
> > > balloon deflation only occurred via the OOM notifier callback
> > which was
> > > invoked only after the page cache had depleted.
> > >
> > > Is this new busy behavior working as intended?
> >
> > Please note that the shrinker will only be registered in case we have
> > VIRTIO_BALLOON_F_DEFLATE_ON_OOM - (which is AFAIK very rare) - to
> > implement automatic balloon deflation when the guest is under memory
> > pressure.
> >
> >
> > Are you actually experiencing issues with that or did you just
> stumble
> > over the code?
> >
> >
> > We have a use case that is encountering this (and that registers
> > DEFLATE_ON_OOM). We can work around this, but it does seem inefficient.
> > I understand there were good reasons for moving away from the OOM
> > notifier callback, but I'm wondering if the balloon driver could specify
> > a "nice" level to the shrinker API that would cause it to be reclaimed
> > from only as a last resort?
> >
>
> Cc-ing linux-mm, Michal and Michael.
>
> Just wondering, how does your workaround look like?
>
The work around is to monitor the memory statistics reported on the stats
queue. Keep inflating (inefficiently) -- despite the activity on the
deflate queue -- until memory statistics indicate the guest is actually low
on available memory.

-- 
> Thanks,
>
> David / dhildenb
>
>
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Balloon pressuring page cache

[Tyler Sanderson via Virtualization]

On Thu, Jan 30, 2020 at 7:31 AM Wang, Wei W  wrote:

> On Thursday, January 30, 2020 11:03 PM, David Hildenbrand wrote:
> > On 29.01.20 20:11, Tyler Sanderson wrote:
> > >
> > >
> > > On Wed, Jan 29, 2020 at 2:31 AM David Hildenbrand  > > > wrote:
> > >
> > > On 29.01.20 01:22, Tyler Sanderson via Virtualization wrote:
> > > > A primary advantage of virtio balloon over other memory reclaim
> > > > mechanisms is that it can pressure the guest's page cache into
> > > shrinking.
> > > >
> > > > However, since the balloon driver changed to using the shrinker
> API
> > > >
> > >
> >  > e99a28e355255a#diff-fd202acf694d9eba19c8c64da3e480c9> this
> > > > use case has become a bit more tricky. I'm wondering what the
> > intended
> > > > device implementation is.
> > > >
> > > > When inflating the balloon against page cache (i.e. no free
> memory
> > > > remains) vmscan.c will both shrink page cache, but also invoke
> the
> > > > shrinkers -- including the balloon's shrinker. So the balloon
> driver
> > > > allocates memory which requires reclaim, vmscan gets this memory
> > by
> > > > shrinking the balloon, and then the driver adds the memory back
> to
> > the
> > > > balloon. Basically a busy no-op.
>
> Per my understanding, the balloon allocation won’t invoke shrinker as
> __GFP_DIRECT_RECLAIM isn't set, no?
>
I could be wrong about the mechanism, but the device sees lots of activity
on the deflate queue. The balloon is being shrunk. And this only starts
once all free memory is depleted and we're inflating into page cache.

>
>
> > > >
> > > > If file IO is ongoing during this balloon inflation then the page
> > > cache
> > > > could be growing which further puts "back pressure" on the
> balloon
> > > > trying to inflate. In testing I've seen periods of > 45 seconds
> where
> > > > balloon inflation makes no net forward progress.
>
> I think this is intentional (but could be improved). As inflation does not
> stop when the allocation fails (it simply sleeps for a while and resumes..
> repeat till there are memory to inflate)
> That's why you see no inflation progress for long time under memory
> pressure.
>
As noted above the deflate queue is active, so it's not just memory
allocation failures.


>
>
> Best,
> Wei
>
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

RE: Balloon pressuring page cache

[Wang, Wei W]

On Thursday, January 30, 2020 11:03 PM, David Hildenbrand wrote:
> On 29.01.20 20:11, Tyler Sanderson wrote:
> >
> >
> > On Wed, Jan 29, 2020 at 2:31 AM David Hildenbrand  > > wrote:
> >
> > On 29.01.20 01:22, Tyler Sanderson via Virtualization wrote:
> > > A primary advantage of virtio balloon over other memory reclaim
> > > mechanisms is that it can pressure the guest's page cache into
> > shrinking.
> > >
> > > However, since the balloon driver changed to using the shrinker API
> > >
> >
>  e99a28e355255a#diff-fd202acf694d9eba19c8c64da3e480c9> this
> > > use case has become a bit more tricky. I'm wondering what the
> intended
> > > device implementation is.
> > >
> > > When inflating the balloon against page cache (i.e. no free memory
> > > remains) vmscan.c will both shrink page cache, but also invoke the
> > > shrinkers -- including the balloon's shrinker. So the balloon driver
> > > allocates memory which requires reclaim, vmscan gets this memory
> by
> > > shrinking the balloon, and then the driver adds the memory back to
> the
> > > balloon. Basically a busy no-op.

Per my understanding, the balloon allocation won’t invoke shrinker as 
__GFP_DIRECT_RECLAIM isn't set, no?


> > >
> > > If file IO is ongoing during this balloon inflation then the page
> > cache
> > > could be growing which further puts "back pressure" on the balloon
> > > trying to inflate. In testing I've seen periods of > 45 seconds where
> > > balloon inflation makes no net forward progress.

I think this is intentional (but could be improved). As inflation does not stop 
when the allocation fails (it simply sleeps for a while and resumes.. repeat 
till there are memory to inflate)
That's why you see no inflation progress for long time under memory pressure.


Best,
Wei
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Balloon pressuring page cache

[David Hildenbrand]

On 30.01.20 16:20, Michael S. Tsirkin wrote:
> On Thu, Jan 30, 2020 at 04:02:34PM +0100, David Hildenbrand wrote:
>> On 29.01.20 20:11, Tyler Sanderson wrote:
>>>
>>>
>>> On Wed, Jan 29, 2020 at 2:31 AM David Hildenbrand >> > wrote:
>>>
>>> On 29.01.20 01:22, Tyler Sanderson via Virtualization wrote:
>>> > A primary advantage of virtio balloon over other memory reclaim
>>> > mechanisms is that it can pressure the guest's page cache into
>>> shrinking.
>>> >
>>> > However, since the balloon driver changed to using the shrinker API
>>> >
>>> 
>>> 
>>>  this
>>> > use case has become a bit more tricky. I'm wondering what the intended
>>> > device implementation is.
>>> >
>>> > When inflating the balloon against page cache (i.e. no free memory
>>> > remains) vmscan.c will both shrink page cache, but also invoke the
>>> > shrinkers -- including the balloon's shrinker. So the balloon driver
>>> > allocates memory which requires reclaim, vmscan gets this memory by
>>> > shrinking the balloon, and then the driver adds the memory back to the
>>> > balloon. Basically a busy no-op.
>>> >
>>> > If file IO is ongoing during this balloon inflation then the page
>>> cache
>>> > could be growing which further puts "back pressure" on the balloon
>>> > trying to inflate. In testing I've seen periods of > 45 seconds where
>>> > balloon inflation makes no net forward progress.
>>> >
>>> > This wasn't a problem before the change to the shrinker API since
>>> forced
>>> > balloon deflation only occurred via the OOM notifier callback
>>> which was
>>> > invoked only after the page cache had depleted.
>>> >
>>> > Is this new busy behavior working as intended?
>>>
>>> Please note that the shrinker will only be registered in case we have
>>> VIRTIO_BALLOON_F_DEFLATE_ON_OOM - (which is AFAIK very rare) - to
>>> implement automatic balloon deflation when the guest is under memory
>>> pressure.
>>>
>>>
>>> Are you actually experiencing issues with that or did you just stumble
>>> over the code?
>>>
>>>
>>> We have a use case that is encountering this (and that registers
>>> DEFLATE_ON_OOM). We can work around this, but it does seem inefficient.
>>> I understand there were good reasons for moving away from the OOM
>>> notifier callback, but I'm wondering if the balloon driver could specify
>>> a "nice" level to the shrinker API that would cause it to be reclaimed
>>> from only as a last resort?
>>>  
>>
>> Cc-ing linux-mm, Michal and Michael.
> 
> 
> Interesting.  VIRTIO_BALLOON_F_DEFLATE_ON_OOM is really
> underspecified in a bunch of ways.
> 
> I'll wait to see what does Michal say from Linux POV.


Just wondering, does implying that we are using the shrinker that a

echo 3 > /proc/sys/vm/drop_caches

Will deflate the whole balloon? If yes, than that's *really* not desired.

-- 
Thanks,

David / dhildenb

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Balloon pressuring page cache

[Michael S. Tsirkin]

On Thu, Jan 30, 2020 at 04:02:34PM +0100, David Hildenbrand wrote:
> On 29.01.20 20:11, Tyler Sanderson wrote:
> > 
> > 
> > On Wed, Jan 29, 2020 at 2:31 AM David Hildenbrand  > > wrote:
> > 
> > On 29.01.20 01:22, Tyler Sanderson via Virtualization wrote:
> > > A primary advantage of virtio balloon over other memory reclaim
> > > mechanisms is that it can pressure the guest's page cache into
> > shrinking.
> > >
> > > However, since the balloon driver changed to using the shrinker API
> > >
> > 
> > Â
> >  this
> > > use case has become a bit more tricky. I'm wondering what the intended
> > > device implementation is.
> > >
> > > When inflating the balloon against page cache (i.e. no free memory
> > > remains) vmscan.c will both shrink page cache, but also invoke the
> > > shrinkers -- including the balloon's shrinker. So the balloon driver
> > > allocates memory which requires reclaim, vmscan gets this memory by
> > > shrinking the balloon, and then the driver adds the memory back to the
> > > balloon. Basically a busy no-op.
> > >
> > > If file IO is ongoing during this balloon inflation then the page
> > cache
> > > could be growing which further puts "back pressure" on the balloon
> > > trying to inflate. In testing I've seen periods of > 45 seconds where
> > > balloon inflation makes no net forward progress.
> > >
> > > This wasn't a problem before the change to the shrinker API since
> > forced
> > > balloon deflation only occurred via the OOM notifier callback
> > which was
> > > invoked only after the page cache had depleted.
> > >
> > > Is this new busy behavior working as intended?
> > 
> > Please note that the shrinker will only be registered in case we have
> > VIRTIO_BALLOON_F_DEFLATE_ON_OOM - (which is AFAIK very rare) - to
> > implement automatic balloon deflation when the guest is under memory
> > pressure.
> > 
> > 
> > Are you actually experiencing issues with that or did you just stumble
> > over the code?
> > 
> > 
> > We have a use case that is encountering this (and that registers
> > DEFLATE_ON_OOM). We can work around this, but it does seem inefficient.
> > I understand there were good reasons for moving away from the OOM
> > notifier callback, but I'm wondering if the balloon driver could specify
> > a "nice" level to the shrinker API that would cause it to be reclaimed
> > from only as a last resort?
> >  
> 
> Cc-ing linux-mm, Michal and Michael.


Interesting.  VIRTIO_BALLOON_F_DEFLATE_ON_OOM is really
underspecified in a bunch of ways.

I'll wait to see what does Michal say from Linux POV.





> Just wondering, how does your workaround look like?
> 
> -- 
> Thanks,
> 
> David / dhildenb

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Balloon pressuring page cache

[David Hildenbrand]

On 29.01.20 20:11, Tyler Sanderson wrote:
> 
> 
> On Wed, Jan 29, 2020 at 2:31 AM David Hildenbrand  > wrote:
> 
> On 29.01.20 01:22, Tyler Sanderson via Virtualization wrote:
> > A primary advantage of virtio balloon over other memory reclaim
> > mechanisms is that it can pressure the guest's page cache into
> shrinking.
> >
> > However, since the balloon driver changed to using the shrinker API
> >
> 
> 
>  this
> > use case has become a bit more tricky. I'm wondering what the intended
> > device implementation is.
> >
> > When inflating the balloon against page cache (i.e. no free memory
> > remains) vmscan.c will both shrink page cache, but also invoke the
> > shrinkers -- including the balloon's shrinker. So the balloon driver
> > allocates memory which requires reclaim, vmscan gets this memory by
> > shrinking the balloon, and then the driver adds the memory back to the
> > balloon. Basically a busy no-op.
> >
> > If file IO is ongoing during this balloon inflation then the page
> cache
> > could be growing which further puts "back pressure" on the balloon
> > trying to inflate. In testing I've seen periods of > 45 seconds where
> > balloon inflation makes no net forward progress.
> >
> > This wasn't a problem before the change to the shrinker API since
> forced
> > balloon deflation only occurred via the OOM notifier callback
> which was
> > invoked only after the page cache had depleted.
> >
> > Is this new busy behavior working as intended?
> 
> Please note that the shrinker will only be registered in case we have
> VIRTIO_BALLOON_F_DEFLATE_ON_OOM - (which is AFAIK very rare) - to
> implement automatic balloon deflation when the guest is under memory
> pressure.
> 
> 
> Are you actually experiencing issues with that or did you just stumble
> over the code?
> 
> 
> We have a use case that is encountering this (and that registers
> DEFLATE_ON_OOM). We can work around this, but it does seem inefficient.
> I understand there were good reasons for moving away from the OOM
> notifier callback, but I'm wondering if the balloon driver could specify
> a "nice" level to the shrinker API that would cause it to be reclaimed
> from only as a last resort?
>  

Cc-ing linux-mm, Michal and Michael.

Just wondering, how does your workaround look like?

-- 
Thanks,

David / dhildenb

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization