[PATCH v1 11/11] MAINTAINERS: Add myself as virtio-mem maintainer

2020-03-02 Thread David Hildenbrand
Let's make sure patches/bug reports find the right person. Cc: "Michael S. Tsirkin" Signed-off-by: David Hildenbrand --- MAINTAINERS | 7 +++ 1 file changed, 7 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 1636ce4613e3..fc7371c4b7eb 100644 --- a/MAINTAINERS ++

[PATCH v1 10/11] virtio-mem: Better retry handling

2020-03-02 Thread David Hildenbrand
Dave Young Cc: Andrew Morton Cc: Dan Williams Cc: Pavel Tatashin Cc: Stefan Hajnoczi Cc: Vlastimil Babka Signed-off-by: David Hildenbrand --- drivers/virtio/virtio_mem.c | 11 --- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/drivers/virtio/virtio_mem.c b/drivers/virt

Re: [PATCH v1 01/11] ACPI: NUMA: export pxm_to_node

2020-03-02 Thread David Hildenbrand
On 02.03.20 15:03, Michal Hocko wrote: > On Mon 02-03-20 14:49:31, David Hildenbrand wrote: >> Will be needed by virtio-mem to identify the node from a pxm. > > No objection to export the symbol. But it is almost always better to add > the export in the patch that actually uses

Re: [PATCH v1 04/11] mm: Export alloc_contig_range() / free_contig_range()

2020-03-02 Thread David Hildenbrand
On 02.03.20 15:05, Michal Hocko wrote: > On Mon 02-03-20 14:49:34, David Hildenbrand wrote: >> A virtio-mem device wants to allocate memory from the memory region it >> manages in order to unplug it in the hypervisor - similar to >> a balloon driver. Also, it might wa

Re: [PATCH v1 00/11] virtio-mem: paravirtualized memory

2020-03-02 Thread David Hildenbrand
On 02.03.20 14:49, David Hildenbrand wrote: > This series is based on latest linux-next. The patches are located at: > https://github.com/davidhildenbrand/linux.git virtio-mem-v1 > > The basic idea of virtio-mem is to provide a flexible, > cross-architecture memory hot(un)plu

Re: [PATCH v1 00/11] virtio-mem: paravirtualized memory

2020-03-02 Thread David Hildenbrand
On 02.03.20 19:29, Michal Hocko wrote: > On Mon 02-03-20 19:15:09, David Hildenbrand wrote: > [...] >> As requested by Michal, I will squash some patches. > > Just to clarify. If I am the only one to care then do not bother. Oh, I do bother about your review comments a lot :) A

Re: [PATCH v1 06/11] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE

2020-03-02 Thread David Hildenbrand
On 02.03.20 18:40, Alexander Duyck wrote: > Reviewed-by: Alexander Duyck Thanks a lot Alex! -- Thanks, David / dhildenb ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virt

Re: [PATCH v1 02/11] virtio-mem: Paravirtualized memory hotplug

2020-03-03 Thread David Hildenbrand
patch is applied to the wrong git tree, please drop us a note to help > improve the system. BTW, we also suggest to use '--base' option to specify the > base tree in git format-patch, please see > https://stackoverflow.com/a/37406982] > > url: > https://github.co

Re: [PATCH] drivers: virtio: Make out_del_vqs dependent on BALLOON_COMPACTION

2020-03-08 Thread David Hildenbrand
> Am 06.03.2020 um 11:56 schrieb Vincenzo Frascino : > > out_del_vqs label is currently used only when BALLOON_COMPACTION > configuration option is enabled. Having it disabled triggers the > following warning at compile time: > > drivers/virtio/virtio_balloon.c: In function ‘virtballoon_probe’

Re: [PATCH v1 3/3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-09 Thread David Hildenbrand
On 08.03.20 05:47, Tyler Sanderson wrote: > Tested-by: Tyler Sanderson > > Test setup: VM with 16 CPU, 64GB RAM. Running Debian 10. We have a 42 > GB file full of random bytes that we continually cat to /dev/null. > This fills the page cache as the file is read. Meanwhile we trigger > the balloon

Re: [PATCH v1 3/3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-09 Thread David Hildenbrand
On 09.03.20 11:14, Michael S. Tsirkin wrote: > On Mon, Mar 09, 2020 at 10:03:14AM +0100, David Hildenbrand wrote: >> On 08.03.20 05:47, Tyler Sanderson wrote: >>> Tested-by: Tyler Sanderson >>> >>> Test setup: VM with 16 CPU, 64GB RAM. Running Debian 10. We

[PATCH v2] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-10 Thread David Hildenbrand
s://www.spinics.net/lists/linux-virtualization/msg40863.html Reported-by: Tyler Sanderson Tested-by: Tyler Sanderson Fixes: 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker") Cc: Michael S. Tsirkin Cc: Wei Wang Cc: Alexander Duyck Cc: David Rientjes Cc: Nadav Amit Cc

Re: [PATCH v2] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-10 Thread David Hildenbrand
>> static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb) >> @@ -971,7 +950,22 @@ static int virtballoon_probe(struct virtio_device *vdev) >>VIRTIO_BALLOON_CMD_ID_STOP); >> spin_lock_init(&vb->free_page_list_lock); >>

Re: [PATCH v2] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-10 Thread David Hildenbrand
On 10.03.20 12:19, Michael S. Tsirkin wrote: > On Tue, Mar 10, 2020 at 12:12:50PM +0100, David Hildenbrand wrote: >>>> static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb) >>>> @@ -971,7 +950,22 @@ static int virtballoon_probe(struc

[PATCH v3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-10 Thread David Hildenbrand
s://www.spinics.net/lists/linux-virtualization/msg40863.html Reported-by: Tyler Sanderson Tested-by: Tyler Sanderson Fixes: 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker") Cc: Michael S. Tsirkin Cc: Wei Wang Cc: Alexander Duyck Cc: David Rientjes Cc: Nadav Amit

Re: [PATCH v1 07/11] virtio-mem: Allow to offline partially unplugged memory blocks

2020-03-10 Thread David Hildenbrand
On 10.03.20 12:43, Michal Hocko wrote: > On Mon 02-03-20 14:49:37, David Hildenbrand wrote: > [...] >> +static void virtio_mem_notify_going_offline(struct virtio_mem *vm, >> +unsigned long mb_id) >> +{ >> +const unsign

Re: [PATCH v1 06/11] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE

2020-03-10 Thread David Hildenbrand
On 10.03.20 12:47, Michal Hocko wrote: > On Mon 02-03-20 14:49:36, David Hildenbrand wrote: >> virtio-mem wants to allow to offline memory blocks of which some parts >> were unplugged (allocated via alloc_contig_range()), especially, to later >> offline and remove comple

[PATCH v1] MAINTAINERS: Add myself as virtio-balloon co-maintainer

2020-03-10 Thread David Hildenbrand
As suggested by Michael, let's add me as co-maintainer of virtio-balloon. While at it, also add "include/linux/balloon_compaction.h" to the file list. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: Greg Kroah-Hartman Signed-off-by: David Hildenbrand --- MAINTAINERS | 9 +

Re: [PATCH v1 07/11] virtio-mem: Allow to offline partially unplugged memory blocks

2020-03-10 Thread David Hildenbrand
On 10.03.20 12:59, Michal Hocko wrote: > On Tue 10-03-20 12:46:05, David Hildenbrand wrote: >> On 10.03.20 12:43, Michal Hocko wrote: >>> On Mon 02-03-20 14:49:37, David Hildenbrand wrote: >>> [...] >>>> +static void virtio_mem_not

Re: [PATCH v3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-10 Thread David Hildenbrand
On 10.03.20 20:05, David Rientjes wrote: > On Tue, 10 Mar 2020, David Hildenbrand wrote: > >> Commit 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker") >> changed the behavior when deflation happens automatically. Instead of >> deflating w

Re: [PATCH v3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-10 Thread David Hildenbrand
> Am 10.03.2020 um 22:25 schrieb Michael S. Tsirkin : > > On Tue, Mar 10, 2020 at 08:13:19PM +0100, David Hildenbrand wrote: >>> Should this have: >>> >>> Cc: sta...@vger.kernel.org # 4.19+ >> >> I guess as nothing will actually "crash&q

Re: [PATCH v3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-11 Thread David Hildenbrand
On 11.03.20 12:12, Michael S. Tsirkin wrote: > On Tue, Mar 10, 2020 at 08:13:19PM +0100, David Hildenbrand wrote: >> On 10.03.20 20:05, David Rientjes wrote: >>> On Tue, 10 Mar 2020, David Hildenbrand wrote: >>> >>>> Commit 71994620bb25 ("virtio

[PATCH v4] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM

2020-03-11 Thread David Hildenbrand
s://www.spinics.net/lists/linux-virtualization/msg40863.html Reported-by: Tyler Sanderson Tested-by: Tyler Sanderson Acked-by: David Rientjes Fixes: 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker") Cc: Michael S. Tsirkin Cc: Wei Wang Cc: Alexander Duyck Cc: David Rient

[PATCH v4 0/1] mm: virtio-balloon fix to go through the -mm tree

2020-03-11 Thread David Hildenbrand
ng instructions/results - Added Fixes: and Tested-by: David Hildenbrand (1): virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM drivers/virtio/virtio_balloon.c | 103 +++- 1 file changed, 47 insertions(+), 56 deletio

[PATCH v2 00/10] virtio-mem: paravirtualized memory

2020-03-11 Thread David Hildenbrand
Q: How big should the initial (!virtio-mem) memory of a VM be? A: virtio-mem memory will not go to the DMA zones. So to avoid running out of DMA memory, I suggest something like 2-3GB on x86-64. But many VMs can most probably deal with less DMA memory - depends on the use case. [1] https://

[PATCH v2 04/10] virtio-mem: Paravirtualized memory hotunplug part 2

2020-03-11 Thread David Hildenbrand
Cc: Andrew Morton Cc: Dan Williams Cc: Pavel Tatashin Cc: Stefan Hajnoczi Cc: Vlastimil Babka Cc: Mel Gorman Cc: Mike Rapoport Cc: Alexander Duyck Cc: Alexander Potapenko Signed-off-by: David Hildenbrand --- drivers/virtio/Kconfig | 1 + drivers/virtio/virtio_m

[PATCH v2 05/10] mm: Allow to offline unmovable PageOffline() pages via MEM_GOING_OFFLINE

2020-03-11 Thread David Hildenbrand
s Cc: Konrad Rzeszutek Wilk Cc: Pavel Tatashin Cc: Alexander Duyck Cc: Vlastimil Babka Cc: Johannes Weiner Cc: Anthony Yznaga Cc: Michal Hocko Cc: Oscar Salvador Cc: Mel Gorman Cc: Mike Rapoport Cc: Dan Williams Cc: Anshuman Khandual Cc: Qian Cai Cc: Pingfan Liu Signed-off-by:

[PATCH v2 03/10] virtio-mem: Paravirtualized memory hotunplug part 1

2020-03-11 Thread David Hildenbrand
Cc: Pavel Tatashin Cc: Stefan Hajnoczi Cc: Vlastimil Babka Signed-off-by: David Hildenbrand --- drivers/virtio/virtio_mem.c | 116 +++- 1 file changed, 114 insertions(+), 2 deletions(-) diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index d8

[PATCH v2 02/10] virtio-mem: Allow to specify an ACPI PXM as nid

2020-03-11 Thread David Hildenbrand
socki" # for the export Cc: "Michael S. Tsirkin" Cc: Jason Wang Cc: Oscar Salvador Cc: Michal Hocko Cc: Igor Mammedov Cc: Dave Young Cc: Andrew Morton Cc: Dan Williams Cc: Pavel Tatashin Cc: Stefan Hajnoczi Cc: Vlastimil Babka Cc: Len Brown Cc: linux-a...@vger.kernel.org

[PATCH v2 01/10] virtio-mem: Paravirtualized memory hotplug

2020-03-11 Thread David Hildenbrand
Vlastimil Babka Cc: "Rafael J. Wysocki" Cc: Len Brown Cc: linux-a...@vger.kernel.org Signed-off-by: David Hildenbrand --- drivers/virtio/Kconfig | 17 + drivers/virtio/Makefile |1 + drivers/virtio/virtio_mem.c | 1526 +++ incl

[PATCH v2 06/10] virtio-mem: Allow to offline partially unplugged memory blocks

2020-03-11 Thread David Hildenbrand
ve Young Cc: Andrew Morton Cc: Dan Williams Cc: Pavel Tatashin Cc: Stefan Hajnoczi Cc: Vlastimil Babka Signed-off-by: David Hildenbrand --- drivers/virtio/virtio_mem.c | 68 - 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/drivers/virtio/vi

[PATCH v2 10/10] MAINTAINERS: Add myself as virtio-mem maintainer

2020-03-11 Thread David Hildenbrand
Let's make sure patches/bug reports find the right person. Cc: "Michael S. Tsirkin" Signed-off-by: David Hildenbrand --- MAINTAINERS | 7 +++ 1 file changed, 7 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index f6dcd1233935..ee0e7cbca5c6 100644 --- a/MAINTAINERS ++

[PATCH v2 07/10] mm/memory_hotplug: Introduce offline_and_remove_memory()

2020-03-11 Thread David Hildenbrand
vices (esp. find_memory_block() and device_offline()) and the device hotplug lock. To keep things simple, allow to only work on a single memory block. Cc: Andrew Morton Cc: David Hildenbrand Cc: Oscar Salvador Cc: Michal Hocko Cc: Pavel Tatashin Cc: Wei Yang Cc: Dan Williams Cc: Qian Cai Sig

[PATCH v2 09/10] virtio-mem: Better retry handling

2020-03-11 Thread David Hildenbrand
Cc: Stefan Hajnoczi Cc: Vlastimil Babka Signed-off-by: David Hildenbrand --- drivers/virtio/virtio_mem.c | 11 --- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index aa322e7732a4..48e96702d4ce 100644

[PATCH v2 08/10] virtio-mem: Offline and remove completely unplugged memory blocks

2020-03-11 Thread David Hildenbrand
nother notifier NACKs the offlining request). Cc: "Michael S. Tsirkin" Cc: Jason Wang Cc: Oscar Salvador Cc: Michal Hocko Cc: Igor Mammedov Cc: Dave Young Cc: Andrew Morton Cc: Dan Williams Cc: Pavel Tatashin Cc: Stefan Hajnoczi Cc: Vlastimil Babka Signed-off-by: David Hildenb

Re: [PATCH v2 07/10] mm/memory_hotplug: Introduce offline_and_remove_memory()

2020-03-11 Thread David Hildenbrand
On 11.03.20 18:14, David Hildenbrand wrote: > virtio-mem wants to offline and remove a memory block once it unplugged > all subblocks (e.g., using alloc_contig_range()). Let's provide > an interface to do that from a driver. virtio-mem already supports to > offline partially

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-03-12 Thread David Hildenbrand
On 12.03.20 08:49, Hui Zhu wrote: > If the guest kernel has many fragmentation pages, use virtio_balloon > will split THP of QEMU when it calls MADV_DONTNEED madvise to release > the balloon pages. > This is an example in a VM with 1G memory 1CPU: > cat /proc/meminfo | grep AnonHugePages: > AnonHug

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-03-12 Thread David Hildenbrand
On 12.03.20 09:47, Michael S. Tsirkin wrote: > On Thu, Mar 12, 2020 at 09:37:32AM +0100, David Hildenbrand wrote: >> 2. You are essentially stealing THPs in the guest. So the fastest >> mapping (THP in guest and host) is gone. The guest won't be able to make >> use of T

Re: [PATCH for QEMU v2] virtio-balloon: Add option cont-pages to set VIRTIO_BALLOON_VQ_INFLATE_CONT

2020-03-26 Thread David Hildenbrand
> Am 26.03.2020 um 08:06 schrieb teawater : > > Ping. > On paid leave this week. Will try to have a look next week, but it could take a bit longer. Cheers > Thanks, > Hui > >> 2020年3月23日 00:04,Hui Zhu 写道: >> >> If the guest kernel has many fragmentation pages, use virtio_balloon >> will

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-03-26 Thread David Hildenbrand
> Am 26.03.2020 um 08:21 schrieb Michael S. Tsirkin : > > On Thu, Mar 12, 2020 at 09:51:25AM +0100, David Hildenbrand wrote: >>> On 12.03.20 09:47, Michael S. Tsirkin wrote: >>> On Thu, Mar 12, 2020 at 09:37:32AM +0100, David Hildenbrand wrote: >>>> 2. Y

Re: [PATCH v2 00/10] virtio-mem: paravirtualized memory

2020-03-27 Thread David Hildenbrand
On 27.03.20 17:58, Pankaj Gupta wrote: > Hi David, > > Trying to test the series with the Qemu branch(virtio-mem) mentioned. > Unfortunately, > not able to hotplug memory. Is anything changed from your previous posting > or I am doing something wrong? > > After giving value to "requested-size", I

Re: [PATCH v2 00/10] virtio-mem: paravirtualized memory

2020-03-27 Thread David Hildenbrand
On 27.03.20 18:03, David Hildenbrand wrote: > On 27.03.20 17:58, Pankaj Gupta wrote: >> Hi David, >> >> Trying to test the series with the Qemu branch(virtio-mem) mentioned. >> Unfortunately, >> not able to hotplug memory. Is anything changed from your previous p

Re: [PATCH v2 00/10] virtio-mem: paravirtualized memory

2020-03-30 Thread David Hildenbrand
On 29.03.20 14:42, Michael S. Tsirkin wrote: > On Wed, Mar 11, 2020 at 06:14:12PM +0100, David Hildenbrand wrote: >> This series is based on latest linux-next. The patches are located at: >> https://github.com/davidhildenbrand/linux.git virtio-mem-v2 >> >> I now ha

Re: [PATCH v2 00/10] virtio-mem: paravirtualized memory

2020-03-30 Thread David Hildenbrand
On 29.03.20 17:41, Pankaj Gupta wrote: >>> Hi David, >>> >>> Trying to test the series with the Qemu branch(virtio-mem) mentioned. >>> Unfortunately, >>> not able to hotplug memory. Is anything changed from your previous posting >>> or I am doing something wrong? >>> >>> After giving value to "requ

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-03-31 Thread David Hildenbrand
On 26.03.20 10:49, Michael S. Tsirkin wrote: > On Thu, Mar 26, 2020 at 08:54:04AM +0100, David Hildenbrand wrote: >> >> >>> Am 26.03.2020 um 08:21 schrieb Michael S. Tsirkin : >>> >>> On Thu, Mar 12, 2020 at 09:51:25AM +0100, David Hildenbrand wrote: &

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-03-31 Thread David Hildenbrand
On 31.03.20 15:24, Michael S. Tsirkin wrote: > On Tue, Mar 31, 2020 at 12:35:24PM +0200, David Hildenbrand wrote: >> On 26.03.20 10:49, Michael S. Tsirkin wrote: >>> On Thu, Mar 26, 2020 at 08:54:04AM +0100, David Hildenbrand wrote: >>>> >>>> >>>&

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-03-31 Thread David Hildenbrand
On 31.03.20 15:37, Michael S. Tsirkin wrote: > On Tue, Mar 31, 2020 at 03:32:05PM +0200, David Hildenbrand wrote: >> On 31.03.20 15:24, Michael S. Tsirkin wrote: >>> On Tue, Mar 31, 2020 at 12:35:24PM +0200, David Hildenbrand wrote: >>>> On 26.03.20 10:49, Michael

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-03-31 Thread David Hildenbrand
On 31.03.20 16:07, Michael S. Tsirkin wrote: > On Tue, Mar 31, 2020 at 04:03:18PM +0200, David Hildenbrand wrote: >> On 31.03.20 15:37, Michael S. Tsirkin wrote: >>> On Tue, Mar 31, 2020 at 03:32:05PM +0200, David Hildenbrand wrote: >>>> On 31.03.20 15:24, Michael

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-03-31 Thread David Hildenbrand
On 31.03.20 16:18, Michael S. Tsirkin wrote: > On Tue, Mar 31, 2020 at 04:09:59PM +0200, David Hildenbrand wrote: > > ... > >>>>>>>>>>>>> So if we want to address this, IMHO this calls for a new API. >>>>>>>>>>&g

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-03-31 Thread David Hildenbrand
On 31.03.20 16:29, David Hildenbrand wrote: > On 31.03.20 16:18, Michael S. Tsirkin wrote: >> On Tue, Mar 31, 2020 at 04:09:59PM +0200, David Hildenbrand wrote: >> >> ... >> >>>>>>>>>>>>>> So if we want to address t

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-04-01 Thread David Hildenbrand
On 31.03.20 18:37, Nadav Amit wrote: >> On Mar 31, 2020, at 7:09 AM, David Hildenbrand wrote: >> >> On 31.03.20 16:07, Michael S. Tsirkin wrote: >>> On Tue, Mar 31, 2020 at 04:03:18PM +0200, David Hildenbrand wrote: >>>> On 31.03.20 15:37, Michael S. Tsirk

Re: [RFC for Linux] virtio_balloon: Add VIRTIO_BALLOON_F_THP_ORDER to handle THP spilt issue

2020-04-01 Thread David Hildenbrand
On 31.03.20 18:27, Nadav Amit wrote: >> On Mar 31, 2020, at 6:32 AM, David Hildenbrand wrote: >> >> On 31.03.20 15:24, Michael S. Tsirkin wrote: >>> On Tue, Mar 31, 2020 at 12:35:24PM +0200, David Hildenbrand wrote: >>>> On 26.03.20 10:49, Michael S. Tsirk

Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

2016-12-06 Thread David Hildenbrand
Am 30.11.2016 um 09:43 schrieb Liang Li: This patch set contains two parts of changes to the virtio-balloon. One is the change for speeding up the inflating & deflating process, the main idea of this optimization is to use bitmap to send the page information to host instead of the PFNs, to reduc

Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

2016-12-07 Thread David Hildenbrand
Am 07.12.2016 um 14:35 schrieb Li, Liang Z: Am 30.11.2016 um 09:43 schrieb Liang Li: This patch set contains two parts of changes to the virtio-balloon. One is the change for speeding up the inflating & deflating process, the main idea of this optimization is to use bitmap to send the page info

Re: [PATCH kernel v5 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

2016-12-07 Thread David Hildenbrand
I did something similar. Filled the balloon with 15GB for a 16GB idle guest, by using bitmap, the madvise count was reduced to 605. when using the PFNs, the madvise count was 3932160. It means there are quite a lot consecutive bits in the bitmap. I didn't test for a guest with heavy memory worklo

Re: [PATCH v6 kernel 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

2017-01-18 Thread David Hildenbrand
Am 21.12.2016 um 07:52 schrieb Liang Li: This patch set contains two parts of changes to the virtio-balloon. One is the change for speeding up the inflating & deflating process, the main idea of this optimization is to use {pfn|length} to present the page information instead of the PFNs, to redu

Re: [PATCH v6 kernel 0/5] Extend virtio-balloon for fast (de)inflating & fast live migration

2017-01-19 Thread David Hildenbrand
> As long as the interface is similar, it seems to make > sense for me - why invent a completely new device that > looks very much like the old one? The only reason would be that this feature could be used independently of virtio-balloon. But this would of course only be the case, if ballooning i

Re: [PATCH v7 kernel 5/5] This patch contains two parts:

2017-03-06 Thread David Hildenbrand
Am 03.03.2017 um 06:40 schrieb Wei Wang: > From: Liang Li I'd prefer to split this into two parts then and to create proper subjects. If I remember correctly, the general concept was accepted by most reviewers. > > One is to add a new API to mm go get the unused page information. > The virtio

Re: [virtio-dev] Re: [PATCH v7 kernel 3/5] virtio-balloon: implementation of VIRTIO_BALLOON_F_CHUNK_TRANSFER

2017-03-10 Thread David Hildenbrand
; This optimization requires the negotation of a new feature bit, >>> VIRTIO_BALLOON_F_CHUNK_TRANSFER. >>> >>> With this new feature, the above ballooning process takes ~590ms >>> resulting in an improvement of ~85%. >>> >>> TODO: optimize stage 1

Re: [PATCH] virtio_balloon: prevent uninitialized variable use

2017-03-24 Thread David Hildenbrand
On 23.03.2017 16:17, Arnd Bergmann wrote: > The latest gcc-7.0.1 snapshot reports a new warning: > > virtio/virtio_balloon.c: In function 'update_balloon_stats': > virtio/virtio_balloon.c:258:26: error: 'events[2]' is used uninitialized in > this function [-Werror=uninitialized] > virtio/virtio_b

[RFC] virtio-mem: paravirtualized memory

2017-06-16 Thread David Hildenbrand
Hi, this is an idea that is based on Andrea Arcangeli's original idea to host enforce guest access to memory given up using virtio-balloon using userfaultfd in the hypervisor. While looking into the details, I realized that host-enforcing virtio-balloon would result in way too many problems (mainl

Re: [RFC] virtio-mem: paravirtualized memory

2017-06-16 Thread David Hildenbrand
On 16.06.2017 17:04, Michael S. Tsirkin wrote: > On Fri, Jun 16, 2017 at 04:20:02PM +0200, David Hildenbrand wrote: >> Hi, >> >> this is an idea that is based on Andrea Arcangeli's original idea to >> host enforce guest access to memory given up using virtio-bal

Re: [RFC] virtio-mem: paravirtualized memory

2017-06-18 Thread David Hildenbrand
>> A Linux guest will deflate the balloon (all or some pages) in the >> following scenarios: >> a) page migration > > It inflates it first, doesn't it? Yes, that that is true. I was just listing all scenarios. > >> b) unload virtio-balloon kernel module >> c) hibernate/suspension >> d) (DEFLATE

Re: [RFC] virtio-mem: paravirtualized memory

2017-06-19 Thread David Hildenbrand
On 19.06.2017 12:08, Stefan Hajnoczi wrote: > On Fri, Jun 16, 2017 at 04:20:02PM +0200, David Hildenbrand wrote: >> Important restrictions of this concept: >> - Guests without a virtio-mem guest driver can't see that memory. >> - We will always require some boot memory

Re: [PATCH v11 4/6] mm: function to offer a page block on the free list

2017-06-20 Thread David Hildenbrand
On 20.06.2017 18:44, Rik van Riel wrote: > On Mon, 2017-06-12 at 07:10 -0700, Dave Hansen wrote: > >> The hypervisor is going to throw away the contents of these pages, >> right? As soon as the spinlock is released, someone can allocate a >> page, and put good data in it. What keeps the hypervis

Re: [PATCH v11 4/6] mm: function to offer a page block on the free list

2017-06-20 Thread David Hildenbrand
On 20.06.2017 20:17, Michael S. Tsirkin wrote: > On Tue, Jun 20, 2017 at 06:49:33PM +0200, David Hildenbrand wrote: >> On 20.06.2017 18:44, Rik van Riel wrote: >>> On Mon, 2017-06-12 at 07:10 -0700, Dave Hansen wrote: >>> >>>> The hypervisor is going to

Re: [PATCH v11 4/6] mm: function to offer a page block on the free list

2017-06-20 Thread David Hildenbrand
>> IMHO even simply writing all-zeros to all free pages before starting >> migration (or even when freeing a page) would be a cleaner interface >> than this (because it atomically works with the entity the host cares >> about for migration). But yes, performance is horrible that's why I am >> not

Re: [RFC] virtio-mem: paravirtualized memory

2017-06-21 Thread David Hildenbrand
On 21.06.2017 13:08, Stefan Hajnoczi wrote: > On Mon, Jun 19, 2017 at 12:26:52PM +0200, David Hildenbrand wrote: >> On 19.06.2017 12:08, Stefan Hajnoczi wrote: >>> On Fri, Jun 16, 2017 at 04:20:02PM +0200, David Hildenbrand wrote: >>>> Important restrictions of this

Re: [PATCH v11 4/6] mm: function to offer a page block on the free list

2017-06-21 Thread David Hildenbrand
On 21.06.2017 14:56, Christian Borntraeger wrote: > On 06/20/2017 06:49 PM, David Hildenbrand wrote: >> On 20.06.2017 18:44, Rik van Riel wrote: >>> On Mon, 2017-06-12 at 07:10 -0700, Dave Hansen wrote: >>> >>>> The hypervisor is going to throw away the c

Re: [RFC] virtio-mem: paravirtualized memory

2017-07-25 Thread David Hildenbrand
(ping) Hi, this has been on these lists for quite some time now. I want to start preparing a virtio spec for virtio-mem soon. So if you have any more comments/ideas/objections/questions, now is the right time to post them :) Thanks! On 16.06.2017 16:20, David Hildenbrand wrote: >

Re: [RFC] virtio-mem: paravirtualized memory

2017-07-28 Thread David Hildenbrand
017 16:20, David Hildenbrand wrote: > Hi, > > this is an idea that is based on Andrea Arcangeli's original idea to > host enforce guest access to memory given up using virtio-balloon using > userfaultfd in the hypervisor. While looking into the details, I > realized that

Re: [RFC] virtio-mem: paravirtualized memory

2017-07-28 Thread David Hildenbrand
On 28.07.2017 17:16, Dan Williams wrote: > On Fri, Jul 28, 2017 at 4:09 AM, David Hildenbrand wrote: >> Btw, I am thinking about the following addition to the concept: >> >> 1. Add a type to each virtio-mem device. >> >> This describes the type of the mem

Re: [RFC] virtio-mem: paravirtualized memory

2017-07-31 Thread David Hildenbrand
On 31.07.2017 16:12, Michael S. Tsirkin wrote: > On Fri, Jul 28, 2017 at 05:48:07PM +0200, David Hildenbrand wrote: >> In general, a paravirtualized interface (for detection of PMEM regions) >> might have one big advantage: not limited to certain architectures. > > What foll

Re: [PATCH RESEND] mm: don't zero ballooned pages

2017-08-07 Thread David Hildenbrand
On 03.08.2017 13:59, Wei Wang wrote: > This patch is a revert of 'commit bb01b64cfab7 ("mm/balloon_compaction.c: > enqueue zero page to balloon device")' > > Ballooned pages will be marked as MADV_DONTNEED by the hypervisor and > shouldn't be given to the host ksmd to scan. Therefore, it is not >

Re: [PATCH RESEND] mm: don't zero ballooned pages

2017-08-07 Thread David Hildenbrand
> Maybe it is my absolute lack of familiarity with what the host actually > does with balloon pages but I fail to see why the above matters at all. > ksm will not try to merge sub page units (4k for hugetlb or a large base > page). And if you need to hide the guest contents then the host can > clea

Re: [PATCH] drivers/s390/virtio: Remove the old KVM virtio transport

2017-09-28 Thread David Hildenbrand
--- > 3 files changed, 531 deletions(-) > delete mode 100644 drivers/s390/virtio/kvm_virtio.c Yup, looks good to me. Reviewed-by: David Hildenbrand -- Thanks, David ___ Virtualization mai

Re: [PATCH 1/1] virtio/s390: implement PM operations for virtio_ccw

2017-12-07 Thread David Hildenbrand
_notify, > .int_class = IRQIO_VIR, > + .freeze = virtio_ccw_freeze, > + .thaw = virtio_ccw_restore, > + .restore = virtio_ccw_restore, > }; > > static int __init pure_hex(char **cp, unsigned int *val, int min_digit, > Really neat, hope we hav

Re: [GIT PULL v2] vhost: cleanups and fixes

2020-04-07 Thread David Hildenbrand
> virtio: fixes, vdpa > > Some bug fixes. > The new vdpa subsystem with two first drivers. > > Signed-off-by: Michael S. Tsirkin > > > David Hildenbrand (1): > virtio-balloon: Switch back to OOM handler for > VIRTIO_BALLOON_F_DE

Re: [PATCH v2 00/10] virtio-mem: paravirtualized memory

2020-04-14 Thread David Hildenbrand
On 29.03.20 14:42, Michael S. Tsirkin wrote: > On Wed, Mar 11, 2020 at 06:14:12PM +0100, David Hildenbrand wrote: >> This series is based on latest linux-next. The patches are located at: >> https://github.com/davidhildenbrand/linux.git virtio-mem-v2 >> >> I now ha

Re: [PATCH v2 00/10] virtio-mem: paravirtualized memory

2020-04-14 Thread David Hildenbrand
On 14.04.20 18:28, Michael S. Tsirkin wrote: > On Tue, Apr 14, 2020 at 11:15:18AM +0200, David Hildenbrand wrote: >> On 29.03.20 14:42, Michael S. Tsirkin wrote: >>> On Wed, Mar 11, 2020 at 06:14:12PM +0100, David Hildenbrand wrote: >>>> This series is based on late

Re: [PATCH] virtio-balloon: Avoid using the word 'report' when referring to free page hinting

2020-04-15 Thread David Hildenbrand
On 15.04.20 19:44, Alexander Duyck wrote: > From: Alexander Duyck > > It can be confusing to have multiple features within the same driver that > are using the same verbage. As such this patch is creating a union of > free_page_report_cmd_id with free_page_hint_cmd_id so that we can clean-up > th

Re: [PATCH v2] virtio-balloon: Avoid using the word 'report' when referring to free page hinting

2020-04-15 Thread David Hildenbrand
+ * need to carry that name for legacy support. > + */ > + union { > + __u32 free_page_hint_cmd_id; > + __u32 free_page_report_cmd_id; /* deprecated */ > + }; > /* Stores PAGE_POISON if page poisoning is in use */ > __u32 poison

Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-17 Thread David Hildenbrand
On 16.04.20 21:30, Alexander Duyck wrote: > From: Alexander Duyck > > If we have free page hinting or page reporting enabled we should disable it > if the pages are poisoned or initialized on free and we cannot notify the > hypervisor. > Please split that up into an actual fix (Fixes:) for free

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-17 Thread David Hildenbrand
On 17.04.20 08:28, Michael S. Tsirkin wrote: > On Thu, Apr 16, 2020 at 04:52:42PM -0700, Alexander Duyck wrote: >> On Thu, Apr 16, 2020 at 3:13 PM Michael S. Tsirkin wrote: >>> >>> On Thu, Apr 16, 2020 at 12:30:38PM -0700, Alexander Duyck wrote: From: Alexander Duyck If we have fre

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-17 Thread David Hildenbrand
On 17.04.20 10:50, Michael S. Tsirkin wrote: > On Fri, Apr 17, 2020 at 09:49:03AM +0200, David Hildenbrand wrote: >> On 17.04.20 08:28, Michael S. Tsirkin wrote: >>> On Thu, Apr 16, 2020 at 04:52:42PM -0700, Alexander Duyck wrote: >>>> On Thu, Apr 16, 2020 at 3:1

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-17 Thread David Hildenbrand
>>> I'll need to think about this. >>> We need to remember that the whole HINT thing is not a mandate for host to >>> corrupt memory. It's just some info about page use guest >>> gives host. If host corrupts memory it's broken ... >> >> I don't think that's true, > > Do you refer to "If host cor

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-17 Thread David Hildenbrand
> What do you call "hinting ends" though? The fact we put > a page in the VQ is not a guarantee that it's been consumed > by the hypervisor. > I'd say hinting ends once the hypervisor sets FREE_PAGE_REPORT_S_DONE. > > I think a strict definition is this: > - hint includes a command ID > - hint

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-17 Thread David Hildenbrand
On 17.04.20 12:19, Michael S. Tsirkin wrote: > On Fri, Apr 17, 2020 at 12:09:38PM +0200, David Hildenbrand wrote: >> > What do you call "hinting ends" though? The fact we put >>> a page in the VQ is not a guarantee that it's been consumed >>> by the hy

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-17 Thread David Hildenbrand
On 17.04.20 12:29, Michael S. Tsirkin wrote: > On Fri, Apr 17, 2020 at 12:26:24PM +0200, David Hildenbrand wrote: >> On 17.04.20 12:19, Michael S. Tsirkin wrote: >>> On Fri, Apr 17, 2020 at 12:09:38PM +0200, David Hildenbrand wrote: >>>> > What do you call "

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-17 Thread David Hildenbrand
On 17.04.20 13:02, Michael S. Tsirkin wrote: > On Fri, Apr 17, 2020 at 12:31:14PM +0200, David Hildenbrand wrote: >> On 17.04.20 12:29, Michael S. Tsirkin wrote: >>> On Fri, Apr 17, 2020 at 12:26:24PM +0200, David Hildenbrand wrote: >>>> On 17.04.20 12:19, Michael

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-20 Thread David Hildenbrand
>>> Now we are talking about what's safe to do with the page. >>> >>> If POISON flag is set by hypervisor but clear by guest, >>> or poison_val is 0, then it's clear it's safe to blow >>> away the page if we can figure out it's unused. >>> >>> Otherwise, it's much less clear :) >> >> Hah! Agreed :D

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-21 Thread David Hildenbrand
>>> 2. Can we assume that the guest will always rewrite the page after the >>> deflate in the case of init_on_free or poison? >> >> Depends on what we think is the right way to do - IOW if we think "some >> other content" as mentioned above is a BUG or not. > > So I wouldn't consider it a but as

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-21 Thread David Hildenbrand
>> >> That leaves us with either >> >> 1. "Pages hinted via VIRTIO_BALLOON_F_FREE_PAGE_HINT might get replaced >> by zero page. However, as soon as the page is written by the guest (even >> before the hinting request was processed by the host), the modified page >> will stay - whereby the unwritten

Re: [virtio-dev] Re: [PATCH] virtio-balloon: Disable free page hinting/reporting if page poison is disabled

2020-04-22 Thread David Hildenbrand
>>> What is the expectation there? I assume we are saying either >>> poison_val or unmodified? If so I would think the inflate case makes >>> much more sense as that is where the madvise is called that will >>> discard the data. If so it would be pretty easy to just add a check >>> for the poison v

Re: [PATCH v2] virtio-balloon: Disable free page reporting if page poison reporting is not enabled

2020-04-27 Thread David Hildenbrand
On 24.04.20 18:24, Alexander Duyck wrote: > From: Alexander Duyck > > We should disable free page reporting if page poisoning is enabled in the > kernel but we cannot report it via the balloon interface. This way we can > avoid the possibility of corrupting guest memory. Normally the page poison

Re: [PATCH v2] virtio-balloon: Disable free page reporting if page poison reporting is not enabled

2020-04-27 Thread David Hildenbrand
On 27.04.20 16:58, Alexander Duyck wrote: > On Mon, Apr 27, 2020 at 1:08 AM David Hildenbrand wrote: >> >> On 24.04.20 18:24, Alexander Duyck wrote: >>> From: Alexander Duyck >>> >>> We should disable free page reporting if page poisoning is enabled in

Re: [PATCH v2] virtio-balloon: Disable free page reporting if page poison reporting is not enabled

2020-04-27 Thread David Hildenbrand
courage you to document the semantics of free page reporting properly - especially, that we decided that the content of a reported page with VIRTIO_BALLOON_F_REPORTING && !VIRTIO_BALLOON_F_PAGE_POISON is *undefined* and needs a reinit. Only with VIRTIO_BALLOON_F_REPORTING && VIRTIO_BALLOON_F_PAGE_POISON the content is either the old page content or a page filled with poison_val. So take my Acked-by: David Hildenbrand -- Thanks, David / dhildenb ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH v1 3/3] virtio-mem: Add memory with MHP_DRIVER_MANAGED

2020-04-29 Thread David Hildenbrand
(Reserved) 0001-00014000 (System RAM) Cc: "Michael S. Tsirkin" Cc: Jason Wang Cc: Michal Hocko Cc: Eric Biederman Signed-off-by: David Hildenbrand --- drivers/virtio/virtio_mem.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/virt

[PATCH v1 0/3] mm/memory_hotplug: Make virtio-mem play nicely with kexec-tools

2020-04-29 Thread David Hildenbrand
4754-1-james.mo...@arm.com David Hildenbrand (3): mm/memory_hotplug: Prepare passing flags to add_memory() and friends mm/memory_hotplug: Introduce MHP_DRIVER_MANAGED virtio-mem: Add memory with MHP_DRIVER_MANAGED arch/powerpc/platforms/powernv/memtrace.c | 2 +- .../platforms/pseries/h

[PATCH v1 1/3] mm/memory_hotplug: Prepare passing flags to add_memory() and friends

2020-04-29 Thread David Hildenbrand
..@vger.kernel.org Cc: linux-a...@vger.kernel.org Cc: linux-nvd...@lists.01.org Cc: linux-hyp...@vger.kernel.org Cc: linux-s...@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: xen-de...@lists.xenproject.org Signed-off-by: David Hildenbrand --- arch/powerpc/platforms/powernv/memtrace

<    1   2   3   4   5   6   7   8   9   >