[PATCH 0/4] make balloon pages movable by compaction

2012-06-25 Thread Rafael Aquini
balloon pages become movable within a guest. Rafael Aquini (4): mm: introduce compaction and migration for virtio ballooned pages virtio_balloon: handle concurrent accesses to virtio_balloon struct elements virtio_balloon: introduce migration primitives to balloon pages mm: add vm event

[PATCH 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-06-25 Thread Rafael Aquini
This patch introduces helper functions that teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux/mm.h | 17

[PATCH 2/4] virtio_balloon: handle concurrent accesses to virtio_balloon struct elements

2012-06-25 Thread Rafael Aquini
This patch introduces access sychronization to critical elements of struct virtio_balloon, in order to allow the thread concurrency compaction/migration bits might ended up imposing to the balloon driver on several situations. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio

[PATCH 3/4] virtio_balloon: introduce migration primitives to balloon pages

2012-06-25 Thread Rafael Aquini
This patch makes balloon pages movable at allocation time and introduces the infrastructure needed to perform the balloon page migration operation. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 96 ++- include/linux

[PATCH 4/4] mm: add vm event counters for balloon pages compaction

2012-06-25 Thread Rafael Aquini
This patch is only for testing report purposes and shall be dropped in case of the rest of this patchset getting accepted for merging. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c |1 + include/linux/vm_event_item.h |2 ++ mm/compaction.c

Re: [PATCH 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-06-25 Thread Rafael Aquini
On Mon, Jun 25, 2012 at 07:31:38PM -0400, Konrad Rzeszutek Wilk wrote: On Mon, Jun 25, 2012 at 7:25 PM, Rafael Aquini aqu...@redhat.com wrote: This patch introduces helper functions that teach compaction and migration bits how to cope with pages which are part of a guest memory balloon

Re: [PATCH 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-06-26 Thread Rafael Aquini
Mel, First and foremost, thank you for taking the time to review these bits and provide such valuable feedback. On Tue, Jun 26, 2012 at 11:17:29AM +0100, Mel Gorman wrote: +/* return 1 if page is part of a guest's memory balloon, 0 otherwise */ +static inline int PageBalloon(struct page

[PATCH v2 2/4] virtio_balloon: handle concurrent accesses to virtio_balloon struct elements

2012-06-28 Thread Rafael Aquini
This patch introduces access sychronization to critical elements of struct virtio_balloon, in order to allow the thread concurrency compaction/migration bits might ended up imposing to the balloon driver on several situations. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio

[PATCH v2 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-06-28 Thread Rafael Aquini
This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com

[PATCH v2 0/4] make balloon pages movable by compaction

2012-06-28 Thread Rafael Aquini
balloon pages become movable within a guest. Rafael Aquini (4): mm: introduce compaction and migration for virtio ballooned pages virtio_balloon: handle concurrent accesses to virtio_balloon struct elements virtio_balloon: introduce migration primitives to balloon pages mm: add vm event

[PATCH v2 3/4] virtio_balloon: introduce migration primitives to balloon pages

2012-06-28 Thread Rafael Aquini
This patch makes balloon pages movable at allocation time and introduces the infrastructure needed to perform the balloon page migration operation. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 96 ++- include/linux

[PATCH v2 4/4] mm: add vm event counters for balloon pages compaction

2012-06-28 Thread Rafael Aquini
This patch is only for testing report purposes and shall be dropped in case of the rest of this patchset getting accepted for merging. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c |1 + include/linux/vm_event_item.h |2 ++ mm/compaction.c

Re: [PATCH v2 0/4] make balloon pages movable by compaction

2012-06-28 Thread Rafael Aquini
On Fri, Jun 29, 2012 at 10:37:12AM +0900, Minchan Kim wrote: Hi Rafael, On 06/29/2012 06:49 AM, Rafael Aquini wrote: This patchset follows the main idea discussed at 2012 LSFMMS section: Ballooning for transparent huge pages -- http://lwn.net/Articles/490114/ Could you summarize

Re: [PATCH v2 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-06-29 Thread Rafael Aquini
On Fri, Jun 29, 2012 at 02:32:11PM +0900, Minchan Kim wrote: On 06/29/2012 06:49 AM, Rafael Aquini wrote: This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon

Re: [PATCH v2 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-06-29 Thread Rafael Aquini
On Fri, Jun 29, 2012 at 04:31:57PM +0100, Mel Gorman wrote: On Thu, Jun 28, 2012 at 06:49:39PM -0300, Rafael Aquini wrote: This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest

Re: [PATCH v2 0/4] make balloon pages movable by compaction

2012-06-29 Thread Rafael Aquini
On Fri, Jun 29, 2012 at 02:01:52PM +0930, Rusty Russell wrote: On Thu, 28 Jun 2012 18:49:38 -0300, Rafael Aquini aqu...@redhat.com wrote: This patchset follows the main idea discussed at 2012 LSFMMS section: Ballooning for transparent huge pages -- http://lwn.net/Articles/490114

Re: [PATCH v2 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-06-29 Thread Rafael Aquini
Howdy Minchan, On Sat, Jun 30, 2012 at 07:03:33AM +0900, Minchan Kim wrote: +static inline bool is_balloon_page(struct page *page) +{ +return (page-mapping == balloon_mapping) ? true : false; +} What lock should it protect? I'm afraid I didn't quite get

Re: RFD: virtio balloon API use (was Re: [PATCH 5 of 5] virtio: expose added descriptors immediately)

2012-07-02 Thread Rafael Aquini
On Mon, Jul 02, 2012 at 10:25:58AM +0300, Michael S. Tsirkin wrote: On Mon, Jul 02, 2012 at 10:35:47AM +0930, Rusty Russell wrote: On Sun, 1 Jul 2012 12:20:51 +0300, Michael S. Tsirkin m...@redhat.com wrote: On Thu, Nov 03, 2011 at 06:12:53PM +1030, Rusty Russell wrote: A virtio

Re: RFD: virtio balloon API use (was Re: [PATCH 5 of 5] virtio: expose added descriptors immediately)

2012-07-03 Thread Rafael Aquini
On Tue, Jul 03, 2012 at 10:17:46AM +0930, Rusty Russell wrote: On Mon, 2 Jul 2012 13:08:19 -0300, Rafael Aquini aqu...@redhat.com wrote: As 'locking in balloon', may I assume the approach I took for the compaction case is OK and aligned to address these concerns of yours? If not, do

Re: [PATCH v2 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-07-03 Thread Rafael Aquini
On Mon, Jul 02, 2012 at 08:36:02AM +0900, Minchan Kim wrote: On 06/30/2012 10:34 AM, Rafael Aquini wrote: void isolate_page_from_balloonlist(struct page* page) { page-mapping-a_ops-invalidatepage(page, 0); } if (is_balloon_page(page) (page_count(page) == 2

[PATCH v3 0/4] make balloon pages movable by compaction

2012-07-03 Thread Rafael Aquini
of ballooned pages and allow memory balloon pages become movable within a guest, thus avoiding the aforementioned fragmentation issue. Rafael Aquini (4): mm: introduce compaction and migration for virtio ballooned pages virtio_balloon: handle concurrent accesses to virtio_balloon struct

[PATCH v3 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-07-03 Thread Rafael Aquini
This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com

[PATCH v3 2/4] virtio_balloon: handle concurrent accesses to virtio_balloon struct elements

2012-07-03 Thread Rafael Aquini
This patch introduces access sychronization to critical elements of struct virtio_balloon, in order to allow the thread concurrency compaction/migration bits might ended up imposing to the balloon driver on several situations. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio

[PATCH v3 3/4] virtio_balloon: introduce migration primitives to balloon pages

2012-07-03 Thread Rafael Aquini
This patch makes balloon pages movable at allocation time and introduces the infrastructure needed to perform the balloon page migration operation. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 96 ++- include/linux

[PATCH v3 4/4] mm: add vm event counters for balloon pages compaction

2012-07-03 Thread Rafael Aquini
This patch is only for testing report purposes and shall be dropped in case of the rest of this patchset getting accepted for merging. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c |1 + include/linux/vm_event_item.h |2 ++ mm/compaction.c

Re: [PATCH v3 2/4] virtio_balloon: handle concurrent accesses to virtio_balloon struct elements

2012-07-04 Thread Rafael Aquini
Howdy Rusty, First and foremost, thank you very much for taking the time to go through this proposal and provide me with such valuable feedback. I really appreciate that. On Wed, Jul 04, 2012 at 04:08:23PM +0930, Rusty Russell wrote: On Tue, 3 Jul 2012 20:48:50 -0300, Rafael Aquini aqu

[PATCH v4 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-07-17 Thread Rafael Aquini
This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com

[PATCH v4 0/3] make balloon pages movable by compaction

2012-07-17 Thread Rafael Aquini
aware of ballooned pages and allow memory balloon pages become movable within a guest, thus avoiding the aforementioned fragmentation issue Rafael Aquini (3): mm: introduce compaction and migration for virtio ballooned pages virtio_balloon: introduce migration primitives to balloon pages mm: add

[PATCH v4 2/3] virtio_balloon: introduce migration primitives to balloon pages

2012-07-17 Thread Rafael Aquini
pages list against concurrent list handling operations; Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 145 --- include/linux/virtio_balloon.h |4 ++ 2 files changed, 141 insertions(+), 8

[PATCH v4 3/3] mm: add vm event counters for balloon pages compaction

2012-07-17 Thread Rafael Aquini
This patch is only for testing report purposes and shall be dropped in case of the rest of this patchset getting accepted for merging. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c |1 + include/linux/vm_event_item.h |2 ++ mm/compaction.c

Re: [PATCH v4 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-07-18 Thread Rafael Aquini
Howdy Andrew, Thanks for taking the time to go through this work and provide me with such good feedback. On Wed, Jul 18, 2012 at 03:46:05PM -0700, Andrew Morton wrote: On Tue, 17 Jul 2012 13:50:41 -0300 Rafael Aquini aqu...@redhat.com wrote: This patch introduces the helper functions

Re: [PATCH v4 2/3] virtio_balloon: introduce migration primitives to balloon pages

2012-07-18 Thread Rafael Aquini
Howdy Andrew, Thanks for taking the time to go through this work and provide me with such good feedback. On Wed, Jul 18, 2012 at 03:49:08PM -0700, Andrew Morton wrote: On Tue, 17 Jul 2012 13:50:42 -0300 Rafael Aquini aqu...@redhat.com wrote: Besides making balloon pages movable

Re: [PATCH v4 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-07-18 Thread Rafael Aquini
On Wed, Jul 18, 2012 at 04:12:39PM -0700, Andrew Morton wrote: On Wed, 18 Jul 2012 20:07:07 -0300 Rafael Aquini aqu...@redhat.com wrote: +} +#else +static inline bool is_balloon_page(struct page *page) { return false; } +static inline bool isolate_balloon_page

Re: [PATCH v4 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-07-20 Thread Rafael Aquini
Howdy Minchan, Once again, thanks for raising such valuable feedback over here. On Wed, Jul 18, 2012 at 02:48:24PM +0900, Minchan Kim wrote: +/* __isolate_lru_page() counterpart for a ballooned page */ +static bool isolate_balloon_page(struct page *page) +{ + if

Re: [PATCH v4 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-07-23 Thread Rafael Aquini
On Mon, Jul 23, 2012 at 11:33:32AM +0900, Minchan Kim wrote: Hi Rafael, On Fri, Jul 20, 2012 at 04:48:59PM -0300, Rafael Aquini wrote: Howdy Minchan, Once again, thanks for raising such valuable feedback over here. On Wed, Jul 18, 2012 at 02:48:24PM +0900, Minchan Kim wrote

Re: [PATCH v4 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-08-03 Thread Rafael Aquini
On Mon, Jul 23, 2012 at 11:33:32AM +0900, Minchan Kim wrote: Look at memory-hotplug, offline_page calls has_unmovable_pages, scan_lru_pages and do_migrate_range which calls isolate_lru_page. They consider only LRU pages to migratable ones. As promised, I looked into those bits. Yes, they

[PATCH v5 2/3] virtio_balloon: introduce migration primitives to balloon pages

2012-08-06 Thread Rafael Aquini
against concurrent list handling operations; Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 138 +--- include/linux/virtio_balloon.h | 4 ++ 2 files changed, 134 insertions(+), 8 deletions(-) diff

[PATCH v5 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-08-06 Thread Rafael Aquini
. This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux

[PATCH v5 0/3] make balloon pages movable by compaction

2012-08-06 Thread Rafael Aquini
aware of ballooned pages and allow memory balloon pages become movable within a guest, thus avoiding the aforementioned fragmentation issue Rafael Aquini (3): mm: introduce compaction and migration for virtio ballooned pages virtio_balloon: introduce migration primitives to balloon pages mm: add

[PATCH v5 3/3] mm: add vm event counters for balloon pages compaction

2012-08-06 Thread Rafael Aquini
This patch is only for testing report purposes and shall be dropped in case of the rest of this patchset getting accepted for merging. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 1 + include/linux/vm_event_item.h | 2 ++ mm/compaction.c

Re: [PATCH v5 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-08-06 Thread Rafael Aquini
On Mon, Aug 06, 2012 at 02:38:23PM -0400, Rik van Riel wrote: On 08/06/2012 09:56 AM, Rafael Aquini wrote: @@ -846,6 +861,21 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private, goto out; rc = __unmap_and_move(page, newpage, force

Re: [PATCH v5 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-08-06 Thread Rafael Aquini
On Mon, Aug 06, 2012 at 03:06:49PM -0400, Rik van Riel wrote: Just a plain rename would work. Ok, I will rename it. +static inline bool is_balloon_page(struct page *page) +{ +return (page-mapping page-mapping == balloon_mapping); +} As an aside, since you are only comparing

[PATCH v6 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-08-08 Thread Rafael Aquini
. This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux

[PATCH v6 3/3] mm: add vm event counters for balloon pages compaction

2012-08-08 Thread Rafael Aquini
This patch is only for testing report purposes and shall be dropped in case of the rest of this patchset getting accepted for merging. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 1 + include/linux/vm_event_item.h | 2 ++ mm/compaction.c

[PATCH v6 0/3] make balloon pages movable by compaction

2012-08-08 Thread Rafael Aquini
aware of ballooned pages and allow memory balloon pages become movable within a guest, thus avoiding the aforementioned fragmentation issue Rafael Aquini (3): mm: introduce compaction and migration for virtio ballooned pages virtio_balloon: introduce migration primitives to balloon pages mm: add

[PATCH v6 2/3] virtio_balloon: introduce migration primitives to balloon pages

2012-08-08 Thread Rafael Aquini
against concurrent list handling operations; Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 138 +--- include/linux/virtio_balloon.h | 4 ++ 2 files changed, 134 insertions(+), 8 deletions(-) diff

Re: [PATCH v6 1/3] mm: introduce compaction and migration for virtio ballooned pages

2012-08-09 Thread Rafael Aquini
On Thu, Aug 09, 2012 at 10:00:19AM +0100, Mel Gorman wrote: On Wed, Aug 08, 2012 at 07:53:19PM -0300, Rafael Aquini wrote: Memory fragmentation introduced by ballooning might reduce significantly the number of 2MB contiguous memory blocks that can be used within a guest, thus imposing

[PATCH v7 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-08-10 Thread Rafael Aquini
. This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux

[PATCH v7 0/4] make balloon pages movable by compaction

2012-08-10 Thread Rafael Aquini
aware of ballooned pages and allow memory balloon pages become movable within a guest, thus avoiding the aforementioned fragmentation issue Rafael Aquini (4): mm: introduce compaction and migration for virtio ballooned pages virtio_balloon: introduce migration primitives to balloon pages mm

[PATCH v7 2/4] virtio_balloon: introduce migration primitives to balloon pages

2012-08-10 Thread Rafael Aquini
against concurrent list handling operations; Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 138 +--- include/linux/virtio_balloon.h | 4 ++ 2 files changed, 134 insertions(+), 8 deletions(-) diff

[PATCH v7 3/4] mm: introduce putback_movable_pages()

2012-08-10 Thread Rafael Aquini
putback_movable_pages() to properly cope with cases where the isolated pageset contains ballooned pages and LRU pages, thus fixing the mentioned inelegant hack around putback_lru_pages(). Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux/migrate.h | 2 ++ mm/compaction.c

Re: [PATCH v7 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-08-14 Thread Rafael Aquini
On Mon, Aug 13, 2012 at 11:26:19AM +0300, Michael S. Tsirkin wrote: +static inline bool movable_balloon_page(struct page *page) +{ + return (page-mapping page-mapping == balloon_mapping); I am guessing this needs smp_read_barrier_depends, and maybe ACCESS_ONCE ... I'm curious about

Re: [PATCH v7 2/4] virtio_balloon: introduce migration primitives to balloon pages

2012-08-14 Thread Rafael Aquini
On Mon, Aug 13, 2012 at 11:41:23AM +0300, Michael S. Tsirkin wrote: @@ -141,7 +151,10 @@ static void fill_balloon(struct virtio_balloon *vb, size_t num) set_page_pfns(vb-pfns + vb-num_pfns, page); vb-num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;

Re: [PATCH v7 2/4] virtio_balloon: introduce migration primitives to balloon pages

2012-08-14 Thread Rafael Aquini
On Tue, Aug 14, 2012 at 11:33:20AM +0300, Michael S. Tsirkin wrote: On Tue, Aug 14, 2012 at 09:29:49AM +0930, Rusty Russell wrote: On Mon, 13 Aug 2012 11:41:23 +0300, Michael S. Tsirkin m...@redhat.com wrote: On Fri, Aug 10, 2012 at 02:55:15PM -0300, Rafael Aquini wrote

Re: [PATCH v7 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-08-14 Thread Rafael Aquini
On Tue, Aug 14, 2012 at 10:35:25PM +0300, Michael S. Tsirkin wrote: +/* __isolate_lru_page() counterpart for a ballooned page */ +bool isolate_balloon_page(struct page *page) +{ + if (WARN_ON(!movable_balloon_page(page))) Looks like this actually can happen if the page

Re: [PATCH v7 1/4] mm: introduce compaction and migration for virtio ballooned pages

2012-08-14 Thread Rafael Aquini
On Tue, Aug 14, 2012 at 10:48:37PM +0300, Michael S. Tsirkin wrote: E.g. kvm can emulate hyperv so it could in theory have hyperv balloon. This is mm stuff it is best not to tie it to specific drivers. But of course I agree this is not top priority, no need to block submission on this,

Re: [PATCH v7 2/4] virtio_balloon: introduce migration primitives to balloon pages

2012-08-14 Thread Rafael Aquini
On Tue, Aug 14, 2012 at 10:59:16PM +0300, Michael S. Tsirkin wrote: What if there is more than one balloon device? Is it possible to load this driver twice, or are you foreseeing a future case where this driver will be able to manage several distinct memory balloons for

Re: [PATCH v7 2/4] virtio_balloon: introduce migration primitives to balloon pages

2012-08-14 Thread Rafael Aquini
On Tue, Aug 14, 2012 at 11:24:01PM +0300, Michael S. Tsirkin wrote: On Tue, Aug 14, 2012 at 05:08:31PM -0300, Rafael Aquini wrote: On Tue, Aug 14, 2012 at 10:59:16PM +0300, Michael S. Tsirkin wrote: What if there is more than one balloon device? Is it possible to load

Re: [PATCH v7 2/4] virtio_balloon: introduce migration primitives to balloon pages

2012-08-14 Thread Rafael Aquini
On Tue, Aug 14, 2012 at 11:49:06PM +0300, Michael S. Tsirkin wrote: On Tue, Aug 14, 2012 at 05:29:50PM -0300, Rafael Aquini wrote: On Tue, Aug 14, 2012 at 11:24:01PM +0300, Michael S. Tsirkin wrote: On Tue, Aug 14, 2012 at 05:08:31PM -0300, Rafael Aquini wrote: On Tue, Aug 14, 2012 at 10

Re: [PATCH v7 2/4] virtio_balloon: introduce migration primitives to balloon pages

2012-08-15 Thread Rafael Aquini
On Tue, Aug 14, 2012 at 10:31:09PM +0300, Michael S. Tsirkin wrote: now CPU1 executes the next instruction: } which would normally return to function's caller, but it has been overwritten by CPU2 so we get corruption. No? At the point CPU2 is unloading the module,

[PATCH v8 0/5] make balloon pages movable by compaction

2012-08-21 Thread Rafael Aquini
aware of ballooned pages and allow memory balloon pages become movable within a guest, thus avoiding the aforementioned fragmentation issue Rafael Aquini (5): mm: introduce a common interface for balloon pages mobility mm: introduce compaction and migration for ballooned pages virtio_balloon

[PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-21 Thread Rafael Aquini
. This patch introduces a common interface to help a balloon driver on making its page set movable to compaction, and thus allowing the system to better leverage the compation efforts on memory defragmentation. Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux/balloon_compaction.h | 113

[PATCH v8 3/5] virtio_balloon: introduce migration primitives to balloon pages

2012-08-21 Thread Rafael Aquini
against concurrent list handling operations; - virtio_baloon-pages list handling sync by RCU operations; Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 210 +--- 1 file changed, 199 insertions(+), 11

[PATCH v8 2/5] mm: introduce compaction and migration for ballooned pages

2012-08-21 Thread Rafael Aquini
. This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com --- mm

[PATCH v8 4/5] mm: introduce putback_movable_pages()

2012-08-21 Thread Rafael Aquini
putback_movable_pages() to properly cope with cases where the isolated pageset contains ballooned pages and LRU pages, thus fixing the mentioned inelegant hack around putback_lru_pages(). Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux/migrate.h | 2 ++ mm/compaction.c

[PATCH v8 5/5] mm: add vm event counters for balloon pages compaction

2012-08-21 Thread Rafael Aquini
This patch introduces a new set of vm event counters to keep track of ballooned pages compaction activity. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 2 ++ include/linux/vm_event_item.h | 8 +++- mm/balloon_compaction.c | 6 -- mm

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-21 Thread Rafael Aquini
On Tue, Aug 21, 2012 at 06:41:42PM +0300, Michael S. Tsirkin wrote: On Tue, Aug 21, 2012 at 05:16:06PM +0200, Peter Zijlstra wrote: On Tue, 2012-08-21 at 16:52 +0300, Michael S. Tsirkin wrote: + rcu_read_lock(); + mapping = rcu_dereference(page-mapping); +

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-21 Thread Rafael Aquini
On Tue, Aug 21, 2012 at 04:52:23PM +0300, Michael S. Tsirkin wrote: + * address_space_operations utilized methods for ballooned pages: + * .migratepage- used to perform balloon's page migration (as is) + * .launder_page - used to isolate a page from balloon's page list + *

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-21 Thread Rafael Aquini
On Tue, Aug 21, 2012 at 09:24:32AM -0700, Paul E. McKenney wrote: On Tue, Aug 21, 2012 at 05:20:11PM +0200, Peter Zijlstra wrote: On Tue, 2012-08-21 at 09:47 -0300, Rafael Aquini wrote: + mapping = rcu_access_pointer(page-mapping); + if (mapping) + mapping

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-21 Thread Rafael Aquini
On Tue, Aug 21, 2012 at 10:13:30PM +0300, Michael S. Tsirkin wrote: I believe rcu_dereference_protected() is what I want/need here, since this code is always called for pages which we hold locked (PG_locked bit). It would only help if we locked the page while updating the mapping, as

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-21 Thread Rafael Aquini
On Tue, Aug 21, 2012 at 10:16:12PM +0300, Michael S. Tsirkin wrote: On Tue, Aug 21, 2012 at 02:55:03PM -0300, Rafael Aquini wrote: On Tue, Aug 21, 2012 at 04:52:23PM +0300, Michael S. Tsirkin wrote: + * address_space_operations utilized methods for ballooned pages: + * .migratepage

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-21 Thread Rafael Aquini
On Tue, Aug 21, 2012 at 10:30:31PM +0300, Michael S. Tsirkin wrote: On Tue, Aug 21, 2012 at 04:23:58PM -0300, Rafael Aquini wrote: On Tue, Aug 21, 2012 at 10:13:30PM +0300, Michael S. Tsirkin wrote: I believe rcu_dereference_protected() is what I want/need here, since this code

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-21 Thread Rafael Aquini
On Wed, Aug 22, 2012 at 03:07:41AM +0300, Michael S. Tsirkin wrote: On Tue, Aug 21, 2012 at 05:45:56PM -0300, Rafael Aquini wrote: On Tue, Aug 21, 2012 at 10:30:31PM +0300, Michael S. Tsirkin wrote: On Tue, Aug 21, 2012 at 04:23:58PM -0300, Rafael Aquini wrote: On Tue, Aug 21, 2012 at 10

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-22 Thread Rafael Aquini
On Wed, Aug 22, 2012 at 12:33:17PM +0300, Michael S. Tsirkin wrote: Hmm, so this will busy wait which is unelegant. We need some event IMO. No, it does not busy wait. leak_balloon() is mutual exclusive with migration steps, so for the case we have one racing against the other, we really want

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-23 Thread Rafael Aquini
On Thu, Aug 23, 2012 at 01:01:07PM +0300, Michael S. Tsirkin wrote: So, when remove_common() calls leak_balloon() looping on vb-num_pages, that won't become a tight loop. The scheme was apparently working before this series, and it will remain working after it. It seems that before

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-23 Thread Rafael Aquini
On Thu, Aug 23, 2012 at 03:34:32PM +0300, Michael S. Tsirkin wrote: So, nothing has changed here. Yes, your patch does change things: leak_balloon now might return without freeing any pages. In that case we will not be making any progress, and just spin, pinning CPU. That's a transitory

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-23 Thread Rafael Aquini
On Thu, Aug 23, 2012 at 04:53:29PM +0300, Michael S. Tsirkin wrote: On Thu, Aug 23, 2012 at 10:06:07AM -0300, Rafael Aquini wrote: On Thu, Aug 23, 2012 at 03:34:32PM +0300, Michael S. Tsirkin wrote: So, nothing has changed here. Yes, your patch does change things: leak_balloon now

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-23 Thread Rafael Aquini
On Thu, Aug 23, 2012 at 12:03:15PM -0400, Rik van Riel wrote: Not longer - apparently forever unless user resend the leak command. It's wrong - it should 1. not tell host if nothing was done 2. after migration finished leak and tell host Agreed. If the balloon is told to leak N pages,

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-23 Thread Rafael Aquini
On Thu, Aug 23, 2012 at 07:25:05PM +0300, Michael S. Tsirkin wrote: On Thu, Aug 23, 2012 at 04:53:28PM +0300, Michael S. Tsirkin wrote: Basically it was very simple: we assumed page-lru was never touched for an allocated page, so it's safe to use it for internal book-keeping by the driver.

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-23 Thread Rafael Aquini
On Fri, Aug 24, 2012 at 02:36:16AM +0300, Michael S. Tsirkin wrote: On Thu, Aug 23, 2012 at 02:28:45PM -0300, Rafael Aquini wrote: On Thu, Aug 23, 2012 at 07:25:05PM +0300, Michael S. Tsirkin wrote: On Thu, Aug 23, 2012 at 04:53:28PM +0300, Michael S. Tsirkin wrote: Basically it was very

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-23 Thread Rafael Aquini
On Fri, Aug 24, 2012 at 02:36:16AM +0300, Michael S. Tsirkin wrote: I would wake it each time after adding a page, then it can stop waiting when it leaks enough. But again, it's cleaner to just keep tracking all pages, let mm hang on to them by keeping a reference. Here is a rough idea on

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-23 Thread Rafael Aquini
On Thu, Aug 23, 2012 at 09:33:53PM -0300, Rafael Aquini wrote: On Fri, Aug 24, 2012 at 02:36:16AM +0300, Michael S. Tsirkin wrote: I would wake it each time after adding a page, then it can stop waiting when it leaks enough. But again, it's cleaner to just keep tracking all pages, let mm

Re: [PATCH v8 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-23 Thread Rafael Aquini
On Thu, Aug 23, 2012 at 09:38:48PM -0300, Rafael Aquini wrote: On Thu, Aug 23, 2012 at 09:33:53PM -0300, Rafael Aquini wrote: On Fri, Aug 24, 2012 at 02:36:16AM +0300, Michael S. Tsirkin wrote: I would wake it each time after adding a page, then it can stop waiting when it leaks enough

[PATCH v9 2/5] mm: introduce compaction and migration for ballooned pages

2012-08-24 Thread Rafael Aquini
. This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com --- mm

[PATCH v9 0/5] make balloon pages movable by compaction

2012-08-24 Thread Rafael Aquini
aware of ballooned pages and allow memory balloon pages become movable within a guest, thus avoiding the aforementioned fragmentation issue Rafael Aquini (5): mm: introduce a common interface for balloon pages mobility mm: introduce compaction and migration for ballooned pages virtio_balloon

[PATCH v9 1/5] mm: introduce a common interface for balloon pages mobility

2012-08-24 Thread Rafael Aquini
. This patch introduces a common interface to help a balloon driver on making its page set movable to compaction, and thus allowing the system to better leverage the compation efforts on memory defragmentation. Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux/balloon_compaction.h | 137

[PATCH v9 4/5] mm: introduce putback_movable_pages()

2012-08-24 Thread Rafael Aquini
putback_movable_pages() to properly cope with cases where the isolated pageset contains ballooned pages and LRU pages, thus fixing the mentioned inelegant hack around putback_lru_pages(). Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux/migrate.h | 2 ++ mm/compaction.c

[PATCH v9 3/5] virtio_balloon: introduce migration primitives to balloon pages

2012-08-24 Thread Rafael Aquini
queue operations; - pages_lock (spinlock): special protection to balloon's pages bookmarking elements (list and atomic counters) against the potential memory compaction concurrency; Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers

[PATCH v9 5/5] mm: add vm event counters for balloon pages compaction

2012-08-24 Thread Rafael Aquini
This patch introduces a new set of vm event counters to keep track of ballooned pages compaction activity. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 1 + include/linux/vm_event_item.h | 8 +++- mm/balloon_compaction.c | 2 ++ mm

Re: [PATCH v9 3/5] virtio_balloon: introduce migration primitives to balloon pages

2012-08-27 Thread Rafael Aquini
On Sun, Aug 26, 2012 at 10:42:44AM +0300, Michael S. Tsirkin wrote: Reading two atomics and doing math? Result can even be negative. I did not look at use closely but it looks suspicious. Doc on atomic_read says: The read is atomic in that the return value is guaranteed to be one of the

Re: [PATCH v9 0/5] make balloon pages movable by compaction

2012-08-27 Thread Rafael Aquini
On Sun, Aug 26, 2012 at 06:44:23PM +0300, Michael S. Tsirkin wrote: I am simply asking how was this patchset tested. It would be nice to have this info in commit log. Since this is an optimization patch it is strange to see one with no numbers at all. For example, you probably run some

Re: [PATCH v9 3/5] virtio_balloon: introduce migration primitives to balloon pages

2012-08-28 Thread Rafael Aquini
On Tue, Aug 28, 2012 at 06:54:10PM +0300, Michael S. Tsirkin wrote: On Mon, Aug 27, 2012 at 04:47:13PM -0300, Rafael Aquini wrote: On Sun, Aug 26, 2012 at 10:42:44AM +0300, Michael S. Tsirkin wrote: Reading two atomics and doing math? Result can even be negative. I did not look at use

[PATCH v10 0/5] make balloon pages movable by compaction

2012-09-17 Thread Rafael Aquini
1378 968 Compaction success 406 595 Compaction failures 972 373 Compaction pages moved 3104073 1790932 Compaction move failure 92713 41252 ===END stress-highalloc Rafael Aquini (5): mm: introduce a common interface

[PATCH v10 1/5] mm: introduce a common interface for balloon pages mobility

2012-09-17 Thread Rafael Aquini
. This patch introduces a common interface to help a balloon driver on making its page set movable to compaction, and thus allowing the system to better leverage the compation efforts on memory defragmentation. Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux/balloon_compaction.h | 147

[PATCH v10 2/5] mm: introduce compaction and migration for ballooned pages

2012-09-17 Thread Rafael Aquini
. This patch introduces the helper functions as well as the necessary changes to teach compaction and migration bits how to cope with pages which are part of a guest memory balloon, in order to make them movable by memory compaction procedures. Signed-off-by: Rafael Aquini aqu...@redhat.com --- mm

[PATCH v10 4/5] mm: introduce putback_movable_pages()

2012-09-17 Thread Rafael Aquini
putback_movable_pages() to properly cope with cases where the isolated pageset contains ballooned pages and LRU pages, thus fixing the mentioned inelegant hack around putback_lru_pages(). Signed-off-by: Rafael Aquini aqu...@redhat.com --- include/linux/migrate.h | 2 ++ mm/compaction.c

[PATCH v10 3/5] virtio_balloon: introduce migration primitives to balloon pages

2012-09-17 Thread Rafael Aquini
bookmarking elements (list and atomic counters) against the potential memory compaction concurrency; Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 305 +--- 1 file changed

[PATCH v10 5/5] mm: add vm event counters for balloon pages compaction

2012-09-17 Thread Rafael Aquini
This patch introduces a new set of vm event counters to keep track of ballooned pages compaction activity. Signed-off-by: Rafael Aquini aqu...@redhat.com --- drivers/virtio/virtio_balloon.c | 1 + include/linux/vm_event_item.h | 8 +++- mm/balloon_compaction.c | 2 ++ mm

Re: [PATCH v10 3/5] virtio_balloon: introduce migration primitives to balloon pages

2012-09-18 Thread Rafael Aquini
On Mon, Sep 17, 2012 at 03:15:52PM -0700, Andrew Morton wrote: + /* Number of balloon pages isolated from 'pages' list for compaction */ + unsigned int num_isolated_pages; Is it utterly inconceivable that this counter could exceed 4G, ever? /* Number of balloon pages we've told

Re: [PATCH v10 1/5] mm: introduce a common interface for balloon pages mobility

2012-09-18 Thread Rafael Aquini
On Mon, Sep 17, 2012 at 03:15:43PM -0700, Andrew Morton wrote: +/* return code to identify when a ballooned page has been migrated */ +#define BALLOON_MIGRATION_RETURN 0xba1100 I didn't really spend enough time to work out why this was done this way, but I know a hack when I see one!

  1   2   >