Re: FreeBSD and memory balloon drivers
On Fri, Jun 20, 2014 at 06:56:20PM +0200, Roger Pau Monn? wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 20/06/14 17:26, Konstantin Belousov wrote:
> > On Fri, Jun 20, 2014 at 05:15:43PM +0200, Roger Pau Monn? wrote:
> >> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1
> >>
> >> On 20/06/14 15:28, Konstantin Belousov wrote:
> >>> On Fri, Jun 20, 2014 at 11:35:53AM +0200, Roger Pau Monn?
> >>> wrote:
> Hello,
>
> I've been looking into the Xen balloon driver, because I've
> experienced problems when ballooning memory down which AFAICT
> are also present in the VirtIO balloon driver. The problem
> I've experienced is that when ballooning memory down, we
> basically allocate a bunch of memory as WIRED, to make sure
> nobody tries to swap it do disk, since it will crash the
> kernel because the memory is not populated. Due to this
> massive amount of memory allocated as WIRED, user-space
> programs that try to use mlock will fail because we hit the
> limit in vm.max_wired.
>
> I'm not sure what's the best way to deal with this
> limitation, should vm.max_wired be changed from the balloon
> drivers when ballooning down/up? Is there anyway to remove
> the pages ballooned down from the memory accounting of wired
> pages?
> >>>
> >>> You could change the type of pages the ballon driver is
> >>> allocating. Instead of wired pages, you may request unmanaged,
> >>> by passing NULL object to vm_page_alloc(). This would also
> >>> save on the trie nodes for managing the radix trie for the
> >>> object. There are still plinks or listq to keep track of the
> >>> allocated pages.
> >>
> >> Thanks for the info, I have the following patch which fixes the
> >> usage of WIRED for both the Xen and the VirtIO balloon drivers,
> >> could someone please test the VirtIO side?
> > I briefly looked at the xen balloon. You do not need
> > balloon_append(). Use struct vm_page plinks field to link the
> > pages.
>
> Sure, thanks for the review, here is an updated version.
It looks fine to me.
>
> Roger.
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.12 (Darwin)
>
> iQEcBAEBAgAGBQJTpGe0AAoJEKXZdqUyumTA5acH/jQ+Iwjw+QTwUtsh9ZLs7Gm0
> h7XaCwa/gASEjzcxK2smIBuKA10LUSoulcYkC3cUWXqaPwFega14JcbySBRI06Z1
> 1bU50DL8TPLuQIzrVWZgtA+QsZkwEu/jhijr0AMxTcnm6Wq1LV5QRDoqhD0kRiHu
> kEW1ez832y/0j+oeQAq0aR67zVw7hoIouH9eLaWXS4Vgxz5YQ2Pal1BMAY/OCgua
> xyF7BHia9KsGTKZ9pPUQfQAW5eJrwAxR0AitQjmOwRKtWyRqymZxhYHjLYOgOKQJ
> w93aLRVeG2oo0NNC6a9JD/c64sGHLABg37mnSTRL/v9gTj9I1DcFoid2q0iDGbM=
> =8nDn
> -END PGP SIGNATURE-
> >From e5c898b1b2b734fdc4aa2acf645efd481f09b71d Mon Sep 17 00:00:00 2001
> From: Roger Pau Monne
> Date: Fri, 20 Jun 2014 16:34:31 +0200
> Subject: [PATCH] xen/virtio: fix balloon drivers to not mark pages as WIRED
>
> Prevent the Xen and VirtIO balloon drivers from marking pages as
> wired. This prevents them from increasing the system wired page count,
> which can lead to mlock failing because of hitting the limit in
> vm.max_wired.
>
> In the Xen case make sure pages are zeroed before giving them back to
> the hypervisor, or else we might be leaking data. Also remove the
> balloon_{append/retrieve} and link pages directly into the
> ballooned_pages queue using the plinks.q field in the page struct.
>
> Sponsored by: Citrix Systems R&D
> Reviewed by: xxx
> Approved by: xxx
>
> dev/virtio/balloon/virtio_balloon.c:
> - Don't allocate pages with VM_ALLOC_WIRED.
>
> dev/xen/balloon/balloon.c:
> - Don't allocate pages with VM_ALLOC_WIRED.
> - Make sure pages are zeroed before giving them back to the
>hypervisor.
> - Remove the balloon_entry struct and the balloon_{append/retrieve}
>functions and use the page plinks.q entry to link the pages
>directly into the ballooned_pages queue.
> ---
> sys/dev/virtio/balloon/virtio_balloon.c |4 +-
> sys/dev/xen/balloon/balloon.c | 87
> ---
> 2 files changed, 23 insertions(+), 68 deletions(-)
>
> diff --git a/sys/dev/virtio/balloon/virtio_balloon.c
> b/sys/dev/virtio/balloon/virtio_balloon.c
> index d540099..6d00ef3 100644
> --- a/sys/dev/virtio/balloon/virtio_balloon.c
> +++ b/sys/dev/virtio/balloon/virtio_balloon.c
> @@ -438,8 +438,7 @@ vtballoon_alloc_page(struct vtballoon_softc *sc)
> {
> vm_page_t m;
>
> - m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_WIRED |
> - VM_ALLOC_NOOBJ);
> + m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ);
> if (m != NULL)
> sc->vtballoon_current_npages++;
>
> @@ -450,7 +449,6 @@ static void
> vtballoon_free_page(struct vtballoon_softc *sc, vm_page_t m)
> {
>
> - vm_page_unwire(m, PQ_INACTIVE);
> vm_page_free(m);
> sc->vtballoon_current_npages--;
> }
> diff --git a/sys/dev/xen/balloon/balloon.c b/sys/dev/xen/balloon/balloon.c
> index fa56c86..0d2bba2 100644
> --- a/sys/dev/xen/balloon/balloon.c
Re: FreeBSD and memory balloon drivers
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 20/06/14 17:26, Konstantin Belousov wrote:
> On Fri, Jun 20, 2014 at 05:15:43PM +0200, Roger Pau Monn? wrote:
>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1
>>
>> On 20/06/14 15:28, Konstantin Belousov wrote:
>>> On Fri, Jun 20, 2014 at 11:35:53AM +0200, Roger Pau Monn?
>>> wrote:
Hello,
I've been looking into the Xen balloon driver, because I've
experienced problems when ballooning memory down which AFAICT
are also present in the VirtIO balloon driver. The problem
I've experienced is that when ballooning memory down, we
basically allocate a bunch of memory as WIRED, to make sure
nobody tries to swap it do disk, since it will crash the
kernel because the memory is not populated. Due to this
massive amount of memory allocated as WIRED, user-space
programs that try to use mlock will fail because we hit the
limit in vm.max_wired.
I'm not sure what's the best way to deal with this
limitation, should vm.max_wired be changed from the balloon
drivers when ballooning down/up? Is there anyway to remove
the pages ballooned down from the memory accounting of wired
pages?
>>>
>>> You could change the type of pages the ballon driver is
>>> allocating. Instead of wired pages, you may request unmanaged,
>>> by passing NULL object to vm_page_alloc(). This would also
>>> save on the trie nodes for managing the radix trie for the
>>> object. There are still plinks or listq to keep track of the
>>> allocated pages.
>>
>> Thanks for the info, I have the following patch which fixes the
>> usage of WIRED for both the Xen and the VirtIO balloon drivers,
>> could someone please test the VirtIO side?
> I briefly looked at the xen balloon. You do not need
> balloon_append(). Use struct vm_page plinks field to link the
> pages.
Sure, thanks for the review, here is an updated version.
Roger.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (Darwin)
iQEcBAEBAgAGBQJTpGe0AAoJEKXZdqUyumTA5acH/jQ+Iwjw+QTwUtsh9ZLs7Gm0
h7XaCwa/gASEjzcxK2smIBuKA10LUSoulcYkC3cUWXqaPwFega14JcbySBRI06Z1
1bU50DL8TPLuQIzrVWZgtA+QsZkwEu/jhijr0AMxTcnm6Wq1LV5QRDoqhD0kRiHu
kEW1ez832y/0j+oeQAq0aR67zVw7hoIouH9eLaWXS4Vgxz5YQ2Pal1BMAY/OCgua
xyF7BHia9KsGTKZ9pPUQfQAW5eJrwAxR0AitQjmOwRKtWyRqymZxhYHjLYOgOKQJ
w93aLRVeG2oo0NNC6a9JD/c64sGHLABg37mnSTRL/v9gTj9I1DcFoid2q0iDGbM=
=8nDn
-END PGP SIGNATURE-
>From e5c898b1b2b734fdc4aa2acf645efd481f09b71d Mon Sep 17 00:00:00 2001
From: Roger Pau Monne
Date: Fri, 20 Jun 2014 16:34:31 +0200
Subject: [PATCH] xen/virtio: fix balloon drivers to not mark pages as WIRED
Prevent the Xen and VirtIO balloon drivers from marking pages as
wired. This prevents them from increasing the system wired page count,
which can lead to mlock failing because of hitting the limit in
vm.max_wired.
In the Xen case make sure pages are zeroed before giving them back to
the hypervisor, or else we might be leaking data. Also remove the
balloon_{append/retrieve} and link pages directly into the
ballooned_pages queue using the plinks.q field in the page struct.
Sponsored by: Citrix Systems R&D
Reviewed by: xxx
Approved by: xxx
dev/virtio/balloon/virtio_balloon.c:
- Don't allocate pages with VM_ALLOC_WIRED.
dev/xen/balloon/balloon.c:
- Don't allocate pages with VM_ALLOC_WIRED.
- Make sure pages are zeroed before giving them back to the
hypervisor.
- Remove the balloon_entry struct and the balloon_{append/retrieve}
functions and use the page plinks.q entry to link the pages
directly into the ballooned_pages queue.
---
sys/dev/virtio/balloon/virtio_balloon.c |4 +-
sys/dev/xen/balloon/balloon.c | 87 ---
2 files changed, 23 insertions(+), 68 deletions(-)
diff --git a/sys/dev/virtio/balloon/virtio_balloon.c
b/sys/dev/virtio/balloon/virtio_balloon.c
index d540099..6d00ef3 100644
--- a/sys/dev/virtio/balloon/virtio_balloon.c
+++ b/sys/dev/virtio/balloon/virtio_balloon.c
@@ -438,8 +438,7 @@ vtballoon_alloc_page(struct vtballoon_softc *sc)
{
vm_page_t m;
- m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_WIRED |
- VM_ALLOC_NOOBJ);
+ m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ);
if (m != NULL)
sc->vtballoon_current_npages++;
@@ -450,7 +449,6 @@ static void
vtballoon_free_page(struct vtballoon_softc *sc, vm_page_t m)
{
- vm_page_unwire(m, PQ_INACTIVE);
vm_page_free(m);
sc->vtballoon_current_npages--;
}
diff --git a/sys/dev/xen/balloon/balloon.c b/sys/dev/xen/balloon/balloon.c
index fa56c86..0d2bba2 100644
--- a/sys/dev/xen/balloon/balloon.c
+++ b/sys/dev/xen/balloon/balloon.c
@@ -94,13 +94,8 @@ SYSCTL_ULONG(_dev_xen_balloon, OID_AUTO, low_mem, CTLFLAG_RD,
SYSCTL_ULONG(_dev_xen_balloon, OID_AUTO, high_mem, CTLFLAG_RD,
&bs.balloon_high, 0, "High-mem balloon");
-struct balloon_entry {
- vm_page_t page;
- STAILQ_ENTRY(balloon_e
Re: FreeBSD and memory balloon drivers
On Fri, Jun 20, 2014 at 05:15:43PM +0200, Roger Pau Monn? wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 20/06/14 15:28, Konstantin Belousov wrote: > > On Fri, Jun 20, 2014 at 11:35:53AM +0200, Roger Pau Monn? wrote: > >> Hello, > >> > >> I've been looking into the Xen balloon driver, because I've > >> experienced problems when ballooning memory down which AFAICT are > >> also present in the VirtIO balloon driver. The problem I've > >> experienced is that when ballooning memory down, we basically > >> allocate a bunch of memory as WIRED, to make sure nobody tries to > >> swap it do disk, since it will crash the kernel because the > >> memory is not populated. Due to this massive amount of memory > >> allocated as WIRED, user-space programs that try to use mlock > >> will fail because we hit the limit in vm.max_wired. > >> > >> I'm not sure what's the best way to deal with this limitation, > >> should vm.max_wired be changed from the balloon drivers when > >> ballooning down/up? Is there anyway to remove the pages ballooned > >> down from the memory accounting of wired pages? > > > > You could change the type of pages the ballon driver is > > allocating. Instead of wired pages, you may request unmanaged, by > > passing NULL object to vm_page_alloc(). This would also save on > > the trie nodes for managing the radix trie for the object. There > > are still plinks or listq to keep track of the allocated pages. > > Thanks for the info, I have the following patch which fixes the usage > of WIRED for both the Xen and the VirtIO balloon drivers, could > someone please test the VirtIO side? I briefly looked at the xen balloon. You do not need balloon_append(). Use struct vm_page plinks field to link the pages. pgpeCYPXBGamx.pgp Description: PGP signature
Re: FreeBSD and memory balloon drivers
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 20/06/14 15:28, Konstantin Belousov wrote:
> On Fri, Jun 20, 2014 at 11:35:53AM +0200, Roger Pau Monn? wrote:
>> Hello,
>>
>> I've been looking into the Xen balloon driver, because I've
>> experienced problems when ballooning memory down which AFAICT are
>> also present in the VirtIO balloon driver. The problem I've
>> experienced is that when ballooning memory down, we basically
>> allocate a bunch of memory as WIRED, to make sure nobody tries to
>> swap it do disk, since it will crash the kernel because the
>> memory is not populated. Due to this massive amount of memory
>> allocated as WIRED, user-space programs that try to use mlock
>> will fail because we hit the limit in vm.max_wired.
>>
>> I'm not sure what's the best way to deal with this limitation,
>> should vm.max_wired be changed from the balloon drivers when
>> ballooning down/up? Is there anyway to remove the pages ballooned
>> down from the memory accounting of wired pages?
>
> You could change the type of pages the ballon driver is
> allocating. Instead of wired pages, you may request unmanaged, by
> passing NULL object to vm_page_alloc(). This would also save on
> the trie nodes for managing the radix trie for the object. There
> are still plinks or listq to keep track of the allocated pages.
Thanks for the info, I have the following patch which fixes the usage
of WIRED for both the Xen and the VirtIO balloon drivers, could
someone please test the VirtIO side?
Roger.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (Darwin)
iQEcBAEBAgAGBQJTpFAfAAoJEKXZdqUyumTA1TQH/22YpAGCQ8oa0hmpfE8oovxz
q8EDRyfDoogEswNYwboI8cBP7GSbuBbe1Z0MTiMHtwyHzqGhJM5B7jKioqFsqxvc
/Qfld8z3vDD94/5iaMX64dV2/VKkLwypR2uU5PkN018FTAJ0FFycC336xVjD8eUz
/DCRQIZRUzNlcrZYlOtSALR2M9bM1/f2++e2C6L7kbSsF4BAH2wmRcdtM1uBMO6/
CnD7ctZsnxxdS05eLWMpv6jfcRH8yDM3vPaHgXa223q74TU1Rh7AFw/TPsyBvBY2
cDUnYLZdUx0NQJfPM9MKGLe8P5o3WLVO1hfGmrZHdmjE2B18mNfzg4SS0CUQhQ0=
=hctw
-END PGP SIGNATURE-
>From 2ed0f82b16753c96e96acc1e26a75a0fd2ee7d34 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne
Date: Fri, 20 Jun 2014 16:34:31 +0200
Subject: [PATCH] xen/virtio: fix balloon drivers to not mark pages as WIRED
Prevent the Xen and VirtIO balloon drivers from marking pages as
wired. This prevents them from increasing the system wired page count,
which can lead to mlock failing because of hitting the limit in
vm.max_wired.
Also, in the Xen case make sure pages are zeroed before giving them
back to the hypervisor, or else we might be leaking data.
Sponsored by: Citrix Systems R&D
Reviewed by: xxx
Approved by: xxx
dev/virtio/balloon/virtio_balloon.c:
- Don't allocate pages with VM_ALLOC_WIRED.
dev/xen/balloon/balloon.c:
- Don't allocate pages with VM_ALLOC_WIRED.
- Make sure pages are zeroed before giving them back to the
hypervisor.
---
sys/dev/virtio/balloon/virtio_balloon.c |4 +---
sys/dev/xen/balloon/balloon.c | 13 ++---
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/sys/dev/virtio/balloon/virtio_balloon.c
b/sys/dev/virtio/balloon/virtio_balloon.c
index d540099..6d00ef3 100644
--- a/sys/dev/virtio/balloon/virtio_balloon.c
+++ b/sys/dev/virtio/balloon/virtio_balloon.c
@@ -438,8 +438,7 @@ vtballoon_alloc_page(struct vtballoon_softc *sc)
{
vm_page_t m;
- m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_WIRED |
- VM_ALLOC_NOOBJ);
+ m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ);
if (m != NULL)
sc->vtballoon_current_npages++;
@@ -450,7 +449,6 @@ static void
vtballoon_free_page(struct vtballoon_softc *sc, vm_page_t m)
{
- vm_page_unwire(m, PQ_INACTIVE);
vm_page_free(m);
sc->vtballoon_current_npages--;
}
diff --git a/sys/dev/xen/balloon/balloon.c b/sys/dev/xen/balloon/balloon.c
index fa56c86..a7ca1e4 100644
--- a/sys/dev/xen/balloon/balloon.c
+++ b/sys/dev/xen/balloon/balloon.c
@@ -255,7 +255,6 @@ increase_reservation(unsigned long nr_pages)
set_phys_to_machine(pfn, frame_list[i]);
- vm_page_unwire(page, PQ_INACTIVE);
vm_page_free(page);
}
@@ -286,18 +285,26 @@ decrease_reservation(unsigned long nr_pages)
for (i = 0; i < nr_pages; i++) {
if ((page = vm_page_alloc(NULL, 0,
VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ |
- VM_ALLOC_WIRED | VM_ALLOC_ZERO)) == NULL) {
+ VM_ALLOC_ZERO)) == NULL) {
nr_pages = i;
need_sleep = 1;
break;
}
+ if ((page->flags & PG_ZERO) == 0) {
+ /*
+* Zero the page, or else we might be leaking
+* important data to other domains on the same
+* host.
+*/
+ pmap_zero_page(page);
+
Re: FreeBSD and memory balloon drivers
On Fri, Jun 20, 2014 at 11:35:53AM +0200, Roger Pau Monn? wrote: > Hello, > > I've been looking into the Xen balloon driver, because I've experienced > problems when ballooning memory down which AFAICT are also present in > the VirtIO balloon driver. The problem I've experienced is that when > ballooning memory down, we basically allocate a bunch of memory as > WIRED, to make sure nobody tries to swap it do disk, since it will crash > the kernel because the memory is not populated. Due to this massive > amount of memory allocated as WIRED, user-space programs that try to use > mlock will fail because we hit the limit in vm.max_wired. > > I'm not sure what's the best way to deal with this limitation, should > vm.max_wired be changed from the balloon drivers when ballooning > down/up? Is there anyway to remove the pages ballooned down from the > memory accounting of wired pages? You could change the type of pages the ballon driver is allocating. Instead of wired pages, you may request unmanaged, by passing NULL object to vm_page_alloc(). This would also save on the trie nodes for managing the radix trie for the object. There are still plinks or listq to keep track of the allocated pages. pgpZmw4Rq2O5i.pgp Description: PGP signature
FreeBSD and memory balloon drivers
Hello, I've been looking into the Xen balloon driver, because I've experienced problems when ballooning memory down which AFAICT are also present in the VirtIO balloon driver. The problem I've experienced is that when ballooning memory down, we basically allocate a bunch of memory as WIRED, to make sure nobody tries to swap it do disk, since it will crash the kernel because the memory is not populated. Due to this massive amount of memory allocated as WIRED, user-space programs that try to use mlock will fail because we hit the limit in vm.max_wired. I'm not sure what's the best way to deal with this limitation, should vm.max_wired be changed from the balloon drivers when ballooning down/up? Is there anyway to remove the pages ballooned down from the memory accounting of wired pages? Thanks, Roger. ___ freebsd-xen@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-xen To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"
