FreeBSD and memory balloon drivers
Hello, I've been looking into the Xen balloon driver, because I've experienced problems when ballooning memory down which AFAICT are also present in the VirtIO balloon driver. The problem I've experienced is that when ballooning memory down, we basically allocate a bunch of memory as WIRED, to make sure nobody tries to swap it do disk, since it will crash the kernel because the memory is not populated. Due to this massive amount of memory allocated as WIRED, user-space programs that try to use mlock will fail because we hit the limit in vm.max_wired. I'm not sure what's the best way to deal with this limitation, should vm.max_wired be changed from the balloon drivers when ballooning down/up? Is there anyway to remove the pages ballooned down from the memory accounting of wired pages? Thanks, Roger. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: Request for Testers: libvirt and bhyve
Hi Craig, I've given it a try and its looking good. I already had some bhyve vms i wanted to import rather than creating new. This worked fine. I didnt ever get the default network which isnt an issue as its the first thing I undefine on any KVM server I've use libvirt on. Also I created the bridges myself as I was unsure how libvirt was supposed to do this and I've always created them manually for linux libvirt/kvm instances, just letting libvirt create the vnet interfaces. I'm using zvols rather than file backed but thats not causing any issues. I'd suggest the port have a pkg-msg to suggest you add uri_default = bhyve:///system to /usr/local/etc/libvirt/libvirt.conf so you can simply use virsh list etc instead of needing to specify the uri each time. Thanks for all the work on this. Vince On 18/06/2014 19:29, Craig Rodrigues wrote: Hi, I've had some success with getting libvirt and bhyve to work. I wrote up a list of the steps that I followed here: http://people.freebsd.org/~rodrigc/libvirt-bhyve/libvirt-bhyve.html If anyone is interested, please try it out, and provide feedback to this list. It would be great to get more testers. Roman, when do you think you will be able to commit all your latest fixes into the devel/libvirt port Makefile? It would greatly increase the usability of the port. Thanks for doing all the work to add bhyve support to libvirt!! -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
bhyve crash
Hi, Just had a bhyve crash on 11-CURRENT. The host is r267362. The guest is r266947. The crash was: pid 38256 (bhyve), uid 0: exited on signal 6 (core dumped) (The formatting of this may be off a bit) vmexit[12] reason VMX rip 0x80d9e11d inst_length 3 status 0 exit_reason 2 qualification 0x inst_type 0 inst_error 0 I can provide the core dump file. Ideas? Thanks, Steve pgprxMlzkxLn_.pgp Description: PGP signature
Re: FreeBSD and memory balloon drivers
On Fri, Jun 20, 2014 at 11:35:53AM +0200, Roger Pau Monn? wrote: Hello, I've been looking into the Xen balloon driver, because I've experienced problems when ballooning memory down which AFAICT are also present in the VirtIO balloon driver. The problem I've experienced is that when ballooning memory down, we basically allocate a bunch of memory as WIRED, to make sure nobody tries to swap it do disk, since it will crash the kernel because the memory is not populated. Due to this massive amount of memory allocated as WIRED, user-space programs that try to use mlock will fail because we hit the limit in vm.max_wired. I'm not sure what's the best way to deal with this limitation, should vm.max_wired be changed from the balloon drivers when ballooning down/up? Is there anyway to remove the pages ballooned down from the memory accounting of wired pages? You could change the type of pages the ballon driver is allocating. Instead of wired pages, you may request unmanaged, by passing NULL object to vm_page_alloc(). This would also save on the trie nodes for managing the radix trie for the object. There are still plinks or listq to keep track of the allocated pages. pgpXqKHJvGntG.pgp Description: PGP signature
Re: FreeBSD and memory balloon drivers
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 20/06/14 15:28, Konstantin Belousov wrote: On Fri, Jun 20, 2014 at 11:35:53AM +0200, Roger Pau Monn? wrote: Hello, I've been looking into the Xen balloon driver, because I've experienced problems when ballooning memory down which AFAICT are also present in the VirtIO balloon driver. The problem I've experienced is that when ballooning memory down, we basically allocate a bunch of memory as WIRED, to make sure nobody tries to swap it do disk, since it will crash the kernel because the memory is not populated. Due to this massive amount of memory allocated as WIRED, user-space programs that try to use mlock will fail because we hit the limit in vm.max_wired. I'm not sure what's the best way to deal with this limitation, should vm.max_wired be changed from the balloon drivers when ballooning down/up? Is there anyway to remove the pages ballooned down from the memory accounting of wired pages? You could change the type of pages the ballon driver is allocating. Instead of wired pages, you may request unmanaged, by passing NULL object to vm_page_alloc(). This would also save on the trie nodes for managing the radix trie for the object. There are still plinks or listq to keep track of the allocated pages. Thanks for the info, I have the following patch which fixes the usage of WIRED for both the Xen and the VirtIO balloon drivers, could someone please test the VirtIO side? Roger. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (Darwin) iQEcBAEBAgAGBQJTpFAfAAoJEKXZdqUyumTA1TQH/22YpAGCQ8oa0hmpfE8oovxz q8EDRyfDoogEswNYwboI8cBP7GSbuBbe1Z0MTiMHtwyHzqGhJM5B7jKioqFsqxvc /Qfld8z3vDD94/5iaMX64dV2/VKkLwypR2uU5PkN018FTAJ0FFycC336xVjD8eUz /DCRQIZRUzNlcrZYlOtSALR2M9bM1/f2++e2C6L7kbSsF4BAH2wmRcdtM1uBMO6/ CnD7ctZsnxxdS05eLWMpv6jfcRH8yDM3vPaHgXa223q74TU1Rh7AFw/TPsyBvBY2 cDUnYLZdUx0NQJfPM9MKGLe8P5o3WLVO1hfGmrZHdmjE2B18mNfzg4SS0CUQhQ0= =hctw -END PGP SIGNATURE- From 2ed0f82b16753c96e96acc1e26a75a0fd2ee7d34 Mon Sep 17 00:00:00 2001 From: Roger Pau Monne roger@citrix.com Date: Fri, 20 Jun 2014 16:34:31 +0200 Subject: [PATCH] xen/virtio: fix balloon drivers to not mark pages as WIRED Prevent the Xen and VirtIO balloon drivers from marking pages as wired. This prevents them from increasing the system wired page count, which can lead to mlock failing because of hitting the limit in vm.max_wired. Also, in the Xen case make sure pages are zeroed before giving them back to the hypervisor, or else we might be leaking data. Sponsored by: Citrix Systems RD Reviewed by: xxx Approved by: xxx dev/virtio/balloon/virtio_balloon.c: - Don't allocate pages with VM_ALLOC_WIRED. dev/xen/balloon/balloon.c: - Don't allocate pages with VM_ALLOC_WIRED. - Make sure pages are zeroed before giving them back to the hypervisor. --- sys/dev/virtio/balloon/virtio_balloon.c |4 +--- sys/dev/xen/balloon/balloon.c | 13 ++--- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/sys/dev/virtio/balloon/virtio_balloon.c b/sys/dev/virtio/balloon/virtio_balloon.c index d540099..6d00ef3 100644 --- a/sys/dev/virtio/balloon/virtio_balloon.c +++ b/sys/dev/virtio/balloon/virtio_balloon.c @@ -438,8 +438,7 @@ vtballoon_alloc_page(struct vtballoon_softc *sc) { vm_page_t m; - m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_WIRED | - VM_ALLOC_NOOBJ); + m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ); if (m != NULL) sc-vtballoon_current_npages++; @@ -450,7 +449,6 @@ static void vtballoon_free_page(struct vtballoon_softc *sc, vm_page_t m) { - vm_page_unwire(m, PQ_INACTIVE); vm_page_free(m); sc-vtballoon_current_npages--; } diff --git a/sys/dev/xen/balloon/balloon.c b/sys/dev/xen/balloon/balloon.c index fa56c86..a7ca1e4 100644 --- a/sys/dev/xen/balloon/balloon.c +++ b/sys/dev/xen/balloon/balloon.c @@ -255,7 +255,6 @@ increase_reservation(unsigned long nr_pages) set_phys_to_machine(pfn, frame_list[i]); - vm_page_unwire(page, PQ_INACTIVE); vm_page_free(page); } @@ -286,18 +285,26 @@ decrease_reservation(unsigned long nr_pages) for (i = 0; i nr_pages; i++) { if ((page = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ | - VM_ALLOC_WIRED | VM_ALLOC_ZERO)) == NULL) { + VM_ALLOC_ZERO)) == NULL) { nr_pages = i; need_sleep = 1; break; } + if ((page-flags PG_ZERO) == 0) { + /* +* Zero the page, or else we might be leaking +* important data to other domains on the same +* host. +*/ + pmap_zero_page(page); + } +
Re: Request for Testers: libvirt and bhyve
On Fri, Jun 20, 2014 at 5:56 AM, Vincent Hoffman vi...@unsane.co.uk wrote: Hi Craig, I've given it a try and its looking good. Nice! Do you have time to try virt-manager: http://people.freebsd.org/~rodrigc/libvirt/virt-manager.html Thanks for all the work on this. Roman did all the hard work to get bhyve support into libvirt, and upstream his changes to the libvirt project. All the thanks go to him! -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: FreeBSD and memory balloon drivers
On Fri, Jun 20, 2014 at 06:56:20PM +0200, Roger Pau Monn? wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 20/06/14 17:26, Konstantin Belousov wrote: On Fri, Jun 20, 2014 at 05:15:43PM +0200, Roger Pau Monn? wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 20/06/14 15:28, Konstantin Belousov wrote: On Fri, Jun 20, 2014 at 11:35:53AM +0200, Roger Pau Monn? wrote: Hello, I've been looking into the Xen balloon driver, because I've experienced problems when ballooning memory down which AFAICT are also present in the VirtIO balloon driver. The problem I've experienced is that when ballooning memory down, we basically allocate a bunch of memory as WIRED, to make sure nobody tries to swap it do disk, since it will crash the kernel because the memory is not populated. Due to this massive amount of memory allocated as WIRED, user-space programs that try to use mlock will fail because we hit the limit in vm.max_wired. I'm not sure what's the best way to deal with this limitation, should vm.max_wired be changed from the balloon drivers when ballooning down/up? Is there anyway to remove the pages ballooned down from the memory accounting of wired pages? You could change the type of pages the ballon driver is allocating. Instead of wired pages, you may request unmanaged, by passing NULL object to vm_page_alloc(). This would also save on the trie nodes for managing the radix trie for the object. There are still plinks or listq to keep track of the allocated pages. Thanks for the info, I have the following patch which fixes the usage of WIRED for both the Xen and the VirtIO balloon drivers, could someone please test the VirtIO side? I briefly looked at the xen balloon. You do not need balloon_append(). Use struct vm_page plinks field to link the pages. Sure, thanks for the review, here is an updated version. It looks fine to me. Roger. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (Darwin) iQEcBAEBAgAGBQJTpGe0AAoJEKXZdqUyumTA5acH/jQ+Iwjw+QTwUtsh9ZLs7Gm0 h7XaCwa/gASEjzcxK2smIBuKA10LUSoulcYkC3cUWXqaPwFega14JcbySBRI06Z1 1bU50DL8TPLuQIzrVWZgtA+QsZkwEu/jhijr0AMxTcnm6Wq1LV5QRDoqhD0kRiHu kEW1ez832y/0j+oeQAq0aR67zVw7hoIouH9eLaWXS4Vgxz5YQ2Pal1BMAY/OCgua xyF7BHia9KsGTKZ9pPUQfQAW5eJrwAxR0AitQjmOwRKtWyRqymZxhYHjLYOgOKQJ w93aLRVeG2oo0NNC6a9JD/c64sGHLABg37mnSTRL/v9gTj9I1DcFoid2q0iDGbM= =8nDn -END PGP SIGNATURE- From e5c898b1b2b734fdc4aa2acf645efd481f09b71d Mon Sep 17 00:00:00 2001 From: Roger Pau Monne roger@citrix.com Date: Fri, 20 Jun 2014 16:34:31 +0200 Subject: [PATCH] xen/virtio: fix balloon drivers to not mark pages as WIRED Prevent the Xen and VirtIO balloon drivers from marking pages as wired. This prevents them from increasing the system wired page count, which can lead to mlock failing because of hitting the limit in vm.max_wired. In the Xen case make sure pages are zeroed before giving them back to the hypervisor, or else we might be leaking data. Also remove the balloon_{append/retrieve} and link pages directly into the ballooned_pages queue using the plinks.q field in the page struct. Sponsored by: Citrix Systems RD Reviewed by: xxx Approved by: xxx dev/virtio/balloon/virtio_balloon.c: - Don't allocate pages with VM_ALLOC_WIRED. dev/xen/balloon/balloon.c: - Don't allocate pages with VM_ALLOC_WIRED. - Make sure pages are zeroed before giving them back to the hypervisor. - Remove the balloon_entry struct and the balloon_{append/retrieve} functions and use the page plinks.q entry to link the pages directly into the ballooned_pages queue. --- sys/dev/virtio/balloon/virtio_balloon.c |4 +- sys/dev/xen/balloon/balloon.c | 87 --- 2 files changed, 23 insertions(+), 68 deletions(-) diff --git a/sys/dev/virtio/balloon/virtio_balloon.c b/sys/dev/virtio/balloon/virtio_balloon.c index d540099..6d00ef3 100644 --- a/sys/dev/virtio/balloon/virtio_balloon.c +++ b/sys/dev/virtio/balloon/virtio_balloon.c @@ -438,8 +438,7 @@ vtballoon_alloc_page(struct vtballoon_softc *sc) { vm_page_t m; - m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_WIRED | - VM_ALLOC_NOOBJ); + m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ); if (m != NULL) sc-vtballoon_current_npages++; @@ -450,7 +449,6 @@ static void vtballoon_free_page(struct vtballoon_softc *sc, vm_page_t m) { - vm_page_unwire(m, PQ_INACTIVE); vm_page_free(m); sc-vtballoon_current_npages--; } diff --git a/sys/dev/xen/balloon/balloon.c b/sys/dev/xen/balloon/balloon.c index fa56c86..0d2bba2 100644 --- a/sys/dev/xen/balloon/balloon.c +++ b/sys/dev/xen/balloon/balloon.c @@ -94,13 +94,8 @@ SYSCTL_ULONG(_dev_xen_balloon, OID_AUTO, low_mem, CTLFLAG_RD, SYSCTL_ULONG(_dev_xen_balloon, OID_AUTO, high_mem, CTLFLAG_RD, bs.balloon_high, 0,