Bug#767295: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
On Sat, 2014-11-08 at 08:32 -0500, Gedalya wrote: On 11/08/2014 05:41 AM, Ian Campbell wrote: Please can you try running xl under valgrind, something similar to what I described earlier should work. I guess it didn't find much.. Indeed not :-/ I managed to get a test box up and running, so I backported a couple of additional upstream patches to help valgrind analyse Xen toolstacks and even with those I'm not seeing leaks. However with pmap I am seeing an additional +7f37f516a000 23852 14464 14464 rw--- [ anon ] with each reboot. The size seems to be the same each time. I can't quite figure out what this new mapping is. I'll keep digging. [...] Could you also post your guest cfg file please. Thanks, nothing unusual here but worth confirming. Ian. [0] b1dfc531d8ad and 171c6d7ac17e, FWIW -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#767295: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
On 11/09/2014 07:02 AM, Ian Campbell wrote: On Sat, 2014-11-08 at 08:32 -0500, Gedalya wrote: On 11/08/2014 05:41 AM, Ian Campbell wrote: Please can you try running xl under valgrind, something similar to what I described earlier should work. I guess it didn't find much.. Indeed not :-/ I managed to get a test box up and running, so I backported a couple of additional upstream patches to help valgrind analyse Xen toolstacks and even with those I'm not seeing leaks. However with pmap I am seeing an additional +7f37f516a000 23852 14464 14464 rw--- [ anon ] with each reboot. The size seems to be the same each time. I can't quite figure out what this new mapping is. I'll keep digging. [...] Yea. That's precisely what I was seeing. And even the first one shouldn't be there. There is no such block on xl processes for squeeze or wheezy VMs. That memory block is just a bit larger than the size of the initrd in the VM, could there be a connection? Could you also post your guest cfg file please. Thanks, nothing unusual here but worth confirming. Ian. [0] b1dfc531d8ad and 171c6d7ac17e, FWIW -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#767295: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
On 11/09/2014 07:18 AM, Gedalya wrote: That memory block is just a bit larger than the size of the initrd in the VM, could there be a connection? Nope. I changed the initrd to 2.6mb and that memory block is still exactly at 23852 / 14464 / 14464. Actually the process size is around 12 mb when pygrub is counting down to boot, then jumps up to 14+ mb -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#767295: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
On Sun, 2014-11-09 at 07:28 -0500, Gedalya wrote: On 11/09/2014 07:18 AM, Gedalya wrote: That memory block is just a bit larger than the size of the initrd in the VM, could there be a connection? Nope. I changed the initrd to 2.6mb and that memory block is still exactly at 23852 / 14464 / 14464. Actually the process size is around 12 mb when pygrub is counting down to boot, then jumps up to 14+ mb Running under gdb and dumping the contents of the leak block it appears to be the uncompressed payout of the vmlinuz i.e. using bzexplode[0]: $ file../bzexplode /boot/vmlinuz-3.16-3-amd64 | xzcat | od -t x4 | less 000 464c457f 00010102 020 003e0002 0001 0100 040 0040 00e1f160 vs: (gdb) x/32xw 0x7fffea8b5000 0x7fffea8b5000: 0x 0x 0x0174b002 0x 0x7fffea8b5010: 0x464c457f 0x00010102 0x 0x 0x7fffea8b5020: 0x003e0002 0x0001 0x0100 0x (the first 4 words are no doubt the allocator's metadata). I think I can see the leak now, it seems to be present in the latest upstream git tree too. Ian. [0] http://lists.xen.org/archives/html/xen-devel/2012-02/txtVnQboj3Whe.txt -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#767295: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
On Fri, 2014-11-07 at 21:32 -0500, Gedalya wrote: On 11/06/2014 09:12 AM, Ian Campbell wrote: I've posted a fix for this upstream: http://lists.xen.org/archives/html/xen-devel/2014-11/msg00542.html I've requested that it go into the next 4.4.x release. I also plan to backport it to the package regardless once it is accepted into the dev branch. Ian. So I've tried building xen 4.4.1-3 with this patch (attached), and I replaced the binary packages libxen-4.4 and xen-utils-4.4 with the new ones. The situation is the same: the xl process for jessie amd64 guests starts off at a resident size of 15mb as opposed to ~500kb for wheezy i386 guests. At guest reboot it jumps to 17mb and then, after a few seconds, to ~34mb. I also rebooted the host to help make sure the test is valid. Please can you try running xl under valgrind, something similar to what I described earlier should work. Could you also post your guest cfg file please. Thanks, Ian. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#767295: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
On 11/08/2014 05:41 AM, Ian Campbell wrote: Please can you try running xl under valgrind, something similar to what I described earlier should work. I guess it didn't find much.. # valgrind --leak-check=full xl cr -F /etc/xen/auto/asterisk_deb80.cfg ==6736== Memcheck, a memory error detector ==6736== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==6736== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info ==6736== Command: /usr/sbin/xl cr -F /etc/xen/auto/asterisk_deb80.cfg ==6736== ==6740== ==6740== HEAP SUMMARY: ==6740== in use at exit: 11,663 bytes in 40 blocks ==6740== total heap usage: 74 allocs, 34 frees, 44,570 bytes allocated ==6740== ==6740== LEAK SUMMARY: ==6740==definitely lost: 0 bytes in 0 blocks ==6740==indirectly lost: 0 bytes in 0 blocks ==6740== possibly lost: 0 bytes in 0 blocks ==6740==still reachable: 11,663 bytes in 40 blocks ==6740== suppressed: 0 bytes in 0 blocks ==6740== Reachable blocks (those to which a pointer was found) are not shown. ==6740== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==6740== ==6740== For counts of detected and suppressed errors, rerun with: -v ==6740== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) ==6739== ==6739== HEAP SUMMARY: ==6739== in use at exit: 10,711 bytes in 35 blocks ==6739== total heap usage: 63 allocs, 28 frees, 35,257 bytes allocated ==6739== ==6739== LEAK SUMMARY: ==6739==definitely lost: 0 bytes in 0 blocks ==6739==indirectly lost: 0 bytes in 0 blocks ==6739== possibly lost: 0 bytes in 0 blocks ==6739==still reachable: 10,711 bytes in 35 blocks ==6739== suppressed: 0 bytes in 0 blocks ==6739== Reachable blocks (those to which a pointer was found) are not shown. ==6739== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==6739== ==6739== For counts of detected and suppressed errors, rerun with: -v ==6739== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) ==6744== ==6744== HEAP SUMMARY: ==6744== in use at exit: 4,827 bytes in 42 blocks ==6744== total heap usage: 91 allocs, 49 frees, 38,343 bytes allocated ==6744== ==6744== LEAK SUMMARY: ==6744==definitely lost: 0 bytes in 0 blocks ==6744==indirectly lost: 0 bytes in 0 blocks ==6744== possibly lost: 0 bytes in 0 blocks ==6744==still reachable: 4,827 bytes in 42 blocks ==6744== suppressed: 0 bytes in 0 blocks ==6744== Reachable blocks (those to which a pointer was found) are not shown. ==6744== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==6744== ==6744== For counts of detected and suppressed errors, rerun with: -v ==6744== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) ==6745== ==6745== HEAP SUMMARY: ==6745== in use at exit: 13,034 bytes in 44 blocks ==6745== total heap usage: 97 allocs, 53 frees, 48,672 bytes allocated ==6745== ==6745== LEAK SUMMARY: ==6745==definitely lost: 0 bytes in 0 blocks ==6745==indirectly lost: 0 bytes in 0 blocks ==6745== possibly lost: 0 bytes in 0 blocks ==6745==still reachable: 13,034 bytes in 44 blocks ==6745== suppressed: 0 bytes in 0 blocks ==6745== Reachable blocks (those to which a pointer was found) are not shown. ==6745== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==6745== ==6745== For counts of detected and suppressed errors, rerun with: -v ==6745== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) ==6738== ==6738== HEAP SUMMARY: ==6738== in use at exit: 11,017 bytes in 41 blocks ==6738== total heap usage: 91 allocs, 50 frees, 48,555 bytes allocated ==6738== ==6738== LEAK SUMMARY: ==6738==definitely lost: 0 bytes in 0 blocks ==6738==indirectly lost: 0 bytes in 0 blocks ==6738== possibly lost: 0 bytes in 0 blocks ==6738==still reachable: 11,017 bytes in 41 blocks ==6738== suppressed: 0 bytes in 0 blocks ==6738== Reachable blocks (those to which a pointer was found) are not shown. ==6738== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==6738== ==6738== For counts of detected and suppressed errors, rerun with: -v ==6738== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) Parsing config from /etc/xen/auto/asterisk_deb80.cfg Waiting for domain asterisk_deb08 (domid 8) to die [pid 6736] Domain 8 has shut down, reason code 1 0x1 Action for shutdown reason code 1 is restart Domain 8 needs to be cleaned up: destroying the domain Done. Rebooting now Waiting for domain asterisk_deb08 (domid 9) to die [pid 6736] Domain 9 has shut down, reason code 0 0x0 Action for shutdown reason code 0 is destroy Domain 9 needs to be cleaned up: destroying the domain Done. Exiting now Could you also post your guest cfg file please. bootloader=pygrub memory = 1024 name = asterisk_deb08 vcpus = 2 pvh = 1 vif = ['mac=00:16:3e:00:00:12, bridge=breth1', 'mac=00:16:3e:00:00:13, bridge=breth0'] disk =
Bug#767295: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
On 11/06/2014 09:12 AM, Ian Campbell wrote: I've posted a fix for this upstream: http://lists.xen.org/archives/html/xen-devel/2014-11/msg00542.html I've requested that it go into the next 4.4.x release. I also plan to backport it to the package regardless once it is accepted into the dev branch. Ian. So I've tried building xen 4.4.1-3 with this patch (attached), and I replaced the binary packages libxen-4.4 and xen-utils-4.4 with the new ones. The situation is the same: the xl process for jessie amd64 guests starts off at a resident size of 15mb as opposed to ~500kb for wheezy i386 guests. At guest reboot it jumps to 17mb and then, after a few seconds, to ~34mb. I also rebooted the host to help make sure the test is valid. Thanks, Gedalya Index: xen-4.4.1/tools/libxl/libxl.c === --- xen-4.4.1.orig/tools/libxl/libxl.c +++ xen-4.4.1/tools/libxl/libxl.c @@ -2678,7 +2678,7 @@ void libxl__device_disk_local_initiate_a } if (dev != NULL) -dls-diskpath = strdup(dev); +dls-diskpath = libxl__strdup(gc, dev); dls-callback(egc, dls, 0); return;
Bug#767295: [Pkg-xen-devel] Bug#767295: Bug#767295: xl: apparent memory leak
On Thu, 2014-11-06 at 12:14 +, Ian Campbell wrote: On Wed, 2014-10-29 at 17:36 -0400, Gedalya wrote: When booting domU's running amd64 jessie, I notice some memory problems with xl. I ran it (actually, upstream staging-4.4) under valgrind and it reported after a reboot: ==5722== 24 bytes in 1 blocks are definitely lost in loss record 35 of 72 ==5722==at 0x4026B2D: malloc (vg_replace_malloc.c:299) ==5722==by 0x41739FF: strdup (strdup.c:43) ==5722==by 0x405F38E: libxl__device_disk_local_initiate_attach (libxl.c:2681) ==5722==by 0x408D2A6: libxl__bootloader_run (libxl_bootloader.c:385) ==5722==by 0x406B50C: initiate_domain_create (libxl_create.c:807) ==5722==by 0x406B50C: do_domain_create (libxl_create.c:1354) ==5722==by 0x406B6AF: libxl_domain_create_new (libxl_create.c:1377) ==5722==by 0x8056182: create_domain (xl_cmdimpl.c:2254) ==5722==by 0x8059F27: main_create (xl_cmdimpl.c:4407) ==5722==by 0x804E119: main (xl.c:360) ==5722== ==5722== 28 bytes in 1 blocks are definitely lost in loss record 39 of 72 ==5722==at 0x4026B2D: malloc (vg_replace_malloc.c:299) ==5722==by 0x4279EBB: read_message (xs.c:1143) ==5722==by 0x427A89B: read_thread (xs.c:1219) ==5722==by 0x40E7953: start_thread (pthread_create.c:304) ==5722==by 0x41CF95D: clone (clone.S:130) I've posted a fix for this upstream: http://lists.xen.org/archives/html/xen-devel/2014-11/msg00542.html I've requested that it go into the next 4.4.x release. I also plan to backport it to the package regardless once it is accepted into the dev branch. Ian. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
