Re: VIMAGE: Freed UMA keg was not empty
Cool! Glad to have helped! -a On 29 November 2013 00:47, Craig Rodrigues rodr...@freebsd.org wrote: On Wed, Nov 27, 2013 at 1:11 PM, Adrian Chadd adr...@freebsd.org wrote: Modify that function to print out keg-uk_name as well. Done. See: http://lists.freebsd.org/pipermail/svn-src-all/2013-November/077349.html Now if I run a kernel with VIMAGE enabled, and run the testcase mentioned here: http://lists.freebsd.org/pipermail/freebsd-current/2010-November/021280.html I get this on the console: Freed UMA keg (udp_inpcb) was not empty (30 items). Lost 3 pages of memory. Freed UMA keg (udpcb) was not empty (249 items). Lost 1 pages of memory. ifa_del_loopback_route: deletion failed: 48 This certainly helps narrow down where to look for problems. I'll see if I can post more fixes to eliminate these error messages. Thanks for your suggestion! -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
Well, the first step is figuring out which UMA zones are actually problematic. Isn't it logging which zones aren't empty? -a On 26 November 2013 20:01, Craig Rodrigues rodr...@freebsd.org wrote: Adrian, As part of looking at this bug in FreeNAS: Panic on new jail with vnet https://bugs.freenas.org/issues/3102 I started looking at this thread: http://lists.freebsd.org/pipermail/freebsd-current/2010-November/021280.html Based on this clue from Thiery: http://lists.freebsd.org/pipermail/freebsd-current/2010-November/021291.html I fixed some VIMAGE related memory leaks related to routetbl. I also fixed some VIMAGE related kernel panics when I tried creating and deleting lots of jails: http://lists.freebsd.org/pipermail/svn-src-all/2013-November/077178.html http://lists.freebsd.org/pipermail/svn-src-all/2013-November/077192.html http://lists.freebsd.org/pipermail/svn-src-all/2013-November/077195.html I haven't eliminated all the Freed UMA keg was not empty error messages. Do you have any recommendations for how I can track down the last few UMA related memory leaks? I am new to debugging UMA so would appreciate any pointers that you may have. -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Wed, Nov 27, 2013 at 2:05 AM, Adrian Chadd adr...@freebsd.org wrote: Well, the first step is figuring out which UMA zones are actually problematic. Isn't it logging which zones aren't empty? The error messages on the console look like this: Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. That doesn't really tell which UMA zone isn't empty. Is there some technqiue to figure this out? I tried vmstat -z and vmstat -m, but while those gave clues, it didn't point to which UMA zone was leaking. Is there some other technique or tool that I can use? -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
Modify that function to print out keg-uk_name as well. -a On 27 November 2013 12:45, Craig Rodrigues rodr...@freebsd.org wrote: On Wed, Nov 27, 2013 at 2:05 AM, Adrian Chadd adr...@freebsd.org wrote: Well, the first step is figuring out which UMA zones are actually problematic. Isn't it logging which zones aren't empty? The error messages on the console look like this: Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. That doesn't really tell which UMA zone isn't empty. Is there some technqiue to figure this out? I tried vmstat -z and vmstat -m, but while those gave clues, it didn't point to which UMA zone was leaking. Is there some other technique or tool that I can use? -- Craig ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Thu, 18 Nov 2010, Thierry Herbelot wrote: Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Thu, 18 Nov 2010, Thierry Herbelot wrote: Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Wed, 17 Nov 2010, Thierry Herbelot wrote: As promised, here are the full logs (in attachment) This is a serial console log showing the command loop that triggers the bug on a debug kernel and ensuing DDB session. the obvious problem line is : routetbl 2684 303890K 3469 (further tests showed an increase of the routetbl malloc zone by 4MBytes for each vnet jail creation/destruction cycle) Hmm, I had fixed that (somewhere). I'll see where the patch went. You are on 8.1-RELEASE or -STABLE? This will be for a -release, thus 8.1 for now, but we will switch to 8.2 ASAP Well wait; I am not sure the changes are in SVN at all. I'll get back to you. OK for us : we are still in the process of deploying the solution, but still we will test any patch you can forward ;-) Do you have FLOWTABLE enabled in your kernel config? /bz -- Bjoern A. Zeeb You have to have visions! ks Going to jail sucks -- bz All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Thu, 18 Nov 2010, Thierry Herbelot wrote: Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Thu, 18 Nov 2010, Thierry Herbelot wrote: Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Wed, 17 Nov 2010, Thierry Herbelot wrote: As promised, here are the full logs (in attachment) This is a serial console log showing the command loop that triggers the bug on a debug kernel and ensuing DDB session. the obvious problem line is : routetbl 2684 303890K 3469 (further tests showed an increase of the routetbl malloc zone by 4MBytes for each vnet jail creation/destruction cycle) Hmm, I had fixed that (somewhere). I'll see where the patch went. You are on 8.1-RELEASE or -STABLE? This will be for a -release, thus 8.1 for now, but we will switch to 8.2 ASAP Well wait; I am not sure the changes are in SVN at all. I'll get back to you. OK for us : we are still in the process of deploying the solution, but still we will test any patch you can forward ;-) Do you have FLOWTABLE enabled in your kernel config? Hello, I will check at work on monday, but I think so, as the kernel configuration was derived from GENERIC. BTW, the setup we wanted (ie with VIMAGE) is showing good stability (with 8.2- stable) as we have removed the need to start and stop the jails. we will definitely deploy 8.2 with VIMAGE and use it for our test bench. TfH /bz ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Fri, 14 Jan 2011, Thierry Herbelot wrote: Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Thu, 18 Nov 2010, Thierry Herbelot wrote: Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Thu, 18 Nov 2010, Thierry Herbelot wrote: Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Wed, 17 Nov 2010, Thierry Herbelot wrote: As promised, here are the full logs (in attachment) This is a serial console log showing the command loop that triggers the bug on a debug kernel and ensuing DDB session. the obvious problem line is : routetbl 2684 303890K 3469 (further tests showed an increase of the routetbl malloc zone by 4MBytes for each vnet jail creation/destruction cycle) Hmm, I had fixed that (somewhere). I'll see where the patch went. You are on 8.1-RELEASE or -STABLE? This will be for a -release, thus 8.1 for now, but we will switch to 8.2 ASAP Well wait; I am not sure the changes are in SVN at all. I'll get back to you. OK for us : we are still in the process of deploying the solution, but still we will test any patch you can forward ;-) Do you have FLOWTABLE enabled in your kernel config? Hello, I will check at work on monday, but I think so, as the kernel configuration was derived from GENERIC. that would have been the reason for the memory leaks. BTW, the setup we wanted (ie with VIMAGE) is showing good stability (with 8.2- stable) as we have removed the need to start and stop the jails. we will but with that you won't see it anymore. /bz -- Bjoern A. Zeeb You have to have visions! ks Going to jail sucks -- bz All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Wed, 17 Nov 2010, Thierry Herbelot wrote: As promised, here are the full logs (in attachment) This is a serial console log showing the command loop that triggers the bug on a debug kernel and ensuing DDB session. the obvious problem line is : routetbl 2684 303890K 3469 (further tests showed an increase of the routetbl malloc zone by 4MBytes for each vnet jail creation/destruction cycle) Hmm, I had fixed that (somewhere). I'll see where the patch went. You are on 8.1-RELEASE or -STABLE? /bz -- Bjoern A. Zeeb Welcome a new stage of life. ks Going to jail sucks -- bz All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Wed, 17 Nov 2010, Thierry Herbelot wrote: As promised, here are the full logs (in attachment) This is a serial console log showing the command loop that triggers the bug on a debug kernel and ensuing DDB session. the obvious problem line is : routetbl 2684 303890K 3469 (further tests showed an increase of the routetbl malloc zone by 4MBytes for each vnet jail creation/destruction cycle) Hmm, I had fixed that (somewhere). I'll see where the patch went. You are on 8.1-RELEASE or -STABLE? This will be for a -release, thus 8.1 for now, but we will switch to 8.2 ASAP Thanks TfH /bz ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Thu, 18 Nov 2010, Thierry Herbelot wrote: Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Wed, 17 Nov 2010, Thierry Herbelot wrote: As promised, here are the full logs (in attachment) This is a serial console log showing the command loop that triggers the bug on a debug kernel and ensuing DDB session. the obvious problem line is : routetbl 2684 303890K 3469 (further tests showed an increase of the routetbl malloc zone by 4MBytes for each vnet jail creation/destruction cycle) Hmm, I had fixed that (somewhere). I'll see where the patch went. You are on 8.1-RELEASE or -STABLE? This will be for a -release, thus 8.1 for now, but we will switch to 8.2 ASAP Well wait; I am not sure the changes are in SVN at all. I'll get back to you. /bz -- Bjoern A. Zeeb Welcome a new stage of life. ks Going to jail sucks -- bz All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Thu, 18 Nov 2010, Thierry Herbelot wrote: Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net a écrit On Wed, 17 Nov 2010, Thierry Herbelot wrote: As promised, here are the full logs (in attachment) This is a serial console log showing the command loop that triggers the bug on a debug kernel and ensuing DDB session. the obvious problem line is : routetbl 2684 303890K 3469 (further tests showed an increase of the routetbl malloc zone by 4MBytes for each vnet jail creation/destruction cycle) Hmm, I had fixed that (somewhere). I'll see where the patch went. You are on 8.1-RELEASE or -STABLE? This will be for a -release, thus 8.1 for now, but we will switch to 8.2 ASAP Well wait; I am not sure the changes are in SVN at all. I'll get back to you. OK for us : we are still in the process of deploying the solution, but still we will test any patch you can forward ;-) TfH /bz ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
Brandon Gooch jamesbrandongo...@... writes: On Wed, Nov 17, 2010 at 7:17 AM, Bjoern A. Zeeb bzeeb-li...@... wrote: On Wed, 17 Nov 2010, Marko Zec wrote: Actually, we never seriously discussed or revisited the issue with separate UMA pools for each vnet instance. My original motivation when O introduced separate UMA pools was primarily in making it easier to spot resource leaks, and to prove the correctness of the whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps the time has come to move on. Having said that, and given that the current VIMAGE resource allocation model is far from being optimal (a lot of memory sits reserved but 99% unused, and cannot be reclaimed later on vnet teardown), perhaps it's time that we reconsider using unified UMA pools. I think there is a misunderstanding here; it can be reclaimed by the time we have the teardown properly sorted out and it will immediately help normal non-VIMAGE systems under memory pressure as well. The problem is that, at least for TCP (and UDP in one special case as I found after lots of testing), we are no there yet. After that, when it comes to resource usage, I am still wondering how trasz' resource limits will plug into that. By the time we can see those coming together we should be able to decide whether to go left or right. I've been running into this memory exhaustion as well, having a need to stop and start my VIMAGE jails frequently. I'm confident that the proper solution will be worked out, but I wonder what sort of time-frame we may be looking at -- is VIMAGE expected to be production by 9.0-RELEASE? Also, does anyone know the current status of trasz's work (which I believe is to be completed December of this year)? I hope it's still on schedule :) -Brandon Hey there, I have been experiencing a similar problem. I am running Freebsd8.1 64bit Release and after closing my server application I come across this type of message Freed UMA keg was not empty (672 items). Lost 4 pages of memory The more virtual nodes I use the more of these message I have. Someone told me this does not mean anything but after reading this it seems I should be worried. It does show up in my log files as well. I wonder if running a fsck will resolve any issues after I have this problem? Hopefully we can find a solution because I rely on my application heavily. Thanks, Dan ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Thu, 18 Nov 2010, Dan wrote: Hi, I have been experiencing a similar problem. I am running Freebsd8.1 64bit Release and after closing my server application I come across this type of message Freed UMA keg was not empty (672 items). Lost 4 pages of memory The more virtual nodes I use the more of these message I have. Someone told me this does not mean anything but after reading this it seems I should be worried. It does show up in my log files as well. I wonder if running a fsck will resolve any issues after I have this problem? Hopefully we can find a solution because I rely on my application heavily. No, fsck will not help you anything; this is a memory (RAM) not a disk storage/file system problem. /bz -- Bjoern A. Zeeb Welcome a new stage of life. ks Going to jail sucks -- bz All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Wednesday 17 November 2010 06:57:53 Bjoern A. Zeeb wrote: On Wed, 17 Nov 2010, Thierry Herbelot wrote: Hi, first of all freebsd-virtualization@ is the better list for this; Cc:ed. We are using FreeBSD + VIMAGE at work, and we have seen an annoying problem : there seems to be a memory leak in the kernel, which eventually causes a panic. (yes, we have seen the following message : WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.) Configuring a network interface in a jail with vnet enabled, then removing that jail causes these messages. In dmesg: [...] Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled (with attached VIMAGE kernel config file). The following commands reproduce the bug: jail -l -u root -c path=/ name=foo persist vnet jexec foo ifconfig lo0 127.0.0.1/8 jail -r foo Running it too many times exhausts kernel memory and crashes the machine. The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN -head. The problem has been present since day 1 and is still present up to HEAD. This is about type stability and teardown. So far we are no able to actually free TCP (and UDP in SVN) states as they might still be accesses after free. It's a general problem in the network stack and has been implemented as a measure to circumvent panics in those cases. Actually, we never seriously discussed or revisited the issue with separate UMA pools for each vnet instance. My original motivation when O introduced separate UMA pools was primarily in making it easier to spot resource leaks, and to prove the correctness of the whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps the time has come to move on. Having said that, and given that the current VIMAGE resource allocation model is far from being optimal (a lot of memory sits reserved but 99% unused, and cannot be reclaimed later on vnet teardown), perhaps it's time that we reconsider using unified UMA pools. Note that the original VIMAGE prototype on FreeBSD 4.x from 2002 or 2003 used (mostly) unified memory zones, so there's no technical reason why this couldn't work again in FreeBSD current or 8-stable. Cheers, Marko I am not sure if that's actually what's biting you wrt to memory or if it's something else. Unfortunately boot logs and kernel configs don't help much; enable ddb and get show uma and show malloc reports after the crash (or watch vmstat -z and vmstat -m) after every 50 jail restarts. I might have some patches for a couple of things but cannot (yet) help with the additional (duplicate) UMA zones showing up at each iteration (for the -z case). /bz ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Wed, 17 Nov 2010, Marko Zec wrote: Actually, we never seriously discussed or revisited the issue with separate UMA pools for each vnet instance. My original motivation when O introduced separate UMA pools was primarily in making it easier to spot resource leaks, and to prove the correctness of the whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps the time has come to move on. Having said that, and given that the current VIMAGE resource allocation model is far from being optimal (a lot of memory sits reserved but 99% unused, and cannot be reclaimed later on vnet teardown), perhaps it's time that we reconsider using unified UMA pools. I think there is a misunderstanding here; it can be reclaimed by the time we have the teardown properly sorted out and it will immediately help normal non-VIMAGE systems under memory pressure as well. The problem is that, at least for TCP (and UDP in one special case as I found after lots of testing), we are no there yet. After that, when it comes to resource usage, I am still wondering how trasz' resource limits will plug into that. By the time we can see those coming together we should be able to decide whether to go left or right. /bz -- Bjoern A. Zeeb Welcome a new stage of life. ks Going to jail sucks -- bz All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Wed, Nov 17, 2010 at 7:17 AM, Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net wrote: On Wed, 17 Nov 2010, Marko Zec wrote: Actually, we never seriously discussed or revisited the issue with separate UMA pools for each vnet instance. My original motivation when O introduced separate UMA pools was primarily in making it easier to spot resource leaks, and to prove the correctness of the whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps the time has come to move on. Having said that, and given that the current VIMAGE resource allocation model is far from being optimal (a lot of memory sits reserved but 99% unused, and cannot be reclaimed later on vnet teardown), perhaps it's time that we reconsider using unified UMA pools. I think there is a misunderstanding here; it can be reclaimed by the time we have the teardown properly sorted out and it will immediately help normal non-VIMAGE systems under memory pressure as well. The problem is that, at least for TCP (and UDP in one special case as I found after lots of testing), we are no there yet. After that, when it comes to resource usage, I am still wondering how trasz' resource limits will plug into that. By the time we can see those coming together we should be able to decide whether to go left or right. I've been running into this memory exhaustion as well, having a need to stop and start my VIMAGE jails frequently. I'm confident that the proper solution will be worked out, but I wonder what sort of time-frame we may be looking at -- is VIMAGE expected to be production by 9.0-RELEASE? Also, does anyone know the current status of trasz's work (which I believe is to be completed December of this year)? I hope it's still on schedule :) -Brandon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
