Re: VIMAGE: Freed UMA keg was not empty
On Wednesday 17 November 2010 06:57:53 Bjoern A. Zeeb wrote: On Wed, 17 Nov 2010, Thierry Herbelot wrote: Hi, first of all freebsd-virtualization@ is the better list for this; Cc:ed. We are using FreeBSD + VIMAGE at work, and we have seen an annoying problem : there seems to be a memory leak in the kernel, which eventually causes a panic. (yes, we have seen the following message : WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.) Configuring a network interface in a jail with vnet enabled, then removing that jail causes these messages. In dmesg: [...] Freed UMA keg was not empty (203 items). Lost 1 pages of memory. Freed UMA keg was not empty (36 items). Lost 2 pages of memory. The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled (with attached VIMAGE kernel config file). The following commands reproduce the bug: jail -l -u root -c path=/ name=foo persist vnet jexec foo ifconfig lo0 127.0.0.1/8 jail -r foo Running it too many times exhausts kernel memory and crashes the machine. The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN -head. The problem has been present since day 1 and is still present up to HEAD. This is about type stability and teardown. So far we are no able to actually free TCP (and UDP in SVN) states as they might still be accesses after free. It's a general problem in the network stack and has been implemented as a measure to circumvent panics in those cases. Actually, we never seriously discussed or revisited the issue with separate UMA pools for each vnet instance. My original motivation when O introduced separate UMA pools was primarily in making it easier to spot resource leaks, and to prove the correctness of the whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps the time has come to move on. Having said that, and given that the current VIMAGE resource allocation model is far from being optimal (a lot of memory sits reserved but 99% unused, and cannot be reclaimed later on vnet teardown), perhaps it's time that we reconsider using unified UMA pools. Note that the original VIMAGE prototype on FreeBSD 4.x from 2002 or 2003 used (mostly) unified memory zones, so there's no technical reason why this couldn't work again in FreeBSD current or 8-stable. Cheers, Marko I am not sure if that's actually what's biting you wrt to memory or if it's something else. Unfortunately boot logs and kernel configs don't help much; enable ddb and get show uma and show malloc reports after the crash (or watch vmstat -z and vmstat -m) after every 50 jail restarts. I might have some patches for a couple of things but cannot (yet) help with the additional (duplicate) UMA zones showing up at each iteration (for the -z case). /bz ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Wed, 17 Nov 2010, Marko Zec wrote: Actually, we never seriously discussed or revisited the issue with separate UMA pools for each vnet instance. My original motivation when O introduced separate UMA pools was primarily in making it easier to spot resource leaks, and to prove the correctness of the whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps the time has come to move on. Having said that, and given that the current VIMAGE resource allocation model is far from being optimal (a lot of memory sits reserved but 99% unused, and cannot be reclaimed later on vnet teardown), perhaps it's time that we reconsider using unified UMA pools. I think there is a misunderstanding here; it can be reclaimed by the time we have the teardown properly sorted out and it will immediately help normal non-VIMAGE systems under memory pressure as well. The problem is that, at least for TCP (and UDP in one special case as I found after lots of testing), we are no there yet. After that, when it comes to resource usage, I am still wondering how trasz' resource limits will plug into that. By the time we can see those coming together we should be able to decide whether to go left or right. /bz -- Bjoern A. Zeeb Welcome a new stage of life. ks Going to jail sucks -- bz All my daemons like it! http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
global resource accounting on a per-jail basis
Hi, Are there any plans to add CPU (at least) accounting on a per jail basis, so that I can distinguish all the CPU cycles used by jail X. While sar could be useful, it only notes user id and that's not unique across jails. I know that some kind of resource limiting is planned and that would naturally require the accounting suggested. Regards, Mark Blackman Exonetric ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: global resource accounting on a per-jail basis
Mark Blackman wrote: Hi, I know that some kind of resource limiting is planned and that would naturally require the accounting suggested. /me bothers to google and finds... http://lists.freebsd.org/pipermail/freebsd-announce/2010-July/001335.html never mind. :) Cheers, Mark ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: VIMAGE: Freed UMA keg was not empty
On Wed, Nov 17, 2010 at 7:17 AM, Bjoern A. Zeeb bzeeb-li...@lists.zabbadoz.net wrote: On Wed, 17 Nov 2010, Marko Zec wrote: Actually, we never seriously discussed or revisited the issue with separate UMA pools for each vnet instance. My original motivation when O introduced separate UMA pools was primarily in making it easier to spot resource leaks, and to prove the correctness of the whole VIMAGE / VNET thing. Having more or less achieved those goals, perhaps the time has come to move on. Having said that, and given that the current VIMAGE resource allocation model is far from being optimal (a lot of memory sits reserved but 99% unused, and cannot be reclaimed later on vnet teardown), perhaps it's time that we reconsider using unified UMA pools. I think there is a misunderstanding here; it can be reclaimed by the time we have the teardown properly sorted out and it will immediately help normal non-VIMAGE systems under memory pressure as well. The problem is that, at least for TCP (and UDP in one special case as I found after lots of testing), we are no there yet. After that, when it comes to resource usage, I am still wondering how trasz' resource limits will plug into that. By the time we can see those coming together we should be able to decide whether to go left or right. I've been running into this memory exhaustion as well, having a need to stop and start my VIMAGE jails frequently. I'm confident that the proper solution will be worked out, but I wonder what sort of time-frame we may be looking at -- is VIMAGE expected to be production by 9.0-RELEASE? Also, does anyone know the current status of trasz's work (which I believe is to be completed December of this year)? I hope it's still on schedule :) -Brandon ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: kern/152047: [vimage] [panic] TUN\TAP under jail with vimage crashes system
Synopsis: [vimage] [panic] TUN\TAP under jail with vimage crashes system State-Changed-From-To: open-analyzed State-Changed-By: bz State-Changed-When: Thu Nov 18 06:13:54 UTC 2010 State-Changed-Why: Problem is well known. Responsible-Changed-From-To: freebsd-bugs-freebsd-virtualization Responsible-Changed-By: bz Responsible-Changed-When: Thu Nov 18 06:13:54 UTC 2010 Responsible-Changed-Why: Re-assign to freebsd virtualization list as it's a VNET specific problem. http://www.freebsd.org/cgi/query-pr.cgi?pr=152047 ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
