Hi Gustaf Thank you for your replays. I am a bit slow with my responses, sorry about that.
I see the same issue on all servers now - original observation that some servers are OK was wrong. This is due to different maxconnections configuration, which shadowed the memory leak. Yes - I am using standard memory allocator in Tcl. Would you recommend to try something different? Thanks for TclLib explanations - very helpful. What would you recommend as a replacement for nssession module? All I need is to keep track on user sessions, being able to set and read data from the session. Sessions shall be persistent, survive server restarts. Thanks, ~ Alex. On Mon, Mar 17, 2008 at 7:28 AM, Gustaf Neumann <[EMAIL PROTECTED]> wrote: > Dear Alex, > > Didn't you mention earlier, that the bloat happens only on > one of your machines? Was this a wrong indication? > > If you have a well behaved system, you can increase "maxconnections" to > e.g. 10000 (the parameter denotes the number of connections to be > served by a connection thread before it restarts) and set minthreads to eg. > 5. This does not solve the memory-leak problem but should > slowdown the memory hunger, if connection thread restart > causes the growth. I assume, you are using the standard > zippy memory allocator from tcl. > > I would not recommend to restart based on the size, since the server > will do this at a time you might not want it (e.g. during a demo). A > nightly reboot is the better option. The size certainly depends on > the amount of tcl code times number of threads and the state management > (use of connection-spanning tcl vars, or use of nsv). > > Concerning tcllib and MD5: > tcllib contains actually 3 versions of md5: > a) a tcl-based code > b) a version based on critcl (see tcllib*/md5/md5c.tcl) > c) a version based on trf > > During startup, it tries to use the "best" version. > (b) and (c) are accelerated versions using c-code, this might > explain the speed differences you are experiencing. I would > recommend to test first with the tcl-only version and check > the memory growth. If it has no effect, it seems safe to > use the accelerated versions. > > Concerning xotcl: > > - there is no special handling needed concening memory freeing. > as long you take care, that only those objects survive a connection > that you want. Typical applications load the frequently uses > classes during startup to speed processing of requests. > > - xotcl-core of OpenACS has support to ease deallocation > of temporary objects at the end of a connection (e.g. > Object o -destroy_on_cleanup) > > - [Class allinstances] returns the classes. To check the objects, > use [Object allinstances] > > Concerning Sig 11: > This seems to come from libnssession.so. I am not sure, > how frequently this is used (we don't use it). It seems as > if the source for the crash is there. > > Is it possible for you to test without nssession (crashes, > memory consumption)? > > -gustaf > > > > > > Hello Tom, Gustaf > > > > I am still trying to pin down the memory leak problem. Could you > > answer few more questions, please? > > > > 1. It looks like that nsd process grows ~1.5 - 2.0 MB every time a > > thread dies due to timeout or number of connections (and new one > > starts). > > Changing configuration just makes the bloating process slower of faster. > > I restart the nsd process when it reaches 0.5GB in size, but I hope > > that's a reasonable number. > > I tried valgrind it, but couldn't make any conclusions. Any better > > ideas how to debug it? > > > > 2. TclLib shall not harm IMHO, it's Tcl- only, no binary code. Please > > confirm that. > > Besides, I need it only for MD5, and standalone md5 Tcl > > implementations for some reason seems to be much slower. > > If TclLib is risky, what would be the best package/module for MD5 ? > > mhash may be? > > > > 3. XoTcl - I verified that I do not leave any objects which I created, > > they all destroyed. (at least that's what [Class allinstances] > > reports) > > However, Gustaf noted that it still may need special handling. What > > kind of handling? Could you point me to the code in openacs that does > > that? > > For me it would be a big hit if I need to rewrite without XoTcl, so I > > would like to make it working. > > > > 4. Memory leaks aside, nsd quite frequently dies on Sig 11. Sometimes > > just a few seconds after being started. > > I remember having similar issues some year+ ago, but those were fixed > > quickly. Anyone experiences the same? > > > > I didnt' debug it in depth yet, that's what gdb said when I quickly tried > it: > > > > Program received signal SIGSEGV, Segmentation fault. > > [Switching to Thread -1296041040 (LWP 32284)] > > 0xb7338ba5 in Ns_CookieGet () from > > /aolserver_bin/aolserver4.5/lib/libnssession.so > > > > Thanks, > > ~ Alex. > > > > > > On Tue, Mar 4, 2008 at 3:29 AM, Gustaf Neumann <[EMAIL PROTECTED]> wrote: > > > >> Dear Alex, > >> > >> The good news is that it bloats only on one server, > >> so the problem must on that configuration. > >> > >> There were a couple of memory leaks reported for postgres 7.4, > >> not sure if these hit you (google around): > >> http://www.postgresql.org/docs/7.4/static/release-6-5.html > >> > >> do you have a chance to install on the affected machine > >> as well postgres 8.2 (maybe in addition to 7.4)? > >> it would be interesting to see, if you still see a bloat. > >> > >> i found a small glitch in your "crude script": > >> The last line of the xotcl install should read: > >> > >> execute make install-aol > >> > >> instead of > >> > >> execute make install > >> > >> -gustaf neumann > >> > >> Alex schrieb: > >> > >> > >> > >>> Tom, Gustaf, All > >>> > >> > > >> > Yes, that happens even if the same page is requested. > >> > Could you tell me more about pre-queue filters? > >> > > >> > The interesting thing is that I only see this memory leak issue on one > >> > of my servers. > >> > All servers run x86 Debian, though might be slightly differently > >> > dist-upgraded, Postgres version differs 7.4 vs 8.2 > >> > The website code itself, nsd config files are exactly the same on all > servers. > >> > > >> > Over the weekend I tried to rebuild tcl, tcllib, xotcl, aolserver. > >> > The details how I did it can be found here > >> > > http://alex-andr.org/notes/tiki-index.php?page=AOLServer+setup+with+XOTcl > >> > I used latest versions however, tcl 8.4.18, tcllib 1.10, xotcl 1.6.0 > >> > > >> > I even made a crude script that fetches the sources and builds a > >> > stand-alone aolserver installation, > >> > in order to make sure the build results are reproducible. > >> > http://alex-andr.org/notes/tiki-index.php?page=nsd+setup+script > >> > > >> > Still, even when all [hopefully] binaries are being copied to all > >> > servers, only one of them > >> > experience the same issue with huge memory leak on each page request. > >> > > >> > Any insightful thoughts? > >> > > >> > Thanks, > >> > ~ Alex. > >> > > >> > > >> > On Fri, Feb 29, 2008 at 6:23 PM, Tom Jackson <[EMAIL PROTECTED]> wrote: > >> > > >> >> Actually 2MB / request sounds like a massive memory leak. Maybe > something > >> >> isn't getting cleaned up? > >> >> > >> >> Question: does this happen if you request the same page over and > over? > >> >> > >> >> I discovered one way to get a massive leak in 4.5: register a > prequeue filter. > >> >> > >> >> Note that both XoTCL and tcllib are compiled third party modules, > which is > >> >> usually where you get memory leaks, not in the modules which have > been used > >> >> with AOLserver for years. My guess would be tcllib, or just failing > to > >> >> release something big. > >> >> > >> >> tom jackson > >> >> > >> >> > >> >> On Friday 29 February 2008 11:47, Alex wrote: > >> >> > What would be a "normal" size of 32 bit nsd process? How can it be > reduced? > >> >> > One of my servers only has 1G or ram, and I am forced to restart > nsd > >> >> > every so often, when it uses up almost all the memory. > >> >> > Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each > page > >> >> > request . > >> >> > >> >> > >> >> > >> >> > >> >> -- > >> >> AOLserver - http://www.aolserver.com/ > >> >> > >> >> To Remove yourself from this list, simply send an email to <[EMAIL > PROTECTED]> with the > >> >> body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. > >> >> > >> >> > >> > > >> > > >> > -- > >> > AOLserver - http://www.aolserver.com/ > >> > > >> > To Remove yourself from this list, simply send an email to <[EMAIL > PROTECTED]> with the > >> > body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. > >> > > >> > >> > >> -- > >> AOLserver - http://www.aolserver.com/ > >> > >> To Remove yourself from this list, simply send an email to <[EMAIL > PROTECTED]> with the > >> body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. > >> > >> > > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
