Re: [AOLSERVER] 32- vs. 64-bit mem bloat
Hi Gustaf Thank you for your replays. I am a bit slow with my responses, sorry about that. I see the same issue on all servers now - original observation that some servers are OK was wrong. This is due to different maxconnections configuration, which shadowed the memory leak. Yes - I am using standard memory allocator in Tcl. Would you recommend to try something different? Thanks for TclLib explanations - very helpful. What would you recommend as a replacement for nssession module? All I need is to keep track on user sessions, being able to set and read data from the session. Sessions shall be persistent, survive server restarts. Thanks, ~ Alex. On Mon, Mar 17, 2008 at 7:28 AM, Gustaf Neumann <[EMAIL PROTECTED]> wrote: > Dear Alex, > > Didn't you mention earlier, that the bloat happens only on > one of your machines? Was this a wrong indication? > > If you have a well behaved system, you can increase "maxconnections" to > e.g. 1 (the parameter denotes the number of connections to be > served by a connection thread before it restarts) and set minthreads to eg. > 5. This does not solve the memory-leak problem but should > slowdown the memory hunger, if connection thread restart > causes the growth. I assume, you are using the standard > zippy memory allocator from tcl. > > I would not recommend to restart based on the size, since the server > will do this at a time you might not want it (e.g. during a demo). A > nightly reboot is the better option. The size certainly depends on > the amount of tcl code times number of threads and the state management > (use of connection-spanning tcl vars, or use of nsv). > > Concerning tcllib and MD5: > tcllib contains actually 3 versions of md5: > a) a tcl-based code > b) a version based on critcl (see tcllib*/md5/md5c.tcl) > c) a version based on trf > > During startup, it tries to use the "best" version. > (b) and (c) are accelerated versions using c-code, this might > explain the speed differences you are experiencing. I would > recommend to test first with the tcl-only version and check > the memory growth. If it has no effect, it seems safe to > use the accelerated versions. > > Concerning xotcl: > > - there is no special handling needed concening memory freeing. >as long you take care, that only those objects survive a connection >that you want. Typical applications load the frequently uses >classes during startup to speed processing of requests. > > - xotcl-core of OpenACS has support to ease deallocation >of temporary objects at the end of a connection (e.g. >Object o -destroy_on_cleanup) > > - [Class allinstances] returns the classes. To check the objects, >use [Object allinstances] > > Concerning Sig 11: > This seems to come from libnssession.so. I am not sure, > how frequently this is used (we don't use it). It seems as > if the source for the crash is there. > > Is it possible for you to test without nssession (crashes, > memory consumption)? > > -gustaf > > > > > > Hello Tom, Gustaf > > > > I am still trying to pin down the memory leak problem. Could you > > answer few more questions, please? > > > > 1. It looks like that nsd process grows ~1.5 - 2.0 MB every time a > > thread dies due to timeout or number of connections (and new one > > starts). > > Changing configuration just makes the bloating process slower of faster. > > I restart the nsd process when it reaches 0.5GB in size, but I hope > > that's a reasonable number. > > I tried valgrind it, but couldn't make any conclusions. Any better > > ideas how to debug it? > > > > 2. TclLib shall not harm IMHO, it's Tcl- only, no binary code. Please > > confirm that. > > Besides, I need it only for MD5, and standalone md5 Tcl > > implementations for some reason seems to be much slower. > > If TclLib is risky, what would be the best package/module for MD5 ? > > mhash may be? > > > > 3. XoTcl - I verified that I do not leave any objects which I created, > > they all destroyed. (at least that's what [Class allinstances] > > reports) > > However, Gustaf noted that it still may need special handling. What > > kind of handling? Could you point me to the code in openacs that does > > that? > > For me it would be a big hit if I need to rewrite without XoTcl, so I > > would like to make it working. > > > > 4. Memory leaks aside, nsd quite frequently dies on Sig 11. Sometimes > > just a few seconds after being started. > > I remember having similar issues some year+ ago, but those were fixed > > quickly. Anyone experiences the same? > > > > I didnt' debug it in depth yet, that's what gdb said when I quickly tried > it: > > > > Program received signal SIGSEGV, Segmentation fault. > > [Switching to Thread -1296041040 (LWP 32284)] > > 0xb7338ba5 in Ns_CookieGet () from > > /aolserver_bin/aolserver4.5/lib/libnssession.so > > > > Thanks, > > ~ Alex. > > > > > > On Tue, Mar 4, 2008 at 3:29 AM, Gustaf Neu
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
Dear Alex, Didn't you mention earlier, that the bloat happens only on one of your machines? Was this a wrong indication? If you have a well behaved system, you can increase "maxconnections" to e.g. 1 (the parameter denotes the number of connections to be served by a connection thread before it restarts) and set minthreads to eg. 5. This does not solve the memory-leak problem but should slowdown the memory hunger, if connection thread restart causes the growth. I assume, you are using the standard zippy memory allocator from tcl. I would not recommend to restart based on the size, since the server will do this at a time you might not want it (e.g. during a demo). A nightly reboot is the better option. The size certainly depends on the amount of tcl code times number of threads and the state management (use of connection-spanning tcl vars, or use of nsv). Concerning tcllib and MD5: tcllib contains actually 3 versions of md5: a) a tcl-based code b) a version based on critcl (see tcllib*/md5/md5c.tcl) c) a version based on trf During startup, it tries to use the "best" version. (b) and (c) are accelerated versions using c-code, this might explain the speed differences you are experiencing. I would recommend to test first with the tcl-only version and check the memory growth. If it has no effect, it seems safe to use the accelerated versions. Concerning xotcl: - there is no special handling needed concening memory freeing. as long you take care, that only those objects survive a connection that you want. Typical applications load the frequently uses classes during startup to speed processing of requests. - xotcl-core of OpenACS has support to ease deallocation of temporary objects at the end of a connection (e.g. Object o -destroy_on_cleanup) - [Class allinstances] returns the classes. To check the objects, use [Object allinstances] Concerning Sig 11: This seems to come from libnssession.so. I am not sure, how frequently this is used (we don't use it). It seems as if the source for the crash is there. Is it possible for you to test without nssession (crashes, memory consumption)? -gustaf Hello Tom, Gustaf I am still trying to pin down the memory leak problem. Could you answer few more questions, please? 1. It looks like that nsd process grows ~1.5 - 2.0 MB every time a thread dies due to timeout or number of connections (and new one starts). Changing configuration just makes the bloating process slower of faster. I restart the nsd process when it reaches 0.5GB in size, but I hope that's a reasonable number. I tried valgrind it, but couldn't make any conclusions. Any better ideas how to debug it? 2. TclLib shall not harm IMHO, it's Tcl- only, no binary code. Please confirm that. Besides, I need it only for MD5, and standalone md5 Tcl implementations for some reason seems to be much slower. If TclLib is risky, what would be the best package/module for MD5 ? mhash may be? 3. XoTcl - I verified that I do not leave any objects which I created, they all destroyed. (at least that's what [Class allinstances] reports) However, Gustaf noted that it still may need special handling. What kind of handling? Could you point me to the code in openacs that does that? For me it would be a big hit if I need to rewrite without XoTcl, so I would like to make it working. 4. Memory leaks aside, nsd quite frequently dies on Sig 11. Sometimes just a few seconds after being started. I remember having similar issues some year+ ago, but those were fixed quickly. Anyone experiences the same? I didnt' debug it in depth yet, that's what gdb said when I quickly tried it: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1296041040 (LWP 32284)] 0xb7338ba5 in Ns_CookieGet () from /aolserver_bin/aolserver4.5/lib/libnssession.so Thanks, ~ Alex. On Tue, Mar 4, 2008 at 3:29 AM, Gustaf Neumann <[EMAIL PROTECTED]> wrote: Dear Alex, The good news is that it bloats only on one server, so the problem must on that configuration. There were a couple of memory leaks reported for postgres 7.4, not sure if these hit you (google around): http://www.postgresql.org/docs/7.4/static/release-6-5.html do you have a chance to install on the affected machine as well postgres 8.2 (maybe in addition to 7.4)? it would be interesting to see, if you still see a bloat. i found a small glitch in your "crude script": The last line of the xotcl install should read: execute make install-aol instead of execute make install -gustaf neumann Alex schrieb: Tom, Gustaf, All > > Yes, that happens even if the same page is requested. > Could you tell me more about pre-queue filters? > > The interesting thing is that I only see this memory leak issue on one > of my servers. > All servers run x86 Debian, though might be slightly differently > dist-upgraded, Postgres version differs 7.4 vs 8.2 > The website code itself, nsd config files are exactly the same on all se
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
I don't know how fast it is, but here is an aolserver module for doing MD5 hashing: http://gwpics.ontiblue.com/nsmd5.c On Mon, Mar 17, 2008 at 12:20 AM, Alex <[EMAIL PROTECTED]> wrote: > 2. TclLib shall not harm IMHO, it's Tcl- only, no binary code. Please > confirm that. > Besides, I need it only for MD5, and standalone md5 Tcl > implementations for some reason seems to be much slower. > If TclLib is risky, what would be the best package/module for MD5 ? > mhash may be? > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
Hello Tom, Gustaf I am still trying to pin down the memory leak problem. Could you answer few more questions, please? 1. It looks like that nsd process grows ~1.5 - 2.0 MB every time a thread dies due to timeout or number of connections (and new one starts). Changing configuration just makes the bloating process slower of faster. I restart the nsd process when it reaches 0.5GB in size, but I hope that's a reasonable number. I tried valgrind it, but couldn't make any conclusions. Any better ideas how to debug it? 2. TclLib shall not harm IMHO, it's Tcl- only, no binary code. Please confirm that. Besides, I need it only for MD5, and standalone md5 Tcl implementations for some reason seems to be much slower. If TclLib is risky, what would be the best package/module for MD5 ? mhash may be? 3. XoTcl - I verified that I do not leave any objects which I created, they all destroyed. (at least that's what [Class allinstances] reports) However, Gustaf noted that it still may need special handling. What kind of handling? Could you point me to the code in openacs that does that? For me it would be a big hit if I need to rewrite without XoTcl, so I would like to make it working. 4. Memory leaks aside, nsd quite frequently dies on Sig 11. Sometimes just a few seconds after being started. I remember having similar issues some year+ ago, but those were fixed quickly. Anyone experiences the same? I didnt' debug it in depth yet, that's what gdb said when I quickly tried it: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1296041040 (LWP 32284)] 0xb7338ba5 in Ns_CookieGet () from /aolserver_bin/aolserver4.5/lib/libnssession.so Thanks, ~ Alex. On Tue, Mar 4, 2008 at 3:29 AM, Gustaf Neumann <[EMAIL PROTECTED]> wrote: > Dear Alex, > > The good news is that it bloats only on one server, > so the problem must on that configuration. > > There were a couple of memory leaks reported for postgres 7.4, > not sure if these hit you (google around): > http://www.postgresql.org/docs/7.4/static/release-6-5.html > > do you have a chance to install on the affected machine > as well postgres 8.2 (maybe in addition to 7.4)? > it would be interesting to see, if you still see a bloat. > > i found a small glitch in your "crude script": > The last line of the xotcl install should read: > > execute make install-aol > > instead of > > execute make install > > -gustaf neumann > > Alex schrieb: > > > > Tom, Gustaf, All > > > > Yes, that happens even if the same page is requested. > > Could you tell me more about pre-queue filters? > > > > The interesting thing is that I only see this memory leak issue on one > > of my servers. > > All servers run x86 Debian, though might be slightly differently > > dist-upgraded, Postgres version differs 7.4 vs 8.2 > > The website code itself, nsd config files are exactly the same on all > servers. > > > > Over the weekend I tried to rebuild tcl, tcllib, xotcl, aolserver. > > The details how I did it can be found here > > http://alex-andr.org/notes/tiki-index.php?page=AOLServer+setup+with+XOTcl > > I used latest versions however, tcl 8.4.18, tcllib 1.10, xotcl 1.6.0 > > > > I even made a crude script that fetches the sources and builds a > > stand-alone aolserver installation, > > in order to make sure the build results are reproducible. > > http://alex-andr.org/notes/tiki-index.php?page=nsd+setup+script > > > > Still, even when all [hopefully] binaries are being copied to all > > servers, only one of them > > experience the same issue with huge memory leak on each page request. > > > > Any insightful thoughts? > > > > Thanks, > > ~ Alex. > > > > > > On Fri, Feb 29, 2008 at 6:23 PM, Tom Jackson <[EMAIL PROTECTED]> wrote: > > > >> Actually 2MB / request sounds like a massive memory leak. Maybe something > >> isn't getting cleaned up? > >> > >> Question: does this happen if you request the same page over and over? > >> > >> I discovered one way to get a massive leak in 4.5: register a prequeue > filter. > >> > >> Note that both XoTCL and tcllib are compiled third party modules, which > is > >> usually where you get memory leaks, not in the modules which have been > used > >> with AOLserver for years. My guess would be tcllib, or just failing to > >> release something big. > >> > >> tom jackson > >> > >> > >> On Friday 29 February 2008 11:47, Alex wrote: > >> > What would be a "normal" size of 32 bit nsd process? How can it be > reduced? > >> > One of my servers only has 1G or ram, and I am forced to restart nsd > >> > every so often, when it uses up almost all the memory. > >> > Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each page > >> > request . > >> > >> > >> > >> > >> -- > >> AOLserver - http://www.aolserver.com/ > >> > >> To Remove yourself from this list, simply send an email to <[EMAIL > PROTECTED]> with the > >> body of "SIGNOFF AOLSERVER" in the email messag
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
Hello Gustaf, Thanks for the suggestions - I will try upgrading to Postgres 8.2. As to the script, there was a problem with install-aol, and install seem to work ok. However I resolved that issue - will update the script. Is there any good and clear documentation for the options available in nsd.tcl configuration file? In particular, new (v4.5) pool parameters (-maxthreads -minthreads -maxconns, etc.) are notoriously unclear. I tried to play with them and found that e.g. -maxconns parameter hugely affects the size of the leak. Would be great to read about all those parameters. Thanks, ~ Alex. On Tue, Mar 4, 2008 at 2:29 AM, Gustaf Neumann <[EMAIL PROTECTED]> wrote: > Dear Alex, > > The good news is that it bloats only on one server, > so the problem must on that configuration. > > There were a couple of memory leaks reported for postgres 7.4, > not sure if these hit you (google around): > http://www.postgresql.org/docs/7.4/static/release-6-5.html > > do you have a chance to install on the affected machine > as well postgres 8.2 (maybe in addition to 7.4)? > it would be interesting to see, if you still see a bloat. > > i found a small glitch in your "crude script": > The last line of the xotcl install should read: > > execute make install-aol > > instead of > > execute make install > > -gustaf neumann > > Alex schrieb: > > > > Tom, Gustaf, All > > > > Yes, that happens even if the same page is requested. > > Could you tell me more about pre-queue filters? > > > > The interesting thing is that I only see this memory leak issue on one > > of my servers. > > All servers run x86 Debian, though might be slightly differently > > dist-upgraded, Postgres version differs 7.4 vs 8.2 > > The website code itself, nsd config files are exactly the same on all > servers. > > > > Over the weekend I tried to rebuild tcl, tcllib, xotcl, aolserver. > > The details how I did it can be found here > > http://alex-andr.org/notes/tiki-index.php?page=AOLServer+setup+with+XOTcl > > I used latest versions however, tcl 8.4.18, tcllib 1.10, xotcl 1.6.0 > > > > I even made a crude script that fetches the sources and builds a > > stand-alone aolserver installation, > > in order to make sure the build results are reproducible. > > http://alex-andr.org/notes/tiki-index.php?page=nsd+setup+script > > > > Still, even when all [hopefully] binaries are being copied to all > > servers, only one of them > > experience the same issue with huge memory leak on each page request. > > > > Any insightful thoughts? > > > > Thanks, > > ~ Alex. > > > > > > On Fri, Feb 29, 2008 at 6:23 PM, Tom Jackson <[EMAIL PROTECTED]> wrote: > > > >> Actually 2MB / request sounds like a massive memory leak. Maybe something > >> isn't getting cleaned up? > >> > >> Question: does this happen if you request the same page over and over? > >> > >> I discovered one way to get a massive leak in 4.5: register a prequeue > filter. > >> > >> Note that both XoTCL and tcllib are compiled third party modules, which > is > >> usually where you get memory leaks, not in the modules which have been > used > >> with AOLserver for years. My guess would be tcllib, or just failing to > >> release something big. > >> > >> tom jackson > >> > >> > >> On Friday 29 February 2008 11:47, Alex wrote: > >> > What would be a "normal" size of 32 bit nsd process? How can it be > reduced? > >> > One of my servers only has 1G or ram, and I am forced to restart nsd > >> > every so often, when it uses up almost all the memory. > >> > Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each page > >> > request . > >> > >> > >> > >> > >> -- > >> AOLserver - http://www.aolserver.com/ > >> > >> To Remove yourself from this list, simply send an email to <[EMAIL > PROTECTED]> with the > >> body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. > >> > >> > > > > > > -- > > AOLserver - http://www.aolserver.com/ > > > > To Remove yourself from this list, simply send an email to <[EMAIL > PROTECTED]> with the > > body of "SIGNOFF AOLSERVER" in the email message. You can leave the > Subject: field of your email blank. > > > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to <[EMAIL > PROTECTED]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: > field of your email blank. > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
Dear Alex, The good news is that it bloats only on one server, so the problem must on that configuration. There were a couple of memory leaks reported for postgres 7.4, not sure if these hit you (google around): http://www.postgresql.org/docs/7.4/static/release-6-5.html do you have a chance to install on the affected machine as well postgres 8.2 (maybe in addition to 7.4)? it would be interesting to see, if you still see a bloat. i found a small glitch in your "crude script": The last line of the xotcl install should read: execute make install-aol instead of execute make install -gustaf neumann Alex schrieb: Tom, Gustaf, All Yes, that happens even if the same page is requested. Could you tell me more about pre-queue filters? The interesting thing is that I only see this memory leak issue on one of my servers. All servers run x86 Debian, though might be slightly differently dist-upgraded, Postgres version differs 7.4 vs 8.2 The website code itself, nsd config files are exactly the same on all servers. Over the weekend I tried to rebuild tcl, tcllib, xotcl, aolserver. The details how I did it can be found here http://alex-andr.org/notes/tiki-index.php?page=AOLServer+setup+with+XOTcl I used latest versions however, tcl 8.4.18, tcllib 1.10, xotcl 1.6.0 I even made a crude script that fetches the sources and builds a stand-alone aolserver installation, in order to make sure the build results are reproducible. http://alex-andr.org/notes/tiki-index.php?page=nsd+setup+script Still, even when all [hopefully] binaries are being copied to all servers, only one of them experience the same issue with huge memory leak on each page request. Any insightful thoughts? Thanks, ~ Alex. On Fri, Feb 29, 2008 at 6:23 PM, Tom Jackson <[EMAIL PROTECTED]> wrote: Actually 2MB / request sounds like a massive memory leak. Maybe something isn't getting cleaned up? Question: does this happen if you request the same page over and over? I discovered one way to get a massive leak in 4.5: register a prequeue filter. Note that both XoTCL and tcllib are compiled third party modules, which is usually where you get memory leaks, not in the modules which have been used with AOLserver for years. My guess would be tcllib, or just failing to release something big. tom jackson On Friday 29 February 2008 11:47, Alex wrote: > What would be a "normal" size of 32 bit nsd process? How can it be reduced? > One of my servers only has 1G or ram, and I am forced to restart nsd > every so often, when it uses up almost all the memory. > Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each page > request . -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
Can you narrow down what code actually runs on this one page? If it happens on any one page, is there some shared code? It is very likely, which would be lucky for you, that some large data structure is being created with each request and not deleted. As Gustaf said, I guess that XOTcl objects might need special handling as well, this isn't really different than creating data and forgetting to delete it, just another potential source to forget to cleanup. Don't worry about the prequeue filters, they shouldn't be used, but they exist, and I just happened to stumble across them, in AOLserver 4.5. My guess is that an iterp is created and never deleted, as they run prior to assigning a request to a conn thread. tom jackson On Monday 03 March 2008 08:55, Alex wrote: > Tom, Gustaf, All > > Yes, that happens even if the same page is requested. > Could you tell me more about pre-queue filters? > > The interesting thing is that I only see this memory leak issue on one > of my servers. > All servers run x86 Debian, though might be slightly differently > dist-upgraded, Postgres version differs 7.4 vs 8.2 > The website code itself, nsd config files are exactly the same on all > servers. > > Over the weekend I tried to rebuild tcl, tcllib, xotcl, aolserver. > The details how I did it can be found here > http://alex-andr.org/notes/tiki-index.php?page=AOLServer+setup+with+XOTcl > I used latest versions however, tcl 8.4.18, tcllib 1.10, xotcl 1.6.0 > > I even made a crude script that fetches the sources and builds a > stand-alone aolserver installation, > in order to make sure the build results are reproducible. > http://alex-andr.org/notes/tiki-index.php?page=nsd+setup+script > > Still, even when all [hopefully] binaries are being copied to all > servers, only one of them > experience the same issue with huge memory leak on each page request. > > Any insightful thoughts? > > Thanks, > ~ Alex. > > On Fri, Feb 29, 2008 at 6:23 PM, Tom Jackson <[EMAIL PROTECTED]> wrote: > > Actually 2MB / request sounds like a massive memory leak. Maybe something > > isn't getting cleaned up? > > > > Question: does this happen if you request the same page over and over? > > > > I discovered one way to get a massive leak in 4.5: register a prequeue > > filter. > > > > Note that both XoTCL and tcllib are compiled third party modules, which > > is usually where you get memory leaks, not in the modules which have been > > used with AOLserver for years. My guess would be tcllib, or just failing > > to release something big. > > > > tom jackson > > > > On Friday 29 February 2008 11:47, Alex wrote: > > > What would be a "normal" size of 32 bit nsd process? How can it be > > > reduced? One of my servers only has 1G or ram, and I am forced to > > > restart nsd every so often, when it uses up almost all the memory. > > > Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each page > > > request . > > > > -- > > AOLserver - http://www.aolserver.com/ > > > > To Remove yourself from this list, simply send an email to > > <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the > > email message. You can leave the Subject: field of your email blank. > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to > <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the > email message. You can leave the Subject: field of your email blank. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
Tom, Gustaf, All Yes, that happens even if the same page is requested. Could you tell me more about pre-queue filters? The interesting thing is that I only see this memory leak issue on one of my servers. All servers run x86 Debian, though might be slightly differently dist-upgraded, Postgres version differs 7.4 vs 8.2 The website code itself, nsd config files are exactly the same on all servers. Over the weekend I tried to rebuild tcl, tcllib, xotcl, aolserver. The details how I did it can be found here http://alex-andr.org/notes/tiki-index.php?page=AOLServer+setup+with+XOTcl I used latest versions however, tcl 8.4.18, tcllib 1.10, xotcl 1.6.0 I even made a crude script that fetches the sources and builds a stand-alone aolserver installation, in order to make sure the build results are reproducible. http://alex-andr.org/notes/tiki-index.php?page=nsd+setup+script Still, even when all [hopefully] binaries are being copied to all servers, only one of them experience the same issue with huge memory leak on each page request. Any insightful thoughts? Thanks, ~ Alex. On Fri, Feb 29, 2008 at 6:23 PM, Tom Jackson <[EMAIL PROTECTED]> wrote: > Actually 2MB / request sounds like a massive memory leak. Maybe something > isn't getting cleaned up? > > Question: does this happen if you request the same page over and over? > > I discovered one way to get a massive leak in 4.5: register a prequeue > filter. > > Note that both XoTCL and tcllib are compiled third party modules, which is > usually where you get memory leaks, not in the modules which have been used > with AOLserver for years. My guess would be tcllib, or just failing to > release something big. > > tom jackson > > > On Friday 29 February 2008 11:47, Alex wrote: > > What would be a "normal" size of 32 bit nsd process? How can it be reduced? > > One of my servers only has 1G or ram, and I am forced to restart nsd > > every so often, when it uses up almost all the memory. > > Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each page > > request . > > > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to <[EMAIL > PROTECTED]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: > field of your email blank. > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
Tom Jackson schrieb: Actually 2MB / request sounds like a massive memory leak. Maybe something isn't getting cleaned up? this would be as wel my first guess. Note, that the aolserver cleans after each request only the global variables in the toplevel (::) namespace, not in other namespaces. note as well, that in cases, where you create c-level objects in a connection tread (e.g. dom-nodes with tdom, or xotcl objects), one has to delete these as well, if they should not survive a request. xotcl-core in openacs has some support for making this more conveniant. Up to my experience, both aolserver 4.0.10 and aolserver 4.5 grow, but not in the numbers you are reporting. below is a monitoring result from one of our larger servers, running aolserver 4.0.10 in 64 bit on REL on POWER5+. The y axis shows memory usage in GB (labels are wrong). During the last week, we had up to 7 mio hits per day (1.3 mio page views), thursday and friday just around 3 mio hits. one can see the rapid growth after start-time (when more connection threads are started) and a certain slowdown in the growth rate (my interpretation is that this is from some saturation from memory fragmentation/over-allocation and less requests in the evening). If we would assume, that the growth rate is just depending on the hits, it would be about 1kb per hit (or 5kb per view). The "leaks" must be actually much less, since we are running up to 100 connection threads, with 9000 tcl procs each, also these take some significant memory. For us, the dailly reboot is not a problem. A view years ago, we did not reboot, but we figured out that for our applications, the aolserver/openacs system became about 5% slower every day running. After a reboot, it is again at "full speed". -gustaf neumann http://media.wu-wien.ac.at/download/nsdsize-week.png -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
Actually 2MB / request sounds like a massive memory leak. Maybe something isn't getting cleaned up? Question: does this happen if you request the same page over and over? I discovered one way to get a massive leak in 4.5: register a prequeue filter. Note that both XoTCL and tcllib are compiled third party modules, which is usually where you get memory leaks, not in the modules which have been used with AOLserver for years. My guess would be tcllib, or just failing to release something big. tom jackson On Friday 29 February 2008 11:47, Alex wrote: > What would be a "normal" size of 32 bit nsd process? How can it be reduced? > One of my servers only has 1G or ram, and I am forced to restart nsd > every so often, when it uses up almost all the memory. > Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each page > request . -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
On 2008.02.29, Alex <[EMAIL PROTECTED]> wrote: > What would be a "normal" size of 32 bit nsd process? How can it be > reduced? One of my servers only has 1G or ram, and I am forced to > restart nsd every so often, when it uses up almost all the memory. > Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each page > request . $ date Fri Feb 29 15:06:33 EST 2008 $ ps -Ao vsz,rss,stime,args | grep nsd 34400 12048 04:00 /home/aolserver/bin/nsd -it config/panoptic.com.tcl -b 96.56.31.42:80,96.56.31.42:443 -u www 186352 87620 04:00 /home/aolserver/bin/nsd -it config/vhosts.tcl -b 96.56.31.44:80,96.56.31.44:443 -u www That's Tcl 8.4.12 and AOLserver 4.0.10. It pretty much grows to the size you see and remains stable at that size. I still restart the process once a day at 04:00 US/Eastern time, just because I can. :-) -- Dossy -- Dossy Shiobara | [EMAIL PROTECTED] | http://dossy.org/ Panoptic Computer Network | http://panoptic.com/ "He realized the fastest way to change is to laugh at your own folly -- then you can let go and quickly move on." (p. 70) -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
What would be a "normal" size of 32 bit nsd process? How can it be reduced? One of my servers only has 1G or ram, and I am forced to restart nsd every so often, when it uses up almost all the memory. Running 4.5 + XoTCL + tcllib, it seams to grow up to 2MB on each page request . Thanks, ~ Alex. On Fri, Feb 29, 2008 at 1:57 PM, Jeff Hobbs <[EMAIL PROTECTED]> wrote: > John Buckman wrote: > >> On Fri, Feb 29, 2008 at 05:41:44PM +0100, Juan Jos? del R?o [Simple > >> Option] wrote: > >> > >>> In my case I run AOLServer with customized code on top of it. No > >>> OpenACS. No modules except nspostgres. In 32 bits it works like a charm. > >>> No memory leaks. It simply works (damn fast!). > >> > >> Hm, so you see memory leaks when compiled for 64 bit, but none when > >> compiling the same code 32 bit - interesting. Can you reproduce the > >> leakage under Valgrind? > > > > FYI, I see no leaks under 64bit, but I do see a fair amount of "bloat". > > > > On 32bit Linux, my nsd process ran around 1.2gb, while under 64bits I'm > > around 7gb. > > > > However, no leaks. > > > > After a few days my nsd process size does not continue to grow, it is > > stable at ~17gb (I have a 10gb BerkeleyDB cache in process). > > > > I'm not clear on why anyone would need 64bit Aolserver unless they > > needed to use more than 2gb of RAM. > > > > I'm also not clear that the bloat is aolserver, and not tcl or other > > libraries I'm using. > > Move from 1.2gb to 7gb stable running state by just shifting from 32-bit > build to 64-bit build sounds a little fishy to me. I would expect, at > most, a 2x increase in process size (given theoretical changes). > However, theory and practice ... it could be that the alignment of data > in Tcl and/or AOLServer for 64-bit processes is just very poor, causing > massive overallocation. > > Jeff > > > -- > AOLserver - http://www.aolserver.com/ > > To Remove yourself from this list, simply send an email to <[EMAIL > PROTECTED]> with the > body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: > field of your email blank. > -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
The key, it seems to me, is that the memory footprint stablizes over time in a real world situation. A memory leak is best proven by watching memory grow while visiting a limited set of pages, essentially a stable load. Visiting a single page (with the same result) over and over should not cause any growth over time if there are no leaks in the code being executed. tom jackson On Friday 29 February 2008 10:57, Jeff Hobbs wrote: > Move from 1.2gb to 7gb stable running state by just shifting from 32-bit > build to 64-bit build sounds a little fishy to me. I would expect, at > most, a 2x increase in process size (given theoretical changes). > However, theory and practice ... it could be that the alignment of data > in Tcl and/or AOLServer for 64-bit processes is just very poor, causing > massive overallocation. -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
Re: [AOLSERVER] 32- vs. 64-bit mem bloat
John Buckman wrote: On Fri, Feb 29, 2008 at 05:41:44PM +0100, Juan Jos? del R?o [Simple Option] wrote: In my case I run AOLServer with customized code on top of it. No OpenACS. No modules except nspostgres. In 32 bits it works like a charm. No memory leaks. It simply works (damn fast!). Hm, so you see memory leaks when compiled for 64 bit, but none when compiling the same code 32 bit - interesting. Can you reproduce the leakage under Valgrind? FYI, I see no leaks under 64bit, but I do see a fair amount of "bloat". On 32bit Linux, my nsd process ran around 1.2gb, while under 64bits I'm around 7gb. However, no leaks. After a few days my nsd process size does not continue to grow, it is stable at ~17gb (I have a 10gb BerkeleyDB cache in process). I'm not clear on why anyone would need 64bit Aolserver unless they needed to use more than 2gb of RAM. I'm also not clear that the bloat is aolserver, and not tcl or other libraries I'm using. Move from 1.2gb to 7gb stable running state by just shifting from 32-bit build to 64-bit build sounds a little fishy to me. I would expect, at most, a 2x increase in process size (given theoretical changes). However, theory and practice ... it could be that the alignment of data in Tcl and/or AOLServer for 64-bit processes is just very poor, causing massive overallocation. Jeff -- AOLserver - http://www.aolserver.com/ To Remove yourself from this list, simply send an email to <[EMAIL PROTECTED]> with the body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
