RE: No memory leak! (was: Re: [Zope] Memory Leak in Session Data Container)
Andy Yates wrote at 2006-4-17 16:35 -0500: ... /Control_Panel/Database/temporary/manage_cacheParameters Total number of objects in the database369885 Total number of objects in memory from all caches15341 Target number of objects in memory per cache5000 It seems odd to me that the total objects in the db are growing so large. This looks fishy, indeed! Somehow, the cleanup code of tempstorage.TemporaryStorage.TemporaryStorage does not seem to work in your installation while it does in mine (Zope 2.8.1 with some modifications). If I understand you correctly a workaround would be to change the number of objects per cache from 5000 to something more reasonable. Only to get down the total number of objects in memory from all caches. As I know now (and contrary to my original expectation) it would not affect the Total number of objects in the database. Even if my original expectation were right, the number in the database is far too high to be explained by the cached objects. Where is this configured and what is reasonable? It moved from the ZODB (configured via the ZMI) to the Zope configuration file (one of the keys of the zope_db section). I no longer know when this move happened. I think, it was Zope 2.7 (but may be wrong). 5, 50, 500. I do not see an item for this in the config file. If you expect X sessions, then you may want to keep about the most recent X/4 sessions in the cache (rather than repickling them over and over again). The exact value depends on your preferences (your balance for speed versus memory consumption). To stress it again: your large number of objects in the database is not explained by the objects in the cache. Something else must be wrong in your installation. Limiting the ZODB cache size for the temporary storage will not drastically effect your memory consumption. Should there be any cache at all on an in-memory temp db? The cache does not save IO, of course, but unpickling time which may be considerably as well. A recent profile showed: IO 3.9 s Unpickle 1.3 s However, it was a singular measurement. You probably cannot generalize it. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: No memory leak! (was: Re: [Zope] Memory Leak in Session Data Container)
Leak, bug, improper configuration, what ever. ;-) With the onDelete in place for 1 week now I checked the temporary Cache Parameters as you requested. Since I put the onDelete on this server memory use has still grown, but at a much slower rate. I think it will go about 10 days like this instead of 1 before using up all system memory. For whatever reason we have 7 caches on this server. I do not see where this is configured. /Control_Panel/Database/temporary/manage_cacheParameters Total number of objects in the database 363761 Total number of objects in memory from all caches34952 Target number of objects in memory per cache5000 I flushed the cache and about 1 hour after flushing, memory use has not gone down at all. During that hour objects in all caches went from 0 to 15341. Objects in the transient object container are still holding steady at 2000-3000. Before the onDelete they grew without bound. /Control_Panel/Database/temporary/manage_cacheParameters Total number of objects in the database369885 Total number of objects in memory from all caches15341 Target number of objects in memory per cache5000 It seems odd to me that the total objects in the db are growing so large. If I understand you correctly a workaround would be to change the number of objects per cache from 5000 to something more reasonable. Where is this configured and what is reasonable? 5, 50, 500. I do not see an item for this in the config file. Should there be any cache at all on an in-memory temp db? Andy Yates -Original Message- From: Dieter Maurer [mailto:[EMAIL PROTECTED] Sent: Friday, April 14, 2006 7:54 AM To: Andy Yates; zope@zope.org Subject: No memory leak! (was: Re: [Zope] Memory Leak in Session Data Container) Dieter Maurer wrote at 2006-4-11 22:14 +0200: Andy Yates wrote at 2006-4-10 12:43 -0500: ... Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located. Congratulations! I will look into this within the next days (in Zope 2.8.1, however). I checked this: There is no memory leak related to session handling in Zope 2.8.1! ... The ZODB cache may keep your session objects (and thereby its content) alive. Almost surely, this is what happened in Andy's test. Here is a description how I checked for the existence of a memory leak: I instrumented the pickles in tempstorage.TemporaryStorage.TemporaryStorage such that I can find out the number of pickles stored (similar to the reference counts in Control_Panel/DebugInfo). I performed a mass test, creating thousands of sessions (similar to Andy's test). The pickle number stored by TemporaryStorage grew as expected (as it stores the sessions). I forced all session objects maintained by /temp_folder/session_data to be discarded. Creating a single new session caused almost all TemporaryStorage pickles to be released. However, the ZODB caches still contained about 20.000 session objects. It was very surprising (and contrary to my previous message) that the storage flushed most of its pickles while references to them were still in the ZODB caches. This looks like a bug (but not a memory leak). I flushed the ZODB caches. I repeated the test and verified that the number of pickles remaining in TemporaryStorage does not increase. Results: * no memory leak! * maybe something that looks like a memory leak caused by the ZODB caches which may hold up to 20.000 (with standard configuration) no longer used objects A more senseful configuration can work around this. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
No memory leak! (was: Re: [Zope] Memory Leak in Session Data Container)
Dieter Maurer wrote at 2006-4-11 22:14 +0200: Andy Yates wrote at 2006-4-10 12:43 -0500: ... Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located. Congratulations! I will look into this within the next days (in Zope 2.8.1, however). I checked this: There is no memory leak related to session handling in Zope 2.8.1! ... The ZODB cache may keep your session objects (and thereby its content) alive. Almost surely, this is what happened in Andy's test. Here is a description how I checked for the existence of a memory leak: I instrumented the pickles in tempstorage.TemporaryStorage.TemporaryStorage such that I can find out the number of pickles stored (similar to the reference counts in Control_Panel/DebugInfo). I performed a mass test, creating thousands of sessions (similar to Andy's test). The pickle number stored by TemporaryStorage grew as expected (as it stores the sessions). I forced all session objects maintained by /temp_folder/session_data to be discarded. Creating a single new session caused almost all TemporaryStorage pickles to be released. However, the ZODB caches still contained about 20.000 session objects. It was very surprising (and contrary to my previous message) that the storage flushed most of its pickles while references to them were still in the ZODB caches. This looks like a bug (but not a memory leak). I flushed the ZODB caches. I repeated the test and verified that the number of pickles remaining in TemporaryStorage does not increase. Results: * no memory leak! * maybe something that looks like a memory leak caused by the ZODB caches which may hold up to 20.000 (with standard configuration) no longer used objects A more senseful configuration can work around this. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Memory Leak in Session Data Container
Andy Yates wrote at 2006-4-10 12:43 -0500: ... Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located. Congratulations! I will look into this within the next days (in Zope 2.8.1, however). ... CONCLUSION To me it looks like data placed in the session object are not getting deleted when the session object expires. Anything placed in a session object is essentially leaked. When the onDelete script explicitly removes the contents of the session data object the memory leak stops. Comment? Questions? Fixes? ;-) The ZODB cache may keep your session objects (and thereby its content) alive. As you probably know, each ZODB connection has an associated cache containing the recently accessed objects. Objects in the cache can be ghosts (they contain no data) or real objects. Ghosts disappear from the cache as soon as the last reference in the application is deleted. For real objects, however, the cache holds an extra reference and thereby prevents the object release. For the storage, an object held by the cache looks exactly equal to an object held be the application. It cannot get rid of it. You can try to flush (minimize) the caches associated with the temporary storage and see whether this releases the memory (the release may be postponed, as the TempStorage performs garbage collection only at certain times). If this should be your problem, you can reduce the size of the ZODB caches. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Memory Leak in Session Data Container
Good day everyone! I hope your weekend was as good as mine. I didn't have to work and I got to watch my son's first soccer (football) game! For the past couple of years (I think) the Session Data Container has been leaking memory. During this time several leaks have been patch but at least for our servers these fixes have only slowed down the process. As our use of Zope grows and traffic to our site increases so does the memory usage of Zope. Unless Zope is restarted it will eventually use all physical memory and swap space and crash the server. Over the past month the problem has once again grown to the point where we must restarted Zope daily, sometimes even every 8 hours. I don't know why this problem does not seem more wide spread. My guess is because our site has a large load and we make extensive use of session variables. Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located. TEST CASE To reproduce the problem I used the following: VA Linux Server w/ 1Gig Fedora Core 3 Zope 2.7.8-final, python 2.3.4, linux2 Python 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] top and/or snmp to watch memory usage Step 1: In Zope I created the following script: session = context.REQUEST.SESSION s= 'Andy' * 102400; session['sessiontest1']=s The size of 's' depends purely on how fast you would like Zope to eat memory. On my server this will consume all physical memory in about 1000 calls. Step 2: Changed the session_data object timeout to 1 minute. This is just to cause the session objects to expire in 1 minute to hurry the test along. Changed the Max subobjects to 0 (unlimited). I don't know if this is needed, but it is how we have our production servers set. Step 3: Use ab (apache bench) to call the script 1000 times using 10 concurrent threads. This causes Zope to consume over 90% of the memory as one would expect. The transient object container (/temp_folder/session_data) will now show 1009 objects. I don't know where the 6-9 extra come from.? Step 4: Wait 1-2 minutes for the objects to expire. Note: Use a browser or ab to access some other resource in Zope and or the script a number of times. This is needed to tickle the transient object container's expiration/garbage collection system. (You've got to keep those buckets moving) As expected the transient object container now shows 0 objects. However, Zope never releases the memory. You can wait minutes, hours, days what ever all the while tickling Zope. Zope will NEVER release the memory until it is restarted. Oh, and new requests just use more memory. Zope is NOT just hanging on to it for future use. THE WORK ARROUND I created a script called onDelete Parameter List sdo, toc sdo.clear() Then in /temp_folder/session_data I set the Script to call when objects are deleted to /onDelete Now when the tests are run the same amount of memory is consumed, but it is released back to the system when the session objects expire. CONCLUSION To me it looks like data placed in the session object are not getting deleted when the session object expires. Anything placed in a session object is essentially leaked. When the onDelete script explicitly removes the contents of the session data object the memory leak stops. Comment? Questions? Fixes? ;-) Thanks! Andy Yates ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Memory Leak in Session Data Container
At Monday 10/4/2006 14:43, Andy Yates wrote: Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located. Quoting Andreas Jung on this list a few weeks ago: Please put such _bug_ into the bugtracker. Reporting bugs to the mailinglist hoping that someone picks it up is insane, sorry. That is, go to http://www.zope.org/Collectors/Zope and file a bug report. Gabriel Genellina Softlab SRL ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
