RE: [Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]
Andy Yates wrote at 2005-5-30 22:52 -0500: ... When you fixed the TempStorage the leak indeed slowed down significantly. However, Zope still leaks memory. If left uncheck Zope will still consume all available memory and crash the server. We have resigned ourselves to restarting Zope every few weeks. Looks acceptable, doesn't it? Such leaks can be caused by so called memory fragmentation. Memory fragmentation describes the situation where the available free memory is spread over large amounts of small blocks. When you need a somewhat larger block, then a new large block is requested from the OS and cut down into smaller pieces. Thus, this block, too, may finally end up as a set of small blocks. Memory fragmentation is something like the entropy increase in the world... Restarting gives you a new world with small entropy... -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]
Please stay on the list! Readded... Dylan Jay wrote at 2005-5-29 23:31 +1000: - Memory grows and mostly doesn't shrink. That's normal behavior. Normal as in desirable or normal as in known bug? Normal as in extremely difficult to avoid. When you read a book about memory management and look at the operating system provided API for memory management you will understand why. - Its not due to objects that can't be collected by the GC - Its not due to the ZODB cache That leaves the following causes I think - RAMCache etc and module/class variables. Either mine, plone's or zope's. If the reference counts as shown by Control_Panel -- Debug information do not show them, then the leaked objects are neither class nor ExtensionClass instances and are probably elementary Python types. Some moths ago, I found such a leak in the TempStorage implementation. However, this is fixed in recent Zope versions (I think since 2.7.5). ... Python has no big chance. The memory management options of the underlaying operating systems are quite limited and Python uses the C memory management at its lowest level. Thus, it must live with what C provides. I don't really understand the implications of what your saying. Are you saying that oses don't allow python to free memory? Something like that: posix provides two ways to add address space for a process: brk and mmap. brk allows to add and release space at the *end* of the address space. You cannot release such space unless a complete range is free at the very end! mmap can add (and release) space anywhere but only in large chunks and only at corresponding boundaries. You can use mmap only for large memory blocks managed atomically (i.e. allocated and released as a whole). The C malloc usually uses brk to allocate large chunks of memory and subdivides them as needed (for efficiency reasons). While it would be possible to check for the possibility to release such blocks again, it would be expensive (and not so often successful). Therefore, it is rarely done. Read more in a book about memory management! ... am I right in saying that using the catalog involves lots of btree objects which are objects and would not be affected by the memory issue above? BTree objects are objects and would be counted in the Reference Counts display of Control_Panel -- Debug Information. You should be able to detect leaks involving them by analysing this display (do not forget to flush the caches before you trust the reference counts!). ... Did I not tell you about a special Python build to analyse memory leaks? No. You can build Python to account for any allocated object. However, all extensions must then be build with this same option (otherwise, the object structures do not agree and chaos results). You will need many hours to analyse the thousands of objects to distinguish between the leaked objects and the one rightfully used by the application... -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]
Some moths ago, I found such a leak in the TempStorage implementation. However, this is fixed in recent Zope versions (I think since 2.7.5). When you fixed the TempStorage the leak indeed slowed down significantly. However, Zope still leaks memory. If left uncheck Zope will still consume all available memory and crash the server. We have resigned ourselves to restarting Zope every few weeks. Andy ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]
Dylan Jay wrote at 2005-5-13 14:01 +1000: Just in case anyone can shed anymore light on zope memory woes, here is where I've got to in my investigation: - Memory grows and mostly doesn't shrink. That's normal behavior. - Its not due to objects that can't be collected by the GC - Its not due to the ZODB cache That leaves the following causes I think - RAMCache etc and module/class variables. Either mine, plone's or zope's. When I remember right, then also the reference counts do not go up. In this case, the leak must be caused by elementary Python types (more precisely by objects the type of with is statically allocated) -- if Python objects cause the leak at all. - Pythons design of not releasing memory from small objects back to the OS http://www.sauria.com/~twl/conferences/pycon2005/20050325/Improving%20Python's%20Memory%20Allocator.html Python has no big chance. The memory management options of the underlaying operating systems are quite limited and Python uses the C memory management at its lowest level. Thus, it must live with what C provides. Has anyone tried to determine how much of an average catalog search is 256kb? What is an average catalog search? How do you measure the size of a catalog search? - Some other kind of reference being help somewhere in the zope machinery. Did I not tell you about a special Python build to analyse memory leaks? -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )