RE: [Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]

2005-05-31 Thread Dieter Maurer
Andy Yates wrote at 2005-5-30 22:52 -0500:
 ...
When you fixed the TempStorage the leak indeed slowed down
significantly.  However, Zope still leaks memory.  If left uncheck Zope
will still consume all available memory and crash the server.  We have
resigned ourselves to restarting Zope every few weeks.  

Looks acceptable, doesn't it?

Such leaks can be caused by so called memory fragmentation.

Memory fragmentation describes the situation where the
available free memory is spread over large amounts of small blocks.

When you need a somewhat larger block, then a new large block is
requested from the OS and cut down into smaller pieces. Thus, this
block, too, may finally end up as a set of small blocks.

Memory fragmentation is something like the entropy increase in the
world...

Restarting gives you a new world with small entropy...

-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


RE: [Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]

2005-05-30 Thread Dieter Maurer
Please stay on the list!
Readded...

Dylan Jay wrote at 2005-5-29 23:31 +1000:
 - Memory grows and mostly doesn't shrink.
 
 That's normal behavior.

Normal as in desirable or normal as in known bug? 

Normal as in extremely difficult to avoid.

When you read a book about memory management and
look at the operating system provided API for
memory management you will understand why.

 - Its not due to objects that can't be collected by the GC
 - Its not due to the ZODB cache
 
 That leaves the following causes I think
 - RAMCache etc and module/class variables. Either mine, plone's or
 zope's.

If the reference counts as shown by Control_Panel -- Debug information
do not show them, then the leaked objects are neither class
nor ExtensionClass instances and are probably elementary
Python types.

Some moths ago, I found such a leak in the TempStorage
implementation. However, this is fixed in recent Zope versions
(I think since 2.7.5).

 ...
 Python has no big chance. The memory management options of the
 underlaying operating systems are quite limited and Python
 uses the C memory management at its lowest level.
 Thus, it must live with what C provides.

I don't really understand the implications of what your saying. Are you
saying that oses don't allow python to free memory?

Something like that:

  posix provides two ways to add address space for a process:
  brk and mmap.
  
  brk allows to add and release space at the *end*
  of the address space. You cannot release such space unless
  a complete range is free at the very end!

  mmap can add (and release) space anywhere but only
  in large chunks and only at corresponding boundaries.
  You can use mmap only for large memory blocks managed
  atomically (i.e. allocated and released as a whole).

  The C malloc usually uses brk to allocate large chunks
  of memory and subdivides them as needed (for efficiency reasons).
  While it would be possible to check for the possibility
  to release such blocks again, it would be expensive (and
  not so often successful). Therefore, it is rarely done.

Read more in a book about memory management!

 ...
am I right in saying that using
the catalog involves lots of btree objects which are objects and would not
be affected by the memory issue above?

BTree objects are objects and would be counted in
the Reference Counts display of Control_Panel -- Debug Information.
You should be able to detect leaks involving them by analysing
this display (do not forget to flush the caches before you trust
the reference counts!).

 ...
 Did I not tell you about a special Python build to
 analyse memory leaks?

No.

You can build Python to account for any allocated object.
However, all extensions must then be build with this same
option (otherwise, the object structures do not agree and chaos results).

You will need many hours to analyse the thousands of objects
to distinguish between the leaked objects and the one rightfully
used by the application...

-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


RE: [Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]

2005-05-30 Thread Andy Yates
 
 Some moths ago, I found such a leak in the TempStorage
 implementation. However, this is fixed in recent Zope versions
 (I think since 2.7.5).

When you fixed the TempStorage the leak indeed slowed down
significantly.  However, Zope still leaks memory.  If left uncheck Zope
will still consume all available memory and crash the server.  We have
resigned ourselves to restarting Zope every few weeks.  

Andy

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]

2005-05-13 Thread Dieter Maurer
Dylan Jay wrote at 2005-5-13 14:01 +1000:
Just in case anyone can shed anymore light on zope memory woes, here is 
where I've got to in my investigation:

- Memory grows and mostly doesn't shrink.

That's normal behavior.

- Its not due to objects that can't be collected by the GC
- Its not due to the ZODB cache

That leaves the following causes I think
- RAMCache etc and module/class variables. Either mine, plone's or zope's.

When I remember right, then also the reference counts do not
go up.

In this case, the leak must be caused by elementary Python types
(more precisely by objects the type of with is statically
allocated) -- if Python objects cause the leak at all.

- Pythons design of not releasing memory from small objects back to the OS
http://www.sauria.com/~twl/conferences/pycon2005/20050325/Improving%20Python's%20Memory%20Allocator.html

Python has no big chance. The memory management options of the
underlaying operating systems are quite limited and Python
uses the C memory management at its lowest level.
Thus, it must live with what C provides.

   Has anyone tried to determine how much of an average catalog search is 
256kb?

What is an average catalog search?
How do you measure the size of a catalog search?

- Some other kind of reference being help somewhere in the zope machinery.

Did I not tell you about a special Python build to
analyse memory leaks?

-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )