I made the following reproducable observation running Zope 2.5 under
Solaris:
I was indexing a CMF site using TextIndexNG with about 1600 objects.
During the commit phase (after all objects were indexed), the Zope
process grow from about 100MB to more than 1.2GB.
I could track this down to TextIndexNG. There is a function written
in C that inserts the list of splittet words into the BTrees
for the forward and reverse index (by generating the WordIDs
internally). Replacing this method with a pure Python implementation
made Zope behave fine.
Torturing the corresponding C implementation in an isolated environment
did not show any memory leaks (neither under Solaris nor under Linux
i386). To make it short: my C implementation has three args: the forward
index, the reverse index and a list of words to be inserted. The indexes
are an OIBTree and an IOBTree. Internally I call PyObject_SetItem()
to store the word-wid mapping. This seems to work fine isolated but
seems to blow up the ZODB for unknown reasons.
Note that his behaviour is only reproducable under Solaris. I made some
tests by indexing about 100.000 docs under Linux with TextIndexNG and
the Zope memory usage was very fine (up to 150MB).
Could you try again with the C version, substituting a different
malloc implementation, or Python 2.3 (which has effectively a
different malloc for small blocks)? It could be a semi-bug in the
platform malloc where the allocation and deallocation pattern of your
test happens to cause worst-case fragmentation. Using Python code
probably changes the allocation pattern enough to avoid the
fragmentation.
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )