I made the following reproducable observation running Zope 2.5 under

I was indexing a CMF site using TextIndexNG with about 1600 objects.
During the commit phase (after all objects were indexed), the Zope
process grow from about 100MB to more than 1.2GB.

I could track this down to TextIndexNG. There is a function written
in C that inserts the list of splittet words into the BTrees
for the forward and reverse index (by generating the WordIDs
internally). Replacing this method with a pure Python implementation
made Zope behave fine.

Torturing the corresponding C implementation in an isolated environment
did not show any memory leaks (neither under Solaris nor under Linux
i386). To make it short: my C implementation has three args: the forward
index, the reverse index and a list of words to be inserted. The indexes
are an OIBTree and an IOBTree. Internally I call PyObject_SetItem()
to store the word-wid mapping. This seems to work fine isolated but
seems to blow up the ZODB for unknown reasons.

Note that his behaviour is only reproducable under Solaris. I made some
tests by indexing about 100.000 docs under Linux with TextIndexNG and
the Zope memory usage was very fine (up to 150MB).


- Andreas Jung http://www.andreas-jung.com -
- EMail: andreas at andreas-jung.com -
- "Life is too short to (re)write parsers" -

Zope-Dev maillist - [EMAIL PROTECTED]
** No cross posts or HTML encoding! **
(Related lists - http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )

Reply via email to