[Zope-dev] Weird BTrees/ZODB interaction under Solaris
I made the following reproducable observation running Zope 2.5 under Solaris: I was indexing a CMF site using TextIndexNG with about 1600 objects. During the commit phase (after all objects were indexed), the Zope process grow from about 100MB to more than 1.2GB. I could track this down to TextIndexNG. There is a function written in C that inserts the list of splittet words into the BTrees for the forward and reverse index (by generating the WordIDs internally). Replacing this method with a pure Python implementation made Zope behave fine. Torturing the corresponding C implementation in an isolated environment did not show any memory leaks (neither under Solaris nor under Linux i386). To make it short: my C implementation has three args: the forward index, the reverse index and a list of words to be inserted. The indexes are an OIBTree and an IOBTree. Internally I call PyObject_SetItem() to store the word-wid mapping. This seems to work fine isolated but seems to blow up the ZODB for unknown reasons. Note that his behaviour is only reproducable under Solaris. I made some tests by indexing about 100.000 docs under Linux with TextIndexNG and the Zope memory usage was very fine (up to 150MB). Cheers, Andreas - -Andreas Jung http://www.andreas-jung.com - - EMail: andreas at andreas-jung.com - -Life is too short to (re)write parsers - - ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Weird BTrees/ZODB interaction under Solaris
I made the following reproducable observation running Zope 2.5 under Solaris: I was indexing a CMF site using TextIndexNG with about 1600 objects. During the commit phase (after all objects were indexed), the Zope process grow from about 100MB to more than 1.2GB. I could track this down to TextIndexNG. There is a function written in C that inserts the list of splittet words into the BTrees for the forward and reverse index (by generating the WordIDs internally). Replacing this method with a pure Python implementation made Zope behave fine. Torturing the corresponding C implementation in an isolated environment did not show any memory leaks (neither under Solaris nor under Linux i386). To make it short: my C implementation has three args: the forward index, the reverse index and a list of words to be inserted. The indexes are an OIBTree and an IOBTree. Internally I call PyObject_SetItem() to store the word-wid mapping. This seems to work fine isolated but seems to blow up the ZODB for unknown reasons. Note that his behaviour is only reproducable under Solaris. I made some tests by indexing about 100.000 docs under Linux with TextIndexNG and the Zope memory usage was very fine (up to 150MB). Could you try again with the C version, substituting a different malloc implementation, or Python 2.3 (which has effectively a different malloc for small blocks)? It could be a semi-bug in the platform malloc where the allocation and deallocation pattern of your test happens to cause worst-case fragmentation. Using Python code probably changes the allocation pattern enough to avoid the fragmentation. --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Weird BTrees/ZODB interaction under Solaris
--On Dienstag, 26. November 2002 08:35 -0500 Guido van Rossum [EMAIL PROTECTED] wrote: ... Could you try again with the C version, substituting a different malloc implementation, or Python 2.3 (which has effectively a different malloc for small blocks)? It could be a semi-bug in the platform malloc where the allocation and deallocation pattern of your test happens to cause worst-case fragmentation. Using Python code probably changes the allocation pattern enough to avoid the fragmentation. What do you mean with a different malloc implementation? Currently we are bound to Python 2.1.3 and Zope 2.5.X and we can not upgrade to other Zope/Python versions at least for the installation where I encountered the problem. Andreas - -Andreas Jung http://www.andreas-jung.com - - EMail: andreas at andreas-jung.com - -Life is too short to (re)write parsers - - ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Weird BTrees/ZODB interaction under Solaris
Andreas Jung writes: ... Internally I call PyObject_SetItem() to store the word-wid mapping. PyObject_SetItem creates new references for key and value. Is this what you expect? I know, this should affect all platforms and be independent of whether or not the code runs within or outside Zope. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
