Re: [Zope-dev] Zope2 / Zope3 catalog index fork / divergence
2009/6/10 Tres Seaver tsea...@palladion.com: This effort could be the focus of a sprint team, if there was interest. Indexing fascinates me. I would be up for putting down some effort on some sort of improved indexing. I'm not sure how much of the time I would help and how much I would be in the way, but there you go. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope2 / Zope3 catalog index fork / divergence
On 10.06.09 18:10, Tres Seaver wrote: This is somewhat the same pattern as Andreas used to merge the ZPT implementations, I think. This effort could be the focus of a sprint team, if there was interest. The better example is TXNG3 where the Zope 2 and Zope 3 parts were completely refactored into eggs: http://pypi.python.org/pypi?%3Aaction=searchterm=zopyx.txngsubmit=search Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope2 / Zope3 catalog index fork / divergence
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tres Seaver wrote: I propose that we consider merging the two implementations, along the lines of the following: 1. Review the change history of the two codebases since the original fork, porting relevant changes from the Z2 indexes to the corresponding Z3 implementations. 2. Review / update the tests for the Z3 indexes, with a goal of 100% coverage, as well as improving the quality of the tests. The zope.index package now has 100% unit test coverage under Python 2.4, 2.5, and 2.6 (measured with nose + coverage: I couldn't figure out how to get the z3c.coverage bit to ignore stuff like conditional imports). 3. Re-implement the Z2 indexes as thin wrappers around the Z3 versions, ensuring that all existing Z2 tests continue to pass. This is somewhat the same pattern as Andreas used to merge the ZPT implementations, I think. This effort could be the focus of a sprint team, if there was interest. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKMM3M+gerLs4ltQ4RAnMAAKCpWqjs+h1gILkYzteXSFqpAYw5jACgnlXy QaVbg+eTgfvPcT3G0tFClKs= =iydI -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope2 / Zope3 catalog index fork / divergence
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christian Theune wrote: On Tue, 2009-06-09 at 20:07 -0400, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris McDonough wrote: Log message for revision 100775: - Port okascore.c optimization used in okapiiindex from Zope2 catalog implementation. This module is compiled conditionally, based on whether your environment has a working C compiler. - Don't use ``len(self._docweight)`` in okapiindex _search_wids method (obtaining the length of a BTree is very expensive at scale). Instead use self.documentCount(). Also a Zope2 port. Crap! this set of performance fixes was added *six years ago* to the Zope2 version from which zope.index was forked! Hasn't anybody been using this package? We're using this package. The text index I can only remember in a not-so-critical piece where performance currently is not an issue. Also, following checkins even today wouldn't have me remember that this code is shared. The code which now lives in 'zope.index' is a copy-and-rename-everything fork of what was in the Zope2 PluggableIndexes / ZCTextIndex products as of around Christmas, 2002. The fix I was replying to is *far* from the first one which didn't get ported from subsequent changes to the Zope2 indexes, which have obviously been in *heavy* production use. I propose that we consider merging the two implementations, along the lines of the following: 1. Review the change history of the two codebases since the original fork, porting relevant changes from the Z2 indexes to the corresponding Z3 implementations. 2. Review / update the tests for the Z3 indexes, with a goal of 100% coverage, as well as improving the quality of the tests. 3. Re-implement the Z2 indexes as thin wrappers around the Z3 versions, ensuring that all existing Z2 tests continue to pass. This is somewhat the same pattern as Andreas used to merge the ZPT implementations, I think. This effort could be the focus of a sprint team, if there was interest. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKL9sQ+gerLs4ltQ4RAksCAKDWRMu820PO+oRQNun0VC1LrFjvOACfU6pE uVtYzoliBjQgI5gark3ALTw= =+WAL -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope2 / Zope3 catalog index fork / divergence
+1 Thanks. Jim On Jun 10, 2009, at 12:10 PM, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christian Theune wrote: On Tue, 2009-06-09 at 20:07 -0400, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris McDonough wrote: Log message for revision 100775: - Port okascore.c optimization used in okapiiindex from Zope2 catalog implementation. This module is compiled conditionally, based on whether your environment has a working C compiler. - Don't use ``len(self._docweight)`` in okapiindex _search_wids method (obtaining the length of a BTree is very expensive at scale). Instead use self.documentCount(). Also a Zope2 port. Crap! this set of performance fixes was added *six years ago* to the Zope2 version from which zope.index was forked! Hasn't anybody been using this package? We're using this package. The text index I can only remember in a not-so-critical piece where performance currently is not an issue. Also, following checkins even today wouldn't have me remember that this code is shared. The code which now lives in 'zope.index' is a copy-and-rename- everything fork of what was in the Zope2 PluggableIndexes / ZCTextIndex products as of around Christmas, 2002. The fix I was replying to is *far* from the first one which didn't get ported from subsequent changes to the Zope2 indexes, which have obviously been in *heavy* production use. I propose that we consider merging the two implementations, along the lines of the following: 1. Review the change history of the two codebases since the original fork, porting relevant changes from the Z2 indexes to the corresponding Z3 implementations. 2. Review / update the tests for the Z3 indexes, with a goal of 100% coverage, as well as improving the quality of the tests. 3. Re-implement the Z2 indexes as thin wrappers around the Z3 versions, ensuring that all existing Z2 tests continue to pass. This is somewhat the same pattern as Andreas used to merge the ZPT implementations, I think. This effort could be the focus of a sprint team, if there was interest. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKL9sQ+gerLs4ltQ4RAksCAKDWRMu820PO+oRQNun0VC1LrFjvOACfU6pE uVtYzoliBjQgI5gark3ALTw= =+WAL -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Jim Fulton Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
