Re: [Zope-dev] Sorting for zope.index
Hi, Chris and other developers. I adapted repoze.catalog field index sorting code and tests for standard zope.index FieldIndex and looks like it works. :) Though I like raising the KeyError when trying to sort docids that are not indexed, I had to remove it from interface declaration, and change it to skipping values as in ZCatalog/repoze.catalog. The problem is that it's not trivial to efficiently implement that checking in some cases. For example in FieldIndex sort method, in the use_lazy branch, set intersections are used and I can't see a way to check if some of docids are not indexed besides storing some additional variable which sucks. :) I also adapted repoze.catalog's KeywordIndex fixes/optimizations to zope.index KeywordIndex and TopicIndex. There is now also a nadako-sorting branch of zope.app.catalog. It adds sorting/limiting/reversing features to the searchResults method of catalog. It also introduces KeywordIndex for catalog, because I added support for IIndexSearch to zope.index KeywordIndex. :) I request some review from developers and if there won't be any objections, I'll merge this code to trunks of zope.index and zope.app.catalog. Thanks in advance. -- WBR, Dan Korostelev ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Zope Tests: 4 OK, 2 Failed
Summary of messages to the zope-tests list. Period Sat Dec 27 12:00:00 2008 UTC to Sun Dec 28 12:00:00 2008 UTC. There were 6 messages: 6 from Zope Tests. Test failures - Subject: FAILED (failures=1) : Zope-trunk Python-2.4.5 : Linux From: Zope Tests Date: Sat Dec 27 20:38:05 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-December/010733.html Subject: FAILED (failures=1) : Zope-trunk Python-2.5.2 : Linux From: Zope Tests Date: Sat Dec 27 20:39:35 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-December/010734.html Tests passed OK --- Subject: OK : Zope-2.8 Python-2.3.7 : Linux From: Zope Tests Date: Sat Dec 27 20:32:03 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-December/010729.html Subject: OK : Zope-2.9 Python-2.4.5 : Linux From: Zope Tests Date: Sat Dec 27 20:33:34 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-December/010730.html Subject: OK : Zope-2.10 Python-2.4.5 : Linux From: Zope Tests Date: Sat Dec 27 20:35:04 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-December/010731.html Subject: OK : Zope-2.11 Python-2.4.5 : Linux From: Zope Tests Date: Sat Dec 27 20:36:34 EST 2008 URL: http://mail.zope.org/pipermail/zope-tests/2008-December/010732.html ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Sorting for zope.index
Dan Korostelev wrote: Hi, Chris and other developers. I adapted repoze.catalog field index sorting code and tests for standard zope.index FieldIndex and looks like it works. :) Though I like raising the KeyError when trying to sort docids that are not indexed, I had to remove it from interface declaration, and change it to skipping values as in ZCatalog/repoze.catalog. The problem is that it's not trivial to efficiently implement that checking in some cases. For example in FieldIndex sort method, in the use_lazy branch, set intersections are used and I can't see a way to check if some of docids are not indexed besides storing some additional variable which sucks. :) I also adapted repoze.catalog's KeywordIndex fixes/optimizations to zope.index KeywordIndex and TopicIndex. There is now also a nadako-sorting branch of zope.app.catalog. It adds sorting/limiting/reversing features to the searchResults method of catalog. It also introduces KeywordIndex for catalog, because I added support for IIndexSearch to zope.index KeywordIndex. :) I request some review from developers and if there won't be any objections, I'll merge this code to trunks of zope.index and zope.app.catalog. Looks good! A couple of things: - I'm tempted to ditch the case normalization feature of keyword indexes. This is actually the application's job; there's no guarantee that values that are passed by the app will be strings. I'm not sure that anyone has actually used keyword indexes, as their results could not until now be intersected with the results from any other index so there don't appear to be any backwards compatibility goals to service here (KeywordIndexes appeared to be abandonware until we paid attention to them). - Likewise, I don't think there are are any consumers of topicindexes, for the same reasons, so the read on write upgrade in FilteredSetBase should probably go. Users that have pickles around which are still IISets can do a migration. Thanks! - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Sorting for zope.index
2008/12/28 Chris McDonough chr...@plope.com: Dan Korostelev wrote: - I'm tempted to ditch the case normalization feature of keyword indexes. This is actually the application's job; there's no guarantee that values that are passed by the app will be strings. I'm not sure that anyone has actually used keyword indexes, as their results could not until now be intersected with the results from any other index so there don't appear to be any backwards compatibility goals to service here (KeywordIndexes appeared to be abandonware until we paid attention to them). I agree that the normalization feature of KeywordIndex can stay in the way sometimes, but I personally find it useful for simple cases. I'd made a base class that doesn't do any normalization and a subclass that does (it's the reverse currently). I doubt as well that there's someone who are using KeywordIndex in its current state, but I'm not sure that we can just refactor things so it will break backward compatibility even in this case. I'd like to hear more community thoughts on that. - Likewise, I don't think there are are any consumers of topicindexes, for the same reasons, so the read on write upgrade in FilteredSetBase should probably go. Users that have pickles around which are still IISets can do a migration. Here's the same question as above. I really don't like the write on read repickle that I did in FilteredSetBase for migrating to IFSet. We could write a paragraph explaining those changes and how to adapt current data/code for those people who uses current KeywordIndex/TopicIndex, if there are any of them. -- WBR, Dan Korostelev ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Sorting for zope.index
2008/12/28 Chris McDonough chr...@plope.com: IMO, the cognitive cost of maintaining the write-on-read and by-default-normalize code (imagine someone looking at this code three years from now) is pretty high. For example, it took me about two or three hours to figure out that the keyword index case normalization code was added by someone who didn't really understand what the kw index was actually used for; I thought I was just missing something. I'm not sure what I'd think if I came across the write-on-read code; it'll be there forever if we don't take it out now. Given that we *know* that the kw index and topic index filtered set can't have been used except for anything but very special-purpose things (they could have only been used outside the catalog), I think it'd be fine to tell the folks who used them to how do the one-time upgrade in CHANGES.txt. IMO, of course. I'm generally fine with that. So if there won't be objections until tommorow, I'll make that refactorings and merge that code to trunk. -- WBR, Dan Korostelev ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Sorting for zope.index
2008/12/28 Chris McDonough chr...@plope.com: BTW, one optimization for fieldindex sorting that I'm fooling around with now (based on profiling data from a real-world application) is to: - Keep around a list of docids in forward value-sorted order. - Keey around a list of docids in reverse value-sorted order. - Iterate over these lists within a generator when it's an ascending sort with a limit or when it's a reverse sort; yield back the docids we're passed in the order implied by the list. When I say list above, I actually mean some ZODB-friendly data structure that isn't just a plain Python list (that's actually what I'm fooling around with creating now). FTR, doing this would let us get the KeyError-when-docid-isnt-there behavior back, and it would make the reverse-sort case (where we use sorted(docids, key=rev_index.get, reverse=reverse) far less sucky. I'll let you know how this goes. I also thought about that. But I'm not sure if we should add it to FieldIndex. It could be a subclass like SortingFieldIndex or something. Because fieldindexes are used wide and zope.index can't depend on something like zope.app.generations for nice migrating. It might be a bit overload when sorting is not needed. Also, the KeyError thing is matter of interface. Even if we create a fieldindex that can efficiently check and raise KeyError, others may not, so I'd leave the interface as it is now. BTW, you might be interested in zc.blist package for the ZODB-friendly balanced lists structure. -- WBR, Dan Korostelev ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Sorting for zope.index
Dan Korostelev wrote: 2008/12/28 Chris McDonough chr...@plope.com: BTW, one optimization for fieldindex sorting that I'm fooling around with now (based on profiling data from a real-world application) is to: - Keep around a list of docids in forward value-sorted order. - Keey around a list of docids in reverse value-sorted order. - Iterate over these lists within a generator when it's an ascending sort with a limit or when it's a reverse sort; yield back the docids we're passed in the order implied by the list. When I say list above, I actually mean some ZODB-friendly data structure that isn't just a plain Python list (that's actually what I'm fooling around with creating now). FTR, doing this would let us get the KeyError-when-docid-isnt-there behavior back, and it would make the reverse-sort case (where we use sorted(docids, key=rev_index.get, reverse=reverse) far less sucky. I'll let you know how this goes. I also thought about that. But I'm not sure if we should add it to FieldIndex. It could be a subclass like SortingFieldIndex or something. Because fieldindexes are used wide and zope.index can't depend on something like zope.app.generations for nice migrating. It might be a bit overload when sorting is not needed. I agree. I'll add it to repoze.catalog for my customer and we'll let it simmer there a while. Also, the KeyError thing is matter of interface. Even if we create a fieldindex that can efficiently check and raise KeyError, others may not, so I'd leave the interface as it is now. Right. BTW, you might be interested in zc.blist package for the ZODB-friendly balanced lists structure. Thanks! I knew I remembered something like this this around somewhere... holy __setitem__ method Batman! ;-) - C ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Proposal: Eliminating Globals as an import façade
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have just checked in work on a separate branch which removes all use of 'Globals' as an indirection for imports within the Zope2 core, restoring the module to its original purpose, which was to hold shared data (e.g., the opened database, etc.). +1 Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
