[Zope-dev] ZCatalog and indexes cleanup
Hi! I did plan to work on a small catalog improvement, but after looking at the code I'd like to do some cleanup first: 1.) remove the deprecated TextIndex The deprecation warning says: 'Using TextIndex is deprecated (will be removed in Zope ' '2.12). Use ZCTextIndex instead.' 2.) remove CHANGES.txt, README.txt and version.txt from Products/ZCatalog These files seem to be obsolete. 3.) remove security declarations from ZCTextIndex and DateRangeIndex All the other indexes don't have security declarations. AFAICS there is no way to access indexes from untrusted code without having the 'Manage ZCatalogIndex Entries' permission. 4.) add 'indexSize' to IPluggableIndex and implement it where missing ZCatalog uses that method and most indexes implement it already. If there are no objections, I'll make these changes on the 2.12 branch and the trunk. Cheers, Yuppie ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and indexes cleanup
On 29.06.09 12:48, yuppie wrote: Hi! I did plan to work on a small catalog improvement, but after looking at the code I'd like to do some cleanup first: 1.) remove the deprecated TextIndex The deprecation warning says: 'Using TextIndex is deprecated (will be removed in Zope ' '2.12). Use ZCTextIndex instead.' +1 2.) remove CHANGES.txt, README.txt and version.txt from Products/ZCatalog These files seem to be obsolete. +1 3.) remove security declarations from ZCTextIndex and DateRangeIndex All the other indexes don't have security declarations. AFAICS there is no way to access indexes from untrusted code without having the 'Manage ZCatalogIndex Entries' permission. I think that all index implementation should have security assertions?! 4.) add 'indexSize' to IPluggableIndex and implement it where missing ZCatalog uses that method and most indexes implement it already. +1 Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and indexes cleanup
Hi Andreas! Andreas Jung wrote: On 29.06.09 12:48, yuppie wrote: 3.) remove security declarations from ZCTextIndex and DateRangeIndex All the other indexes don't have security declarations. AFAICS there is no way to access indexes from untrusted code without having the 'Manage ZCatalogIndex Entries' permission. I think that all index implementation should have security assertions?! Why? '_catalog.indexes' is protected by the underscore and using the 'Indexes' alias is protected by 'Manage ZCatalogIndex Entries'. Only additional security restrictions would have any effect. Or am I missing a security hole? Cheers, Yuppie ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and indexes cleanup
On 29.06.09 19:33, yuppie wrote:
Hi Andreas!
Andreas Jung wrote:
On 29.06.09 12:48, yuppie wrote:
3.) remove security declarations from ZCTextIndex and DateRangeIndex
All the other indexes don't have security declarations. AFAICS there is
no way to access indexes from untrusted code without having the 'Manage
ZCatalogIndex Entries' permission.
I think that all index implementation should have security assertions?!
Why?
'_catalog.indexes' is protected by the underscore and using the
'Indexes' alias is protected by 'Manage ZCatalogIndex Entries'. Only
additional security restrictions would have any effect.
Or am I missing a security hole?
Not sure. I created a catalog /catalog and an index 'my_index'.
Within a debug shell:
app.catalog.Indexes['my_index']
FieldIndex at my_index
app.unrestrictedTraverse('catalog/Indexes/my_index')
FieldIndex at /catalog//my_index
app.restrictedTraverse('catalog/Indexes/my_index')
Traceback (most recent call last):
File stdin, line 1, in ?
File
/Users/ajung/sandboxes/Zope-2.11/2.11/lib/python/OFS/Traversable.py,
line 301, in restrictedTraverse
return self.unrestrictedTraverse(path, default, restricted=True)
File
/Users/ajung/sandboxes/Zope-2.11/2.11/lib/python/OFS/Traversable.py,
line 236, in unrestrictedTraverse
next = guarded_getattr(obj, name)
AccessControl.unauthorized.Unauthorized: You are not allowed to access
'Indexes' in this context
h...
Andreas
begin:vcard
fn:Andreas Jung
n:Jung;Andreas
org:ZOPYX Ltd. Co. KG
adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany
email;internet:i...@zopyx.com
title:CEO
tel;work:+49-7071-793376
tel;fax:+49-7071-7936840
tel;home:+49-7071-793257
x-mozilla-html:FALSE
url:www.zopyx.com
version:2.1
end:vcard
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hedley Roos wrote: Since memcached is distributed only a single Zope client needs to perform that query and the result is available to all other Zope clients. This is where you'll get the big win: no need to load all the catalog-related objects into the zodb cache on all the clients which has the twin drawbacks of needing to be done and trashing your zodb cache... And the cache is persistent as long as memcached runs, so you can merrily restart Zope instances and have a warm cache. I didn't even realise this until Roche pointed it out to me. Coool :-) cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. Before (or while) we work on caching, can we try to improve the underlying indexes, and the way that applications use them? I'm pretty sure that there is a lot of room for improvement: - Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around just in case is a sure lose a write time, and may even be expensive at query time (depending on the query). - Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing. - I'm pretty sure that we haven't yet found the best data structure for hierarchy indexes (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies. - I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time. - Other special-purpose indexes (e.g., a recent items index) would be worth a look, especially for applications with large volumes of content. I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it? -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 27, 2008, at 13:08 , Roché Compaan wrote: On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote: - Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around just in case is a sure lose a write time, and may even be expensive at query time (depending on the query). - Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing. - I'm pretty sure that we haven't yet found the best data structure for hierarchy indexes (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies. - I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time. - Other special-purpose indexes (e.g., a recent items index) would be worth a look, especially for applications with large volumes of content. I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it? It's the typical trade-off. One course is expedient and fast for your use case now. The other requires more resources, but benefits everyone. Including those who don't want to depend on yet another package, like memcached, for performance. When it comes to integrating anything in Zope itself I'd choose the latter. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkkFssEACgkQRAx5nvEhZLITiQCgskifGYaixaj6lVLk85l6rz6E aQwAoI9PRcJHL8oZPatlHWADA0h6orCe =YLhP -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 27, 2008, at 13:08 , Roché Compaan wrote: On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote: - Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around just in case is a sure lose a write time, and may even be expensive at query time (depending on the query). - Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing. - I'm pretty sure that we haven't yet found the best data structure for hierarchy indexes (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies. - I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time. - Other special-purpose indexes (e.g., a recent items index) would be worth a look, especially for applications with large volumes of content. I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it? It's the typical trade-off. One course is expedient and fast for your use case now. The other requires more resources, but benefits everyone. Including those who don't want to depend on yet another package, like memcached, for performance. I'm not tied to memcached. We started out using module level caches like zope.cache.ram but that has obvious problems when using ZEO. When it comes to integrating anything in Zope itself I'd choose the latter. Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one). -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 27, 2008, at 13:32 , Roché Compaan wrote: On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote: When it comes to integrating anything in Zope itself I'd choose the latter. Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one). Right, it's perfectly valid to create such a third party package. The discussion just highlights a greater issue. Personally, I don't think it's good practice to focus on the expediency of working around a problem as opposed to tackling the problem directly. The Zope world is littered with add-ons that act as band-aids on real or perceived shortcomings in Zope itself. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkkFtvQACgkQRAx5nvEhZLLOrwCaA+X3iGaTDmyt3vP4q93OoTfx CNsAoJXppoHwI17ISetv4iAwoJeb+Phd =auan -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Mon, 2008-10-27 at 13:41 +0100, Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 27, 2008, at 13:32 , Roché Compaan wrote: On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote: When it comes to integrating anything in Zope itself I'd choose the latter. Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one). Right, it's perfectly valid to create such a third party package. The discussion just highlights a greater issue. Personally, I don't think it's good practice to focus on the expediency of working around a problem as opposed to tackling the problem directly. The Zope world is littered with add-ons that act as band-aids on real or perceived shortcomings in Zope itself. Improving the performance of indexes is really really hard. In this case I really don't think caching is a band-aid, it is a good solution. Even with optimised indexes, you will find that you need caching to get reasonable performance if you have a catalog with close to a million or more documents indexed. Given a large enough catalog, I would argue that caching is equally as necessary as having a large cache for a ZEO client. But caches expire and results get invalidated, and therefor we should continue to optimise indexes. With some help we should be able to contribute at this level too. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Mon, Oct 27, 2008 at 12:33 PM, Andreas Jung [EMAIL PROTECTED] wrote: On 27.10.2008 16:28 Uhr, Rudá Porto Filgueiras wrote: I will sugest a package called zope.memcached (like zope.sqlalchemy does for SQLAlchemy integration). That way any application who need to talk memcached can do it with out loose atomicit. I don't see a particular reason for creating a new package for here. Extend lovely.memcached and your done. There is not much need for scattering tiny functionalies into two modules here. The module world is already complicated enough. If lovely,memcached alredy is safe when some Exception is raised, discard my sugestion. It's also compatible with zope2? Andreas -- = Rudá Porto Filgueiras Weimar Consultoria http://python-blog.blogspot.com Hospedagem Plone, Django, Zope 3, Grok... http://www.pytown.com = ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 27.10.2008 17:18 Uhr, Rudá Porto Filgueiras wrote: On Mon, Oct 27, 2008 at 12:33 PM, Andreas Jung[EMAIL PROTECTED] wrote: On 27.10.2008 16:28 Uhr, Rudá Porto Filgueiras wrote: I will sugest a package called zope.memcached (like zope.sqlalchemy does for SQLAlchemy integration). That way any application who need to talk memcached can do it with out loose atomicit. I don't see a particular reason for creating a new package for here. Extend lovely.memcached and your done. There is not much need for scattering tiny functionalies into two modules here. The module world is already complicated enough. If lovely,memcached alredy is safe when some Exception is raised, discard my sugestion. It's also compatible with zope2? We are using it together with our cache tool I mentioned earlier with Zope 2.8.1. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 26.10.2008 18:43 Uhr, Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. Path indexes and fulltext indexes have a much more complicated implementation compared to field or keyword indexes. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hi Roché, I can see your funkload profile? On Sun, Oct 26, 2008 at 3:43 PM, Roché Compaan [EMAIL PROTECTED]wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Fábio Rizzo Matos ThreePointsWeb [EMAIL PROTECTED] http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto! ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hi Fabio The funkload tests were project specific. I plan to write up my findings and to do benchmarks on a standard Plone instance and blog about it. This will unfortunately have to wait since I'm on holiday this week :-) -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za On Sun, 2008-10-26 at 15:54 -0200, Fabio Rizzo Matos wrote: Hi Roché, I can see your funkload profile? On Sun, Oct 26, 2008 at 3:43 PM, Roché Compaan [EMAIL PROTECTED] wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Fábio Rizzo Matos ThreePointsWeb [EMAIL PROTECTED] http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto! ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sun, 2008-10-26 at 18:50 +0100, Andreas Jung wrote: On 26.10.2008 18:43 Uhr, Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. Path indexes and fulltext indexes have a much more complicated implementation compared to field or keyword indexes. I know, and this alone makes a good argument for caching at catalog level. In our case we used membrane, which makes an excessive amount of catalog queries when looking up users so some level of caching was essential. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. Before (or while) we work on caching, can we try to improve the underlying indexes, and the way that applications use them? I'm pretty sure that there is a lot of room for improvement: - Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around just in case is a sure lose a write time, and may even be expensive at query time (depending on the query). - Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing. - I'm pretty sure that we haven't yet found the best data structure for hierarchy indexes (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies. - I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time. - Other special-purpose indexes (e.g., a recent items index) would be worth a look, especially for applications with large volumes of content. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJBLHb+gerLs4ltQ4RAp59AJwNlfjI0tBv4PdMiDdH4TLKSm5YfwCgu8xB F3u1G0onXKKZ4s7MbLj9B2w= =r0oE -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Very Nice. Have a nice holiday :-) On Sun, Oct 26, 2008 at 3:58 PM, Roché Compaan [EMAIL PROTECTED]wrote: Hi Fabio The funkload tests were project specific. I plan to write up my findings and to do benchmarks on a standard Plone instance and blog about it. This will unfortunately have to wait since I'm on holiday this week :-) -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za On Sun, 2008-10-26 at 15:54 -0200, Fabio Rizzo Matos wrote: Hi Roché, I can see your funkload profile? On Sun, Oct 26, 2008 at 3:43 PM, Roché Compaan [EMAIL PROTECTED] wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Fábio Rizzo Matos ThreePointsWeb [EMAIL PROTECTED] http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto! ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Fábio Rizzo Matos ThreePointsWeb [EMAIL PROTECTED] http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto! ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 26.10.2008 19:05 Uhr, Roché Compaan wrote: On Sun, 2008-10-26 at 18:50 +0100, Andreas Jung wrote: On 26.10.2008 18:43 Uhr, Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. Path indexes and fulltext indexes have a much more complicated implementation compared to field or keyword indexes. I know, and this alone makes a good argument for caching at catalog level. In our case we used membrane, which makes an excessive amount of catalog queries when looking up users so some level of caching was essential. First caching is good thing :-) But how about the following issue: CMF/Plone inject additional subqueries for expires/effective/typesAndRoles. At least the security related aubqueries make a cached catalog result very specific to a particular user. That seems to be very ok for a site with lots of anonymous users - it might be an issue with lots of authenticated users. It might be necessary to add some kind of intelligence to decide what to cache and what not. I don't think it does not make sense to cache the result of a fulltext search. I am just thinking if it would make sense to cache on the index level instead of catalog level? So you could for example cache expensive index queries (path index) and combine them with uncached index which are supposed to be fast..however such decisions require detailed mesurements on real systems. One other thing concerning memcached: there is obviously a limit to 1MB for data you can store as a value. We have not found an obvious way for increasing this limit other by patching the memcached sources. We came up with an implementation where data 1MB is split up into individual junks (we have a dedicated set_huge(), get_huge()) implementation. Andreas -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: [EMAIL PROTECTED] - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK E-Publishing, Python, Zope Plone development, Consulting begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most. If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that. It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly. Run a few tests and let me know please. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 25.10.2008 8:48 Uhr, Hedley Roos wrote: The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most. If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that. It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly. Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. As for standard queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now. Since memcached is distributed only a single Zope client needs to perform that query and the result is available to all other Zope clients. And the cache is persistent as long as memcached runs, so you can merrily restart Zope instances and have a warm cache. I didn't even realise this until Roche pointed it out to me. To answer the question: I believe catalogcache will win every time since the return time of a cached query is not dependent on the complexity of the query. We should get a few benchmarks running at query level. I'll have a bit of time next week. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hedley Roos wrote: As for standard queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now. sortable_title is a field index and shouldn't be slower than any other index. This all sounds very cool, by the way. :) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hi, On Fri, 2008-10-24 at 15:41 +0200, Hedley Roos wrote: The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions. I'd love if this wouldn't be a monkey patch. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Method search just inspects queries and stores results to memcache. Can you give me an example where the cache would deliver non-committed data? ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
In addition, you need to include a serial in your cache keys to avoid dirty reads. The cache invalidation code actively removes items from the cache. Am I understanding you correctly? H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 25.10.2008 14:53 Uhr, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Using a DataManager is likely the better and more safe choice. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, 2008-10-25 at 14:53 +0200, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. This is probably harmless but will cause unnecessary cache flushes for other clients. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Right. This is the same as clear. Method search just inspects queries and stores results to memcache. That's the issue. If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed. Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, Oct 25, 2008 at 2:57 PM, Andreas Jung [EMAIL PROTECTED] wrote: On 25.10.2008 14:53 Uhr, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Using a DataManager is likely the better and more safe choice. Andreas Thanks Andreas. I'll have a look at your code when available. Christian, I do have a mistake in my reasoning. If an object is added to the catalog in a transaction and I cache that object as result of a query in that same transaction, and then the transaction fails I'll have a bad cache. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed. Kind of like how I came to the same conclusion in parallel to you and stuffed up this thread :) Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. So this is the area I have to focus on right now. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. A possible solution is to keep a module level cache which can be committed to the memcache on transaction boundaries. That way I'll incur no performance penalty. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, 2008-10-25 at 14:55 +0200, Hedley Roos wrote: In addition, you need to include a serial in your cache keys to avoid dirty reads. The cache invalidation code actively removes items from the cache. Am I understanding you correctly? I wasn't even talking about invalidation as your cache wouldn't see 'invalidations' anyways. It's memcached's task to forget stuff: it's a cache anyway. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog caching with memcached
Hi all The past few weeks I've been optimizing a busy Plone site and so collective.catalogcache was born. It uses memcached as a distributed ZCatalog cache. It is currently in production and seems to be holding just fine. The site went from being unusable to serving quite a bit of data. I'll modify it in future to not be too tied to just memcached, but that's still some way off. BTW, if you do not have memcached available the catalog functions as usual. It's only been tested for Zope 2.9.6 - 2.10.6. The repo is at http://dev.plone.org/collective/browser/collective.catalogcache. The URL may contain the word Plone but it's all Zope :) The readme contains more info and instructions. The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog sorting issue
--On 10. Juli 2008 15:06:53 +0200 Martijn Jacobs [EMAIL PROTECTED] wrote: Wat denk je hiervan? Hello. In zope 2.10.5 (and probably 2.10.6 and 2.11 and, as I've read, all releases above 2.7) we've encountered a sorting bug in a dtml-in call when querying the catalog. I don't think it's dtml only related, but I'm not sure about that. It is the same bug as found on : https://bugs.launchpad.net/zope2/+bug/143504 seems to correct the problem though and I was wondering why this isn't added in the zope core. Does it break other stuff or should the problem fixed somewhere else? Somebody has some thoughts? Here is the deal: you provide a unittest for the patch and I commit the patch + tests. Andreas pgpjOA2XtXB7c.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog sorting issue
Wat denk je hiervan? Hello. In zope 2.10.5 (and probably 2.10.6 and 2.11 and, as I've read, all releases above 2.7) we've encountered a sorting bug in a dtml-in call when querying the catalog. I don't think it's dtml only related, but I'm not sure about that. It is the same bug as found on : https://bugs.launchpad.net/zope2/+bug/143504 Adding def __cmp__(self, other): return 0 to Products.ZCatalog.CatalogBrains.AbstractCatalogBrain seems to correct the problem though and I was wondering why this isn't added in the zope core. Does it break other stuff or should the problem fixed somewhere else? Somebody has some thoughts? Regards, Martijn -- Martijn Jacobs Four Digits, Internet Solutions a: Willemsplein 15-1 6811 KB Arnhem NL kvk: 09162137 | btw: 8161.22.234.B01 e-mail: [EMAIL PROTECTED] | web: http://www.fourdigits.nl tel: +31 (0)26 44 22 700 | fax: +31 (0)84 22 06 117 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog: updateMetadata and comparing string and unicode
Maurits van Rees wrote at 2008-3-5 23:57 +: ... I have an item in the portal_catalog of my Plone site that has some string as description. The real object meanwhile has had a code change so the description field now returns unicode. When I now recatalog that object it throws an error: Module Products.ZCatalog.Catalog, line 359, in catalogObject Module Products.ZCatalog.Catalog, line 318, in updateMetadata UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 159: ordinal not in range(128) /home/maurits/buildout/projectdeploy/parts/zope2/lib/python/Products/ZCatalog/Catalog.py(318)updateMetadata() - if data.get(index, 0) != newDataRecord: You must not mix unicode and str as keys in the same index. If you do, errors as the above are very likely. You can try the following approaches: * if you know the encoding used by your str objects, you can set Python's default encoding to this encoding. Whenever unicode and str come together, the str is converted to unicode using this encoding (which hopefully is the correct one in all such cases). sys.setdefaultencoding is only available at startup. Thus, setting defaultencoding must happen in a sitecustomize or site module. * You completely switch to unicode for the given index and convert the BTrees used be the index. An index usually uses two BTrees: the so called forward index (usually called _index) (it maps the index terms to sets of record ids indexed under this term) and the reverse index (usually called _unindex) (it maps record ids to the values corresponding to these objects). You need to convert the keys of the forward index and the values of the reverse index. For a FieldIndex, the value is the index term, for a KeywordIndex it it a sequence of index terms (all need be converted). The forward index can be converted as follows: self._index = OOBTree(((s.decode(your encoding), v) for (s,v) in self._index.items())) The reverse index uses an IOBTree and is similar to the above. But the details depend on index type. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog: updateMetadata and comparing string and unicode
Dieter Maurer wrote: sys.setdefaultencoding is only available at startup. Thus, setting defaultencoding must happen in a sitecustomize or site module. Or if you're sufficiently devious, it's available any time (not that actually using it is a good idea, but...): import sys sys.setdefaultencoding Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'module' object has no attribute 'setdefaultencoding' del sys.modules['sys'] import sys sys.setdefaultencoding built-in function setdefaultencoding -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog: updateMetadata and comparing string and unicode
Hi all,
I have an item in the portal_catalog of my Plone site that has some
string as description. The real object meanwhile has had a code
change so the description field now returns unicode. When I now
recatalog that object it throws an error:
Module Products.ZCatalog.Catalog, line 359, in catalogObject
Module Products.ZCatalog.Catalog, line 318, in updateMetadata
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 159:
ordinal not in range(128)
/home/maurits/buildout/projectdeploy/parts/zope2/lib/python/Products/ZCatalog/Catalog.py(318)updateMetadata()
- if data.get(index, 0) != newDataRecord:
This happens when the current data in the catalog get compared to the
new data. If there is a difference, the new data is stored. But to
compare the old string with the new unicode the string is converted to
unicode. This fails because the string has non ascii characters in
it. So basically what happens is this error:
unicode(ä, 'utf-8') == uä
True
ä == uä
Traceback (most recent call last):
File stdin, line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal
not in range(128)
Logical enough. This can be fixed in ZCatalog:
[EMAIL PROTECTED]:~/svn/Zope-210/lib/python/Products/ZCatalog $ svn diff
Index: Catalog.py
===
--- Catalog.py (revision 84388)
+++ Catalog.py (working copy)
@@ -304,7 +304,15 @@
# meta_data is stored as a tuple for efficiency
data[index] = newDataRecord
else:
-if data.get(index, 0) != newDataRecord:
+try:
+changed = data.get(index, 0) != newDataRecord
+except UnicodeDecodeError:
+# Converting some string to unicode fails. This
+# conversion happens when a string and a unicode need
+# to be compared. Those two are not the same, so
+# logically there has been a change, so:
+changed = True
+if changed:
data[index] = newDataRecord
return index
Index: tests/testCatalog.py
===
--- tests/testCatalog.py(revision 84388)
+++ tests/testCatalog.py(working copy)
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
##
#
# Copyright (c) 2002 Zope Corporation and Contributors. All Rights Reserved.
@@ -177,6 +177,13 @@
def __nonzero__(self):
self.fail(__nonzero__() was called)
+class zdummyText(ExtensionClass.Base):
+def __init__(self, text):
+self.text = text
+
+def title(self):
+return self.text
+
class FakeTraversalError(KeyError):
fake traversal exception for testing
@@ -261,6 +268,12 @@
data = self._catalog.getMetadataForUID('1')
self.assertEqual(data['title'], '1')
+text = zdummyText('A string with an accent: \xc3\xa4.')
+self._catalog.catalog_object(text, '1')
+text.text = unicode(A simple unicode.)
+self._catalog.catalog_object(text, '1')
+
+
def testReindexIndexDoesntDoMetadata(self):
self.d['0'].num =
self._catalog.reindexIndex('title', {})
===
With that change it works: on the live site I can edit and save that
item without errors.
Without the change to the code, the added test fails at precisely the
point where the change should be done. But if I change the code the
test still fails because something similar goes wrong in the
KeywordIndex, with this traceback:
===
Error in test testUpdateMetadata
(Products.ZCatalog.tests.testCatalog.TestZCatalog)
Traceback (most recent call last):
File unittest.py, line 260, in run
testMethod()
File
/home/maurits/svn/Zope-210/lib/python/Products/ZCatalog/tests/testCatalog.py,
line 274, in testUpdateMetadata
self._catalog.catalog_object(text, '1')
File /home/maurits/svn/Zope-210/lib/python/Products/ZCatalog/ZCatalog.py,
line 536, in catalog_object
update_metadata=update_metadata)
File /home/maurits/svn/Zope-210/lib/python/Products/ZCatalog/Catalog.py,
line 368, in catalogObject
blah = x.index_object(index, object, threshold)
File
/home/maurits/svn/Zope-210/lib/python/Products/PluginIndexes/common/UnIndex.py,
line 235, in index_object
res += self._index_object(documentId, obj, threshold, attr)
File
/home/maurits/svn/Zope-210/lib/python/Products/PluginIndexes/KeywordIndex/KeywordIndex.py,
line 85, in _index_object
fdiff = difference(oldKeywords, newKeywords)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25:
ordinal not in range(128)
===
This is a bit trickier to fix,
[Zope] Re: ZCatalog indexes and unit testing
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Eric Bréhault wrote:
Hello,
I have created a ZCatalog object where I declare the following index:
self.addIndex('featureType','FieldIndex')
which works fine in my application
But when I run it in my unit tests, I get the following error:
self.addIndex('featureType','FieldIndex')
File /opt/Plone3/lib/python/Products/ZCatalog/ZCatalog.py, line 971,
in
addIndex
raise ValueError, Index of type %s not found % type
ValueError: Index of type FieldIndex not found
I am using the unitest package, I have declare a lot of stuff in
configurationSetUp, but something is probably missing:
(is there something to import from PluginIndexes ?)
import unittest
from zope.component.testing import setUp, tearDown
from zope.configuration.xmlconfig import XMLConfig
from zope.testing import doctest
from zope.testing.doctestunit import DocFileSuite
def configurationSetUp(self):
setUp()
import Products.zgeo
import zope.component
import zope.annotation
import zope.app.publisher.browser
import Products.Five
import Products.Archetypes
import Products.CMFCore
import Products.GenericSetup
XMLConfig('meta.zcml', zope.component)()
XMLConfig('meta.zcml', zope.app.publisher.browser)()
XMLConfig('meta.zcml', Products.Five)()
XMLConfig('meta.zcml', Products.GenericSetup)()
XMLConfig('meta.zcml', Products.CMFCore)()
XMLConfig('configure.zcml', zope.annotation)()
XMLConfig('configure.zcml', Products.Five)()
XMLConfig('configure.zcml', Products.GenericSetup)()
XMLConfig('configure.zcml', Products.Archetypes)()
XMLConfig('configure.zcml', Products.zgeo)()
You need to get the 'PluginIndexes' product initialization triggered.
Look into ZopeTestCase's 'installProduct' helper function.
Tres.
- --
===
Tres Seaver +1 540-429-0999 [EMAIL PROTECTED]
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGqLxU+gerLs4ltQ4RAsF7AJ9HzKOCqIVefVxtKpViXsbK/AFS7QCg15IT
Nzc/HTM4C1ksfOV/03CIwJg=
=65tV
-END PGP SIGNATURE-
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Weird ZCatalog issue...
Chris Withers wrote: Chris Withers wrote: This ain't right, anyone else seen this or anything like it? zopectl debug: len(app.some_catalog.unrestrictedSearchResults()) 2937144 i = 0 for b in app.some_catalog.unrestrictedSearchResults(): i+=1 i 821165 Also: len(tuple(app.some_catalog.unrestrictedSearchResults()) 821165 Why is the lazy list of brains getting its length wrong? Hmm, I was hoping for more response... Oh well, filed in the collector: http://www.zope.org/Collectors/Zope/2226 cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Weird ZCatalog issue...
This ain't right, anyone else seen this or anything like it? zopectl debug: len(app.some_catalog.unrestrictedSearchResults()) 2937144 i = 0 for b in app.some_catalog.unrestrictedSearchResults(): i+=1 i 821165 wtf? (Zope 2.8) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Weird ZCatalog issue...
Chris Withers wrote: This ain't right, anyone else seen this or anything like it? zopectl debug: len(app.some_catalog.unrestrictedSearchResults()) 2937144 i = 0 for b in app.some_catalog.unrestrictedSearchResults(): i+=1 i 821165 Also: len(tuple(app.some_catalog.unrestrictedSearchResults()) 821165 Why is the lazy list of brains getting its length wrong? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope-dev] ZCatalog Key Error question
[EMAIL PROTECTED] wrote at 2006-9-3 18:06 -0400: Thank you so much for following-up with me, here. I think you're right about this having to do with the indexes on the ZCatalog. The catalog has: 1. Seven FieldIndex indexes. 2. A DateIndex index 3. Three ZCTextIndex indexes I have not seen reports that these indexes cause inconsistencies of the type you have described. A long time ago, there have been problems in the BTrees conflict resolution which caused catalog inconsistencies (of a different kind). They affected document lists of length one in the index. When the last remaining document was removed and a new one concurrently inserted, the insertion could be lost. But this was fixed a long time ago. Apart from bugs, inconsistencies are often caused by try: ... except: ... clauses. If the try clause modifies the persistent state and then is aborted by an exception, it is vital that the exception reaches the ZPublisher such that it aborts the transaction. If the except prevents this, then the ZPublisher will commit and you are likely to get inconsistencies. In older Zope versions (before Zope 2.8.1), such a dangerous try: ... except: ... has been in OFS.ObjectManager.ObjectManager.manage_beforeDelete. In more modern versions, the code has been slightly cleaned up but it still is error prone. Skim your log files whether you see manage_beforeDelete () threw ERROR log entries. You can get more hints towards the problem by examining the query that results in your exception: If this query just consists of a subquery to a single index, then you know the problematic index. If it combines several indexes (by an and), then all of these indexes would need to be inconsistent (which is possible but far less likely than a single index). In this case, one would expect that not the index but the catalog primary data got corrupted. In the latter case, the problem would need to happen during initial indexing or final unindexing. Studying the code may exclude some possibilities. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
[EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400: Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): ... - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 This does not look like a catalog corruption. Instead, it looks as if the catalog changed since the search was done. Are you caching your search results? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Dieter Maurer wrote at 2006-9-3 19:22 +0200: [EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400: Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): ... - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 This does not look like a catalog corruption. Instead, it looks as if the catalog changed since the search was done. Are you caching your search results? Sorry! I was wrong! It has nothing to do with a catalog modification after the search. Looks indeed as if an index reported a document which is no longer indexed. Looks like an index bug. Which (types of) indexes are involved in your failing queries? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Thank you so much for following-up with me, here. I think you're right
about this having to do with the indexes on the ZCatalog.
The catalog has:
1. Seven FieldIndex indexes.
2. A DateIndex index
3. Three ZCTextIndex indexes
I actually create these from python in a function that accepts the
catalog as a parameter during the Product Initialization process.
I have a function catalog_add that creates and initializes the a
ZCatalog when this product is created and initialized, so let me
actually post the sourcecode here for that function and the functions it
calls. It's fairly short. There are two functions of note
catalog_add and addIndexes. I'd actually be grateful if you would
take a look and let me know if you see anything. In particular, am I
using an index that you know to have problems, or that I'm using
improperly.
class Newsdesk(Folder,Persistent,Implicit):
meta_type = Newsdesk
# ...
def catalog_add(self):
setup the catalog
self.manage_addProduct[ZCatalog].manage_addZCatalog(Catalog,
Default Catalog)
catalog = self._getOb('Catalog')
self.addLexicon(catalog)
self.addIndexes(catalog)
self.addMetas(catalog)
catalog = self._getOb('Catalog')
return catalog
def addIndexes(self, cat):
cat.addIndex('date_found', 'FieldIndex', 'date_found')
cat.addIndex('deleted_p', 'FieldIndex', 'deleted_p')
cat.addIndex('uri', 'FieldIndex', 'uri')
cat.addIndex('categories_field', 'FieldIndex',
'categories')
cat.addIndex('categorized_p', 'FieldIndex',
'categorized_p')
cat.addIndex('approved_p', 'FieldIndex', 'approved_p')
cat.addIndex('search_id', 'FieldIndex', 'search_id')
extras = Empty()
extras.doc_attr = 'title'
extras.index_type = 'Okapi BM25 Rank'
extras.lexicon_id = 'Lexicon'
cat.addIndex('title', 'ZCTextIndex', extras)
extras2 = Empty()
extras2.doc_attr = 'source_name'
extras2.index_type = 'Okapi BM25 Rank'
extras2.lexicon_id = 'Lexicon'
cat.addIndex('source_name', 'ZCTextIndex', extras2)
extras3 = Empty()
extras3.doc_attr = 'categories'
extras3.index_type = 'Okapi BM25 Rank'
extras3.lexicon_id = 'Lexicon'
cat.addIndex('categories', 'ZCTextIndex', extras3)
cat.addIndex('bobobase_modification_time', 'DateIndex')
def addMetas(self, cat):
create the metadata columns
cat.manage_addColumn(categorized_p)
cat.manage_addColumn(deleted_p)
cat.manage_addColumn(search_id)
cat.manage_addColumn(categories)
cat.manage_addColumn(date_found)
cat.manage_addColumn(source_name)
cat.manage_addColumn(title)
cat.manage_addColumn(uri)
def addLexicon(self, cat):
elem = []
wordSplitter = Empty()
wordSplitter.group = 'Word Splitter'
wordSplitter.name = 'HTML aware splitter'
caseNormalizer = Empty()
caseNormalizer.group = 'Case Normalizer'
caseNormalizer.name = 'Case Normalizer'
stopWords = Empty()
stopWords.group = 'Stop Words'
stopWords.name = 'Remove listed and single char words'
elem.append(wordSplitter)
elem.append(caseNormalizer)
elem.append(stopWords)
cat.manage_addProduct['ZCTextIndex'].manage_addLexicon('Lexicon',
'Default Lexicon', elem)
On Sun, 3 Sep 2006 20:11:33 +0200, Dieter Maurer [EMAIL PROTECTED]
said:
Dieter Maurer wrote at 2006-9-3 19:22 +0200:
[EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400:
Traceback for my Key Error (Dieter, thank you for alerting me on where
to find this):
Traceback (innermost last):
...
- Physical Path: /nephron_links/news_dtml
Module DocumentTemplate.DT_String, line 476, in __call__
Module DocumentTemplate.DT_In, line 465, in renderwb
Module DocumentTemplate.DT_In, line 747, in sort_sequence
Module Products.ZCatalog.Lazy, line 158, in __getitem__
Module Products.ZCatalog.Catalog, line 520, in getScoredResult
KeyError: -464571725
This does not look like a catalog corruption.
Instead, it looks as if the catalog changed since the search was
done. Are you caching your search results?
Sorry! I was wrong!
It has nothing to do with a catalog modification after the search.
Looks indeed as if an index reported a document which is no longer
indexed. Looks like an index bug. Which
[Zope-dev] ZCatalog Key Error question
I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Here are the clues: 1. When I re-index the catalog, everything works fine. Unfortunately, I cannot re-index the catalog very frequently because it takes a long time 2. The objects that are indexed are Citation objects, a class I defined. 3. The Citation class extends CatalogAware (as well as Item,Persistent,Implicit, PropertyManager) 4. Every time I change a citation, I call reindex_object() (on the citation) 5. When I add a citation, I call reindex_object() on the citation immediately. 6. I'm using Zope 2.8.4-final, built from source. 7. I'm using python 2.3.5 Thank you in advance. I can provide more information upon request. Best, Brian Rosenthal Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
--On 1. September 2006 02:54:26 -0400 [EMAIL PROTECTED] wrote: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Read or Write conflict errors? Are the objects modified at the same time by other threads? -aj pgpRrhqcwgEtE.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Read or Write conflict errors? I'm getting the Key Error traceback on Reads, but my guess is that the actual error happens sometime before the Key Error, when the ZCatalog itself is corrupted. I am not sure what that is, because it does not generate an error. However, I see the error when I call ZCatalogInstance.Search(...), and only when the Search includes objects that are corrupted in the ZCatalog. Are the objects modified at the same time by other threads? I'm not sure about this. I'm running ZEO, and in general, I think I can cause the error when I am the only user on the system, only issuing one request. However, I'm getting another ConflictError when I try to set an attribute value of a ZODB object during the execution of a script, and I'm told that may be because there is a problem accessing the same object from multiple threads. Is there a good article that you know of that I could find out more about thread-safe programming in ZEO? Best, Brian On Fri, 01 Sep 2006 09:27:41 +0200, Andreas Jung [EMAIL PROTECTED] said: --On 1. September 2006 02:54:26 -0400 [EMAIL PROTECTED] wrote: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Read or Write conflict errors? Are the objects modified at the same time by other threads? -aj Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
[EMAIL PROTECTED] wrote at 2006-9-1 02:54 -0400: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Here are the clues: You forget to provide us with an essential glue: The complete error information (error type, error value and traceback). You find it in your error_log object (in Zope's Root Folder). -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): Module ZPublisher.Publish, line 113, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 40, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.PythonScripts.PythonScript, line 323, in _exec Module None, line 11, in to_javascript - PythonScript at /nephron_links/to_javascript used for /nephron_links/news_dtml - Line 11 Module OFS.DTMLDocument, line 128, in __call__ - DTMLDocument at /nephron_links/news_dtml - URL: http://nephron.org/news_dtml/manage_main - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 On Fri, 1 Sep 2006 19:11:12 +0200, Dieter Maurer [EMAIL PROTECTED] said: [EMAIL PROTECTED] wrote at 2006-9-1 02:54 -0400: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Here are the clues: You forget to provide us with an essential glue: The complete error information (error type, error value and traceback). You find it in your error_log object (in Zope's Root Folder). -- Dieter Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope] Pausing ZCatalog and re-indexing later
Hi :-) The system I am working on is using a ZCatalog with some indexes (over 10) for faster searching. Unfortunatelly there are some use cases, where I have to add really many new objects to ZODB. It works, but very slowly. I suspect that this is because when I add an object to Folder (with _setObject) it gets indexed - couple thousands of objects multiplied by over 10 indexes = long. Am I guessing right or am I completely lost? ;-) If this is the probable reason - is there some way I could disable indexing temporarily and after adding all of the objects make it re-index the new ones? I think that even a full re-index would work faster than it works now... I would be very grateful for any suggestions. Best regards, Maciek ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Pausing ZCatalog and re-indexing later
--On 11. August 2006 17:10:29 + [EMAIL PROTECTED] wrote: Hi :-) The system I am working on is using a ZCatalog with some indexes (over 10) for faster searching. Unfortunatelly there are some use cases, where I have to add really many new objects to ZODB. It works, but very slowly. What kind of objects? I suspect that this is because when I add an object to Folder (with _setObject) it gets indexed - couple thousands of objects multiplied by over 10 indexes = long. Consider using CatalogQueue to defer indexing. Am I guessing right or am I completely lost? ;-) No idea, you might investigate your bottleneck(s) using ZopeProfiler. -aj pgpD2fQSIOfZn.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Pausing ZCatalog and re-indexing later
- Original Message - From: [EMAIL PROTECTED] To: zope@zope.org Sent: Friday, August 11, 2006 1:10 PM Subject: [Zope] Pausing ZCatalog and re-indexing later Hi :-) The system I am working on is using a ZCatalog with some indexes (over 10) for faster searching. Unfortunatelly there are some use cases, where I have to add really many new objects to ZODB. It works, but very slowly. I suspect that this is because when I add an object to Folder (with _setObject) it gets indexed - couple thousands of objects multiplied by over 10 indexes = long. Am I guessing right or am I completely lost? ;-) If this is the probable reason - is there some way I could disable indexing temporarily and after adding all of the objects make it re-index the new ones? I think that even a full re-index would work faster than it works now... I would be very grateful for any suggestions. Another approach is to build/update a new ZCatalog in another .fs file (you can do this offline, on another box, etc), and then just swap in the new .fs file. You can also create a mount point for a folder that contains nothing but the ZCatalog and then just swap in that (instead of swapping the entire Data.fs file). hth Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope-Checkins] SVN: Zope/trunk/ - ZCatalog: removed manage_deleteIndex(), manage_delColumns()
Log message for revision 67631: - ZCatalog: removed manage_deleteIndex(), manage_delColumns() which were deprecated since Zope 2.4 Changed: U Zope/trunk/doc/CHANGES.txt U Zope/trunk/lib/python/Products/ZCatalog/ZCatalog.py -=- Modified: Zope/trunk/doc/CHANGES.txt === --- Zope/trunk/doc/CHANGES.txt 2006-04-27 00:54:03 UTC (rev 67630) +++ Zope/trunk/doc/CHANGES.txt 2006-04-27 07:55:45 UTC (rev 67631) @@ -18,6 +18,9 @@ Restructuring + - ZCatalog: removed manage_deleteIndex(), manage_delColumns() +which were deprecated since Zope 2.4 + - deprecated the zLOG module. Use Pythons 'logging' module instead. - replaced all zLOG occurences (expect the zLOG module itself) with Modified: Zope/trunk/lib/python/Products/ZCatalog/ZCatalog.py === --- Zope/trunk/lib/python/Products/ZCatalog/ZCatalog.py 2006-04-27 00:54:03 UTC (rev 67630) +++ Zope/trunk/lib/python/Products/ZCatalog/ZCatalog.py 2006-04-27 07:55:45 UTC (rev 67631) @@ -372,25 +372,6 @@ '/manage_catalogSchema?manage_tabs_message=Column%20Added') -security.declareProtected(manage_zcatalog_entries, 'manage_delColumns') -def manage_delColumns(self, names, REQUEST=None, RESPONSE=None, URL1=None): - Deprecated method. Use manage_delColumn instead. -# log a deprecation warning -import warnings -warnings.warn( -The manage_delColumns method of ZCatalog is deprecated -since Zope 2.4.2.\n -This method is only kept for backwards compatibility -for a while\n -and will go away in a future release.\n -\n -Please use instead the manage_delColumn method.\n -,DeprecationWarning) - -self.manage_delColumn(names, REQUEST=REQUEST, RESPONSE=RESPONSE, - URL1=URL1) - - security.declareProtected(manage_zcatalog_entries, 'manage_delColumn') def manage_delColumn(self, names, REQUEST=None, RESPONSE=None, URL1=None): delete a column or some columns @@ -418,26 +399,6 @@ '/manage_catalogIndexes?manage_tabs_message=Index%20Added') -security.declareProtected(manage_zcatalog_entries, 'manage_deleteIndex') -def manage_deleteIndex(self, ids=None, REQUEST=None, RESPONSE=None, -URL1=None): - Deprecated method. Use manage_delIndex instead. -# log a deprecation warning -import warnings -warnings.warn( -The manage_deleteIndex method of ZCatalog is deprecated -since Zope 2.4.2.\n -This method is only kept for backwards compatibility for a -while\n -and will go away in a future release.\n -\n -Please use instead the manage_delIndex method.\n -,DeprecationWarning) - -self.manage_delIndex(ids=ids, REQUEST=REQUEST, RESPONSE=RESPONSE, - URL1=URL1) - - security.declareProtected(manage_zcatalog_entries, 'manage_delIndex') def manage_delIndex(self, ids=None, REQUEST=None, RESPONSE=None, URL1=None): ___ Zope-Checkins maillist - Zope-Checkins@zope.org http://mail.zope.org/mailman/listinfo/zope-checkins
[Zope] Force ZCatalog to UTF-8
Hi all! How can I force ZCatalog to use only unicode strings (UTF-8)? I'm trying to change all zope to use these charset. For that I install Zope 2.9.1 and I change on zope.conf: rest-input-encoding utf-8 rest-output-encoding utf-8 default-zpublisher-encoding utf-8 but ZCatalog don't catalog unicode if the string is not an unicode string (at least I think so) Any idea? Thanks a lot! -- Mis Cosas http://blogs.sistes.net/Garito/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Force ZCatalog to UTF-8
The ZCatalog has little in mind with encodings. It indexes the stuff that an object offers to be indexed. You might look at TXNG 3 which deals usually much nicer with encoding, unicode etc. than ZCTextIndex. -aj --On 10. März 2006 13:57:30 +0100 Garito [EMAIL PROTECTED] wrote: Hi all! How can I force ZCatalog to use only unicode strings (UTF-8)? I'm trying to change all zope to use these charset. For that I install Zope 2.9.1 and I change on zope.conf: rest-input-encoding utf-8 rest-output-encoding utf-8 default-zpublisher-encoding utf-8 but ZCatalog don't catalog unicode if the string is not an unicode string (at least I think so) Any idea? Thanks a lot! -- Mis Cosas http://blogs.sistes.net/Garito/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) pgpZH0EjaCnDl.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Force ZCatalog to UTF-8
Andreas Jung wrote at 2006-3-10 14:12 +0100: The ZCatalog has little in mind with encodings. It indexes the stuff that an object offers to be indexed. You might look at TXNG 3 which deals usually much nicer with encoding, unicode etc. than ZCTextIndex. Or ManagableIndex for non full text indexes. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] bad ZCatalog behavior
On Wednesday 10 August 2005 04:32 pm, Dieter Maurer wrote: Andy Altepeter wrote at 2005-8-8 08:15 -0500: There seems to be a problem with my ZCatalog, and it seems to be getting worse as time goes by. ... Module ZODB.Connection, line 562, in setstate Module ZODB.Connection, line 601, in _set_ghost_state AttributeError: 'str' object has no attribute 'append' In fact, this looks like a broken pickle in your ZODB storage (and not like a ZCatalog problem). I think I've fixed this issue, but for future reference, is there any way to remove a broken pickle? ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] bad ZCatalog behavior
Andy Altepeter wrote at 2005-8-11 07:22 -0500: ... In fact, this looks like a broken pickle in your ZODB storage (and not like a ZCatalog problem). I think I've fixed this issue, but for future reference, is there any way to remove a broken pickle? No, because there should be no broken pickle in the first place... ...and because the problem did not yet occured often enough... -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] bad ZCatalog behavior
Andy Altepeter wrote at 2005-8-8 08:15 -0500: There seems to be a problem with my ZCatalog, and it seems to be getting worse as time goes by. ... Module ZODB.Connection, line 562, in setstate Module ZODB.Connection, line 601, in _set_ghost_state AttributeError: 'str' object has no attribute 'append' In fact, this looks like a broken pickle in your ZODB storage (and not like a ZCatalog problem). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] bad ZCatalog behavior
Hi there, There seems to be a problem with my ZCatalog, and it seems to be getting worse as time goes by. When my CatalogPathAware objects unindex themselves, sometimes I get an attribute error like: AttributeError: 'str' object has no attribute 'append' Traceback (innermost last): Module ZPublisher.Publish, line 101, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Products.FileSystemSite.FSPythonScript, line 104, in __call__ Module Shared.DC.Scripts.Bindings, line 306, in __call__ Module Shared.DC.Scripts.Bindings, line 343, in _bindAndExec Module Products.FileSystemSite.FSPythonScript, line 160, in _exec Module None, line 33, in tab_status_approve - FSPythonScript at /silva/service_views/Silva/edit/VersionedContent/tab_status_approve used for /silva/service_views/SilvaDocument/edit/VersionedContent/Document - Line 33 Module Products.Silva.Versioning, line 125, in approve_version Module Products.Silva.Versioning, line 512, in _update_publication_status Module Products.Silva.VersionedContent, line 334, in _unindex_version Module Products.Silva.Version, line 191, in unindex_object Module Products.ZCatalog.ZCatalog, line 558, in uncatalog_object Module Products.ZCatalog.Catalog, line 411, in uncatalogObject Module Products.ZCTextIndex.ZCTextIndex, line 181, in unindex_object Module Products.ZCTextIndex.OkapiIndex, line 68, in unindex_doc Module Products.ZCTextIndex.BaseIndex, line 176, in unindex_doc Module Products.ZCTextIndex.BaseIndex, line 303, in _del_wordinfo Module ZODB.Connection, line 562, in setstate Module ZODB.Connection, line 601, in _set_ghost_state AttributeError: 'str' object has no attribute 'append' In the logfile, the last line of code is shown in the traceback: File /export1/Zope-2.7.2-0/lib/python/ZODB/Connection.py, line 601, in _set_ghost_state state = unpickler.load() Perhaps this is a ZEO issue. I'm using Zope-2.7.2-0 with ZEO. Both the zeo server and the client are on the same box. The zeo logfile reports no errors when this occurs. Any ideas why this may be happening? ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope-dev] ZCatalog getObject broken
Guys, Dieter Maurer [EMAIL PROTECTED] wrote: Roché Compaan wrote at 2005-2-25 17:22 +0200: Last year in March the following checkin was made that changed ZCatalog's getObject to use restrictedTraverse instead of unrestrictedTraverse. See: http://mail.zope.org/pipermail/zope-checkins/2004-March/026846.html In my opininion this is wrong, I agree with you! Me also. ... I would propose that getObject does an unrestrictedTraverse of the path and then checks if the user has permission to access that the object. I argued precisely this approach with the person who made the change. I had the impression that I have convinced him -- but apparently, he did not change the code accordingly :-( Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope Roché has added http://www.zope.org/Collectors/Zope/1713 I intend to fix this before 2.7.5 final, probably today or tonight. I feel this is sufficiently important to warrant a fix now. I guess it'll mean an RC2. Please shout if you find problems with this approach. Florent -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
--On Donnerstag, 10. März 2005 12:49 Uhr +0100 Florent Guillaume [EMAIL PROTECTED] wrote: Guys, Dieter Maurer [EMAIL PROTECTED] wrote: Roché Compaan wrote at 2005-2-25 17:22 +0200: Last year in March the following checkin was made that changed ZCatalog's getObject to use restrictedTraverse instead of unrestrictedTraverse. See: http://mail.zope.org/pipermail/zope-checkins/2004-March/026846.html In my opininion this is wrong, I agree with you! Me also. ... I would propose that getObject does an unrestrictedTraverse of the path and then checks if the user has permission to access that the object. I argued precisely this approach with the person who made the change. I had the impression that I have convinced him -- but apparently, he did not change the code accordingly :-( Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope Roché has added http://www.zope.org/Collectors/Zope/1713 I intend to fix this before 2.7.5 final, probably today or tonight. I feel this is sufficiently important to warrant a fix now. I guess it'll mean an RC2. Please see my remark on this issue in the collector. Andreas pgphJkbk8eW1O.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
Roché Compaan wrote at 2005-3-3 22:36 +0200: On Thu, 2005-03-03 at 19:36 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2005-3-3 09:53 +0200: ... -return self.aq_parent.restrictedTraverse(self.getPath(), None) +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): I think this is not correct: validate needs at least a value parameter (this is the forth parameter). I thought this much but what value? And doesn't this make the implementation of restrictedTraverse suspect too? When code is calling getObject on a catalog brain we don't know what attribute or method of that object the calling code will access. Does it then make any sense at all to do security checks in getObject? IMO it doesn't. Value means the accessed value. In your case, this is obj. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
Roché Compaan wrote: +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): +return obj +else: +return None Urm, Roche, doesn't the above seek to do exactly what... return self.aq_parent.restrictedTraverse(self.getPath(), None) ...does? The problem is that an error should be raised, Unauthorized in my opinion, rather than None being returned. None should never be returned in place of a brain, although I'll soften that to say that if it does, it means something weird has happened (used to mean the object the catalog entry mapped to had gone away) I think: self.aq_parent.restrictedTraverse(self.getPath()) ...should be fine, no? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
On Thu, 2005-03-03 at 14:56 +, Chris Withers wrote: Roché Compaan wrote: +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): +return obj +else: +return None Urm, Roche, doesn't the above seek to do exactly what... return self.aq_parent.restrictedTraverse(self.getPath(), None) ...does? No it doesn't, restrictedTraverse fails along the way. If the path is /a/b and the user doesn't have access to /a/ restrictedTraverse will return None even though the user has access to /a/b/. In my code above we only do a security check on the object that the full path resolves to. The problem is that an error should be raised, Unauthorized in my opinion, rather than None being returned. I would be ok with raising Unauthorized but it is not backwards compatible. I suppose changing to 'unrestrictedTraverse' is also not backward compatible but the current 'getObject' seems to suggest that we do not want to raise an exception when the user does not have permission to access the object. Is there some use case for 'getObject' that we are missing here? None should never be returned in place of a brain, although I'll soften that to say that if it does, it means something weird has happened (used to mean the object the catalog entry mapped to had gone away) I agree. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
Roché Compaan wrote at 2005-3-3 09:53 +0200: ... -return self.aq_parent.restrictedTraverse(self.getPath(), None) +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): I think this is not correct: validate needs at least a value parameter (this is the forth parameter). There is a validateValue method (instead of validate) that does what you want. You find it in AccessControl/ImplPython.py. Drawback: it might disappear in Zope 2.8. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
On Thu, 2005-03-03 at 19:36 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2005-3-3 09:53 +0200: ... -return self.aq_parent.restrictedTraverse(self.getPath(), None) +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): I think this is not correct: validate needs at least a value parameter (this is the forth parameter). I thought this much but what value? And doesn't this make the implementation of restrictedTraverse suspect too? When code is calling getObject on a catalog brain we don't know what attribute or method of that object the calling code will access. Does it then make any sense at all to do security checks in getObject? IMO it doesn't. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
--On Mittwoch, 2. März 2005 9:56 Uhr + Chris Withers [EMAIL PROTECTED] wrote: Roché Compaan wrote: Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope Well, I posted just such an issue a few months back. I'm working offline so can't give you the exact number but have a search and you should find it. I seem to remember Andreas rejecting it without thinking, as is his way ;-) I would appreciate it you would help resolve outstanding issues instead of making such statement :-) Andreas pgpGFKGurRtBC.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
On Fri, 2005-02-25 at 21:06 +0100, Andreas Jung wrote: --On Freitag, 25. Februar 2005 20:21 Uhr +0100 Dieter Maurer [EMAIL PROTECTED] wrote: Roché Compaan wrote at 2005-2-25 17:22 +0200: Last year in March the following checkin was made that changed ZCatalog's getObject to use restrictedTraverse instead of unrestrictedTraverse. See: http://mail.zope.org/pipermail/zope-checkins/2004-March/026846.html In my opininion this is wrong, I agree with you! ... I would propose that getObject does an unrestrictedTraverse of the path and then checks if the user has permission to access that the object. I argued precisely this approach with the person who made the change. I had the impression that I have convinced him -- but apparently, he did not change the code accordingly :-( Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope Best to include a patch as well :-) -aj I'm unsure about the security check in the patch below - I copied the way restrictedTraverse does it. I read through validate in the default security policy but it is one of those methods where all the security implications doesn't fit in your head all at once. --- CatalogBrains.py~ 2004-03-23 22:27:23.0 +0200 +++ CatalogBrains.py2005-03-03 09:43:48.0 +0200 @@ -47,7 +47,11 @@ (i.e., it was deleted or moved without recataloging), or if the user is not authorized to access an object along the path. -return self.aq_parent.restrictedTraverse(self.getPath(), None) +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): +return obj +else: +return None def getRID(self): Return the record ID for this object. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog getObject broken
Last year in March the following checkin was made that changed ZCatalog's getObject to use restrictedTraverse instead of unrestrictedTraverse. See: http://mail.zope.org/pipermail/zope-checkins/2004-March/026846.html In my opininion this is wrong, and just cost me a good three hours to figure out why big parts of one of our apps broke after an upgrade to Zope 2.7.3. What's worse is that there is nowhere a trace of this change in HISTORY.txt or CHANGES.txt. Anybody with a site structure that has less restrictive access deeper into the site would agree that getObject should do an unrestrictedTraverse. restrictedTraverse returns None as soon as it traverses an object a user does not have access to, regardless if the user has access to the object referred to by the full path. To illustrate imagine the following: You have a site with a folder containg documents at '/documents'. Inside that folder you have a whole bunch of documents where users have a local role of owner to give them permission to access only their own documents. You use a Catalog query to get the list of documents belonging to a particular user and want to use getObject to retrieve the objects found. But, it won't work because restrictedTraverse already fails when traversing the 'documents' folder. I would propose that getObject does an unrestrictedTraverse of the path and then checks if the user has permission to access that the object. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
Roché Compaan wrote at 2005-2-25 17:22 +0200: Last year in March the following checkin was made that changed ZCatalog's getObject to use restrictedTraverse instead of unrestrictedTraverse. See: http://mail.zope.org/pipermail/zope-checkins/2004-March/026846.html In my opininion this is wrong, I agree with you! ... I would propose that getObject does an unrestrictedTraverse of the path and then checks if the user has permission to access that the object. I argued precisely this approach with the person who made the change. I had the impression that I have convinced him -- but apparently, he did not change the code accordingly :-( Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and Zope 2.8
--On Sonntag, 30. Januar 2005 19:17 Uhr +0100 Dieter Maurer [EMAIL PROTECTED] wrote: - some indexes show the number of indexed objects vs the number of indexes terms within the ZMI which is totally inconsistent. I think the ZMI should show the number of indexed objects. Index specific information e.g. the number of indexed terms should be shown within the indexes default view (if necessary). Objections? This was already discussed (-- mailing list archives). I (and others) find it more informative to get a feeling about the size of the index (rather than the number of indexed objects) in the overview. There are two points of view: the normal user is confused if some indexes show up the number of indexed objects and others show the size of the index within the same column. This should be consistent. If would prefer the number of indexed objects within the default ZMI view and put index specific size information into their own default view. - Indexes derived from UnIndex also store informations about objects although they do not index useful information. ... An optimised version of UnIndex would store only values evaluating to non-zero. It would essentially change the number of indexed objects (and make it a bit more informative when one is interested in the size of the index) ;-) right. You must be a bit careful with the non-zero. Some indexes interpret (some) zero values in a special way, e.g. DateRangeIndex. It interprets None as no limit. Yes, but DateRangeIndexes overwrite index_object() so that a change of UnIndex itself would not harm. I would also prefer when None would consistently means: I do not have a value (in the current context). But, this will interfere with some indexes. None could be a problem with other indexes...at least there should be a unique marker saying: I have nothing of interest to be indexed Andreas ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and Zope 2.8
Andreas Jung wrote at 2005-1-31 18:50 +0100: ... [AJ] - some indexes show the number of indexed objects vs the number of indexes terms within the ZMI which is totally inconsistent. I think the ZMI should show the number of indexed objects. Index specific information e.g. the number of indexed terms should be shown within the indexes default view (if necessary). Objections? [DM] This was already discussed (-- mailing list archives). I (and others) find it more informative to get a feeling about the size of the index (rather than the number of indexed objects) in the overview. [AJ] There are two points of view: the normal user is confused if some indexes show up the number of indexed objects and others show the size of the index within the same column. This should be consistent. If would prefer the number of indexed objects within the default ZMI view and put index specific size information into their own default view. [DM] All are with you (including myself) when you strive for consistency. The display should be consistent and correspond to the label in the table head. I am not with you with respect to number of indexed objects versus size of the index. In fact *BOTH* as index specific (otherwise, it would not make any sense to list it in an index specific column). Maybe, a compromize would be to include both numbers? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and Zope 2.8
--On Montag, 31. Januar 2005 20:20 Uhr +0100 Dieter Maurer [EMAIL PROTECTED] wrote: I am not with you with respect to number of indexed objects versus size of the index. In fact *BOTH* as index specific (otherwise, it would not make any sense to list it in an index specific column). Maybe, a compromize would be to include both numbers? I am currently working on a solution for all these issues on a dedicated 2.8 branch. Andreas ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog and Zope 2.8
I changed the ZCatalog and indexes codebase for Zope 2.8 in the following way: - ZCatalog and indexes derived from UnIndex used a nasty implementation of __len__ which lead to problems in Zope 2.8 with new-style classes. The corresponding code has been cleaned up. For ZCatalog instances there is an implicit migration built-in for the __len__ attribute. For all indexes I added a manage_convertIndexes() method that re-creates and reindexes all indexes of a given ZCatalog instance. Outstanding issues: - some indexes show the number of indexed objects vs the number of indexes terms within the ZMI which is totally inconsistent. I think the ZMI should show the number of indexed objects. Index specific information e.g. the number of indexed terms should be shown within the indexes default view (if necessary). Objections? - Indexes derived from UnIndex also store informations about objects although they do not index useful information. E..g. if you have one thousand objects and a keyword index indexing only two objects (because these two objects have the required property or method returning indexable values) than you have the RIDs of one thousand objects within the internal data structure of the indexa total waste of space. An optimised version of UnIndex would store only values evaluating to non-zero. Such an optimisation is already used in Dieter's Managablexes. This might change the behaviour of some applications but I am not completely sure about this issue. Thoughts? Andreas ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and Zope 2.8
Andreas Jung wrote at 2005-1-30 15:30 +0100: ... Outstanding issues: - some indexes show the number of indexed objects vs the number of indexes terms within the ZMI which is totally inconsistent. I think the ZMI should show the number of indexed objects. Index specific information e.g. the number of indexed terms should be shown within the indexes default view (if necessary). Objections? This was already discussed (-- mailing list archives). I (and others) find it more informative to get a feeling about the size of the index (rather than the number of indexed objects) in the overview. - Indexes derived from UnIndex also store informations about objects although they do not index useful information. ... An optimised version of UnIndex would store only values evaluating to non-zero. It would essentially change the number of indexed objects (and make it a bit more informative when one is interested in the size of the index) ;-) You must be a bit careful with the non-zero. Some indexes interpret (some) zero values in a special way, e.g. DateRangeIndex. It interprets None as no limit. However, I agree with you: at least when an object o does not define a value for index i, then i should not index o. I would also prefer when None would consistently means: I do not have a value (in the current context). But, this will interfere with some indexes. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog fast as admin, dog slow as anonymous and other users
Zopistas, My ZCatalog is fast as admin, but dog slow as anonymous and other users. Anyone had this same experience? Details: marjors: Zope 2.6.2 CMF 1.3 Plone 1.0.5 about 50,000 cataloged objects (dual xenon server, plenty of ram, RAID) User folder has 15k users in it, but admin is one of those users. i've factored it way down and a simple search with just 1 Event loaded, like: return context.portal_catalog(Type='Event') is instantaneous for the 'admin' user, but takes about 10 seconds to return 1 SINGLE object. (only 1 Event loaded) imagine the time it takes to search and display 1300 events (which is what's normally in there). ouch. i've rebuilt the indexes, and still no dice. Any help would be greatly appreciated. All my best, Jason Spisak ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog fast as admin, dog slow as anonymous and other users
Jason Spisak wrote: Zopistas, My ZCatalog is fast as admin, but dog slow as anonymous and other users. Anyone had this same experience? Details: This is certainly to do with queries which run against FieldIndexes only for anonymous users. This is something the CMF does (e.g. expires and effective dates only applied for anonymous). FieldIndexes are really bad for DateTimes. Delete your date-related FieldIndexes and replace them with DateIndexes. seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog fast as admin, dog slow as anonymous and other users
Seb, thanks! i'll give that a shot. i also thought of adding daterange indexes for the calendar portion or things. does that also garner any speed improvement? Seb Bacon wrote: Jason Spisak wrote: Zopistas, My ZCatalog is fast as admin, but dog slow as anonymous and other users. Anyone had this same experience? Details: This is certainly to do with queries which run against FieldIndexes only for anonymous users. This is something the CMF does (e.g. expires and effective dates only applied for anonymous). FieldIndexes are really bad for DateTimes. Delete your date-related FieldIndexes and replace them with DateIndexes. seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog fast as admin, dog slow as anonymous and other users
Seb, I've replaced the following date indexes that werer FieldIndexes with DateIndexes Date effective expires modified i'll let you know what kind of speed improvement I get. Jason Spisak wrote: Zopistas, My ZCatalog is fast as admin, but dog slow as anonymous and other users. Anyone had this same experience? Details: This is certainly to do with queries which run against FieldIndexes only for anonymous users. This is something the CMF does (e.g. expires and effective dates only applied for anonymous). FieldIndexes are really bad for DateTimes. Delete your date-related FieldIndexes and replace them with DateIndexes. seb Jason Spisak ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Dieter Maurer wrote: But overall, unless you have special (non DC derived) indexes, That can well be the case... Anyway, what are we going to do about this crawling tab? Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Chris Withers wrote at 2003-7-21 08:22 +0100: Dieter Maurer wrote: #objects suggests that it is the number of objects indexed by this index. Who is interested in this information? Well, it's been useful to be on several occasions when I've seen one index has less objects in than another... Unless one has inhomogeous objects, almost all objects are indexed by every index. Thus, #objects is likely to be similar for many indexes. Hmmm... I use ZCatalogs a _lot_ for searching over inhomogenous sets of objects. For example, that's it's primary role in the CMF... CMF's catalog is highly standardized, thanks to Dublin Core. All standard CMF content types define DC attributes. Therefore, each CMF content object is indexed under each DC field index. The Subjects index may lack some objects (because they do not define any Subjects). A text index may lack a few objects (because some objects may have have both an empty Title and an empty Descritpion). But overall, unless you have special (non DC derived) indexes, all #objects should be very similar. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Dieter Maurer wrote: #objects suggests that it is the number of objects indexed by this index. Who is interested in this information? Well, it's been useful to be on several occasions when I've seen one index has less objects in than another... Unless one has inhomogeous objects, almost all objects are indexed by every index. Thus, #objects is likely to be similar for many indexes. Hmmm... I use ZCatalogs a _lot_ for searching over inhomogenous sets of objects. For example, that's it's primary role in the CMF... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Casey Duncan wrote: Actually I regard the current behavior as a feature. Using a stopwatch and a slide-rule I can estimate to within 100 objects, how many values are indexed in a catalog by measuring the time it takes to draw the indexes page. Please do not remove this most valued feature! I see now winks so am scared ;-) Seriously though, it is kinda problematic when you want to get to the ZMI of an index and have to guess the URL 'cos hitting the indexes page cripples the server... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Dieter Maurer wrote: I suggest to change the title to # index terms and revert for the indexes to the old behaviour. If that'll make it quicker, cool :-) Others pointed out, that also the size determination for an index may be expensive. However, it is at most linear in the number (rather than quadratic) and all recently created indexes now use BTrees.Length to maintain their size (which gives constant time). Having a feeling how large an index is is valuable information. Indeed... Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Anthony Baxter wrote: if there's no counter present: create one, do a count of the docs, initialise the counter display counter Sounds good, what needs to happen to make this happen? Since this is a bug fix, can it go on the 2.6 branch? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
--On Freitag, 18. Juli 2003 13:52 Uhr +0100 Chris Withers [EMAIL PROTECTED] wrote: Dieter Maurer wrote: I suggest to change the title to # index terms and revert for the indexes to the old behaviour. If that'll make it quicker, cool :-) I am usually not interested in the number of index terms but in the number of documents that are indexed. This is much more meaningful. -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
--On Freitag, 18. Juli 2003 13:53 Uhr +0100 Chris Withers [EMAIL PROTECTED] wrote: Anthony Baxter wrote: if there's no counter present: create one, do a count of the docs, initialise the counter display counter Sounds good, what needs to happen to make this happen? Since this is a bug fix, can it go on the 2.6 branch? First write the fix and then let's see if it might break something :-) -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
On Friday 18 July 2003 01:29 pm, Dieter Maurer wrote: Anthony Baxter wrote at 2003-7-18 15:14 +1000: Andreas Jung wrote I agree but the current implementation sux. Switching to a counter based solution would solve the problem. The only problem I see is to keep the code fully backward compatible. if there's no counter present: create one, do a count of the docs, initialise the counter We can use the size of the _unindex. However, is it really worth it? #objects suggests that it is the number of objects indexed by this index. Who is interested in this information? Unless one has inhomogeous objects, almost all objects are indexed by every index. Thus, #objects is likely to be similar for many indexes. A much more interesting information would be the size of the index measured by the number of index terms. I agree. and as a plus, its a minor change to the software... -Casey ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Anthony Baxter wrote at 2003-7-18 15:14 +1000: Andreas Jung wrote I agree but the current implementation sux. Switching to a counter based solution would solve the problem. The only problem I see is to keep the code fully backward compatible. if there's no counter present: create one, do a count of the docs, initialise the counter We can use the size of the _unindex. However, is it really worth it? #objects suggests that it is the number of objects indexed by this index. Who is interested in this information? Unless one has inhomogeous objects, almost all objects are indexed by every index. Thus, #objects is likely to be similar for many indexes. A much more interesting information would be the size of the index measured by the number of index terms. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog Indexes tab crawls...
Hi, Has anyone noticed that the ZCatalog Indexes tab crawls if you have loads of objects indexed. My guess is that some types of index take way too long to figure out how many objects are indexed. Anyone know which index types those could be? BTW, would anyone object if I removed that object count, since it's not often very useful... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog Indexes tab crawl reason confirmed
Chris Withers wrote: My guess is that some types of index take way too long to figure out how many objects are indexed. This was confirmed by commenting out: dtml-var numObjects missing=n/a ...in catalogIndexes.dtml BTW, would anyone object if I removed that object count, since it's not often very useful... so... would anyone mind? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawl reason confirmed
so... would anyone mind? Well, I've often been interested to note the numbers. It gave me a feeling for which indexes are heavily used. Sure, I could figure this out without looking at this page, but the (lack of) speed hasn't bugged me .. -- Jean Jordaan http://www.upfrontsystems.co.za ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawl reason confirmed
--On Donnerstag, 17. Juli 2003 12:26 Uhr +0200 Jean Jordaan [EMAIL PROTECTED] wrote: so... would anyone mind? Well, I've often been interested to note the numbers. It gave me a feeling for which indexes are heavily used. Sure, I could figure this out without looking at this page, but the (lack of) speed hasn't bugged me . The problem is caused by calling len() on the indexes btrees. Instead a counter implemented btree.Length should be used in the future. -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog indexes tab - which Index Types are guilty?
--On Donnerstag, 17. Juli 2003 12:29 Uhr +0100 Chris Withers [EMAIL PROTECTED] wrote: The problem is caused by calling len() on the indexes btrees. Instead a counter implemented btree.Length should be used in the future. Which Index types are currently guilty of this? I think all except ZCTextIndex. How about re-naming the column to Number of Documents Indexed and making sure this is actually what the indexes return. ?? I have a feeling that not all index types actually return the number of objects indexed. Can anyone confirm this? TextIndex in an older version returned the number of indexed words but this is fixed at least since 2.6. -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Chris Withers wrote at 2003-7-17 11:12 +0100: Has anyone noticed that the ZCatalog Indexes tab crawls if you have loads of objects indexed. My guess is that some types of index take way too long to figure out how many objects are indexed. Anyone know which index types those could be? The one that provide the correct number of indexed objects (rather than just the number of indexed terms). Because the same object can be indexed under several terms, determining the number of indexed objects requires to build the union of all the index values. This almost surely has quadratic (worst case) runtime characteristics. BTW, would anyone object if I removed that object count, since it's not often very useful... You probably should replace it with the size of the index (i.e. the number of index terms). Formerly, the index overview displayed this information but under a buggy # objects title. Someone fixed this for most indexes, they now show the number of objects but at a high price. I suggest to change the title to # index terms and revert for the indexes to the old behaviour. Others pointed out, that also the size determination for an index may be expensive. However, it is at most linear in the number (rather than quadratic) and all recently created indexes now use BTrees.Length to maintain their size (which gives constant time). Having a feeling how large an index is is valuable information. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Actually I regard the current behavior as a feature. Using a stopwatch and a slide-rule I can estimate to within 100 objects, how many values are indexed in a catalog by measuring the time it takes to draw the indexes page. Please do not remove this most valued feature! -Casey On Thursday 17 July 2003 04:35 pm, Dieter Maurer wrote: Chris Withers wrote at 2003-7-17 11:12 +0100: Has anyone noticed that the ZCatalog Indexes tab crawls if you have loads of objects indexed. My guess is that some types of index take way too long to figure out how many objects are indexed. Anyone know which index types those could be? The one that provide the correct number of indexed objects (rather than just the number of indexed terms). Because the same object can be indexed under several terms, determining the number of indexed objects requires to build the union of all the index values. This almost surely has quadratic (worst case) runtime characteristics. BTW, would anyone object if I removed that object count, since it's not often very useful... You probably should replace it with the size of the index (i.e. the number of index terms). Formerly, the index overview displayed this information but under a buggy # objects title. Someone fixed this for most indexes, they now show the number of objects but at a high price. I suggest to change the title to # index terms and revert for the indexes to the old behaviour. Others pointed out, that also the size determination for an index may be expensive. However, it is at most linear in the number (rather than quadratic) and all recently created indexes now use BTrees.Length to maintain their size (which gives constant time). Having a feeling how large an index is is valuable information. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
--On Donnerstag, 17. Juli 2003 18:22 Uhr -0400 Casey Duncan [EMAIL PROTECTED] wrote: Actually I regard the current behavior as a feature. Using a stopwatch and a slide-rule I can estimate to within 100 objects, how many values are indexed in a catalog by measuring the time it takes to draw the indexes page. Please do not remove this most valued feature! I agree but the current implementation sux. Switching to a counter based solution would solve the problem. The only problem I see is to keep the code fully backward compatible. -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
