Re: [Zope-dev] ZCatalog and indexes cleanup
On 29.06.09 12:48, yuppie wrote: Hi! I did plan to work on a small catalog improvement, but after looking at the code I'd like to do some cleanup first: 1.) remove the deprecated TextIndex The deprecation warning says: 'Using TextIndex is deprecated (will be removed in Zope ' '2.12). Use ZCTextIndex instead.' +1 2.) remove CHANGES.txt, README.txt and version.txt from Products/ZCatalog These files seem to be obsolete. +1 3.) remove security declarations from ZCTextIndex and DateRangeIndex All the other indexes don't have security declarations. AFAICS there is no way to access indexes from untrusted code without having the 'Manage ZCatalogIndex Entries' permission. I think that all index implementation should have security assertions?! 4.) add 'indexSize' to IPluggableIndex and implement it where missing ZCatalog uses that method and most indexes implement it already. +1 Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and indexes cleanup
Hi Andreas! Andreas Jung wrote: On 29.06.09 12:48, yuppie wrote: 3.) remove security declarations from ZCTextIndex and DateRangeIndex All the other indexes don't have security declarations. AFAICS there is no way to access indexes from untrusted code without having the 'Manage ZCatalogIndex Entries' permission. I think that all index implementation should have security assertions?! Why? '_catalog.indexes' is protected by the underscore and using the 'Indexes' alias is protected by 'Manage ZCatalogIndex Entries'. Only additional security restrictions would have any effect. Or am I missing a security hole? Cheers, Yuppie ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and indexes cleanup
On 29.06.09 19:33, yuppie wrote:
Hi Andreas!
Andreas Jung wrote:
On 29.06.09 12:48, yuppie wrote:
3.) remove security declarations from ZCTextIndex and DateRangeIndex
All the other indexes don't have security declarations. AFAICS there is
no way to access indexes from untrusted code without having the 'Manage
ZCatalogIndex Entries' permission.
I think that all index implementation should have security assertions?!
Why?
'_catalog.indexes' is protected by the underscore and using the
'Indexes' alias is protected by 'Manage ZCatalogIndex Entries'. Only
additional security restrictions would have any effect.
Or am I missing a security hole?
Not sure. I created a catalog /catalog and an index 'my_index'.
Within a debug shell:
app.catalog.Indexes['my_index']
FieldIndex at my_index
app.unrestrictedTraverse('catalog/Indexes/my_index')
FieldIndex at /catalog//my_index
app.restrictedTraverse('catalog/Indexes/my_index')
Traceback (most recent call last):
File stdin, line 1, in ?
File
/Users/ajung/sandboxes/Zope-2.11/2.11/lib/python/OFS/Traversable.py,
line 301, in restrictedTraverse
return self.unrestrictedTraverse(path, default, restricted=True)
File
/Users/ajung/sandboxes/Zope-2.11/2.11/lib/python/OFS/Traversable.py,
line 236, in unrestrictedTraverse
next = guarded_getattr(obj, name)
AccessControl.unauthorized.Unauthorized: You are not allowed to access
'Indexes' in this context
h...
Andreas
begin:vcard
fn:Andreas Jung
n:Jung;Andreas
org:ZOPYX Ltd. Co. KG
adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany
email;internet:i...@zopyx.com
title:CEO
tel;work:+49-7071-793376
tel;fax:+49-7071-7936840
tel;home:+49-7071-793257
x-mozilla-html:FALSE
url:www.zopyx.com
version:2.1
end:vcard
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hedley Roos wrote: Since memcached is distributed only a single Zope client needs to perform that query and the result is available to all other Zope clients. This is where you'll get the big win: no need to load all the catalog-related objects into the zodb cache on all the clients which has the twin drawbacks of needing to be done and trashing your zodb cache... And the cache is persistent as long as memcached runs, so you can merrily restart Zope instances and have a warm cache. I didn't even realise this until Roche pointed it out to me. Coool :-) cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. Before (or while) we work on caching, can we try to improve the underlying indexes, and the way that applications use them? I'm pretty sure that there is a lot of room for improvement: - Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around just in case is a sure lose a write time, and may even be expensive at query time (depending on the query). - Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing. - I'm pretty sure that we haven't yet found the best data structure for hierarchy indexes (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies. - I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time. - Other special-purpose indexes (e.g., a recent items index) would be worth a look, especially for applications with large volumes of content. I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it? -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 27, 2008, at 13:08 , Roché Compaan wrote: On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote: - Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around just in case is a sure lose a write time, and may even be expensive at query time (depending on the query). - Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing. - I'm pretty sure that we haven't yet found the best data structure for hierarchy indexes (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies. - I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time. - Other special-purpose indexes (e.g., a recent items index) would be worth a look, especially for applications with large volumes of content. I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it? It's the typical trade-off. One course is expedient and fast for your use case now. The other requires more resources, but benefits everyone. Including those who don't want to depend on yet another package, like memcached, for performance. When it comes to integrating anything in Zope itself I'd choose the latter. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkkFssEACgkQRAx5nvEhZLITiQCgskifGYaixaj6lVLk85l6rz6E aQwAoI9PRcJHL8oZPatlHWADA0h6orCe =YLhP -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 27, 2008, at 13:08 , Roché Compaan wrote: On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote: - Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around just in case is a sure lose a write time, and may even be expensive at query time (depending on the query). - Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing. - I'm pretty sure that we haven't yet found the best data structure for hierarchy indexes (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies. - I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time. - Other special-purpose indexes (e.g., a recent items index) would be worth a look, especially for applications with large volumes of content. I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it? It's the typical trade-off. One course is expedient and fast for your use case now. The other requires more resources, but benefits everyone. Including those who don't want to depend on yet another package, like memcached, for performance. I'm not tied to memcached. We started out using module level caches like zope.cache.ram but that has obvious problems when using ZEO. When it comes to integrating anything in Zope itself I'd choose the latter. Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one). -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 27, 2008, at 13:32 , Roché Compaan wrote: On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote: When it comes to integrating anything in Zope itself I'd choose the latter. Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one). Right, it's perfectly valid to create such a third party package. The discussion just highlights a greater issue. Personally, I don't think it's good practice to focus on the expediency of working around a problem as opposed to tackling the problem directly. The Zope world is littered with add-ons that act as band-aids on real or perceived shortcomings in Zope itself. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkkFtvQACgkQRAx5nvEhZLLOrwCaA+X3iGaTDmyt3vP4q93OoTfx CNsAoJXppoHwI17ISetv4iAwoJeb+Phd =auan -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Mon, 2008-10-27 at 13:41 +0100, Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Oct 27, 2008, at 13:32 , Roché Compaan wrote: On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote: When it comes to integrating anything in Zope itself I'd choose the latter. Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one). Right, it's perfectly valid to create such a third party package. The discussion just highlights a greater issue. Personally, I don't think it's good practice to focus on the expediency of working around a problem as opposed to tackling the problem directly. The Zope world is littered with add-ons that act as band-aids on real or perceived shortcomings in Zope itself. Improving the performance of indexes is really really hard. In this case I really don't think caching is a band-aid, it is a good solution. Even with optimised indexes, you will find that you need caching to get reasonable performance if you have a catalog with close to a million or more documents indexed. Given a large enough catalog, I would argue that caching is equally as necessary as having a large cache for a ZEO client. But caches expire and results get invalidated, and therefor we should continue to optimise indexes. With some help we should be able to contribute at this level too. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Mon, Oct 27, 2008 at 12:33 PM, Andreas Jung [EMAIL PROTECTED] wrote: On 27.10.2008 16:28 Uhr, Rudá Porto Filgueiras wrote: I will sugest a package called zope.memcached (like zope.sqlalchemy does for SQLAlchemy integration). That way any application who need to talk memcached can do it with out loose atomicit. I don't see a particular reason for creating a new package for here. Extend lovely.memcached and your done. There is not much need for scattering tiny functionalies into two modules here. The module world is already complicated enough. If lovely,memcached alredy is safe when some Exception is raised, discard my sugestion. It's also compatible with zope2? Andreas -- = Rudá Porto Filgueiras Weimar Consultoria http://python-blog.blogspot.com Hospedagem Plone, Django, Zope 3, Grok... http://www.pytown.com = ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 27.10.2008 17:18 Uhr, Rudá Porto Filgueiras wrote: On Mon, Oct 27, 2008 at 12:33 PM, Andreas Jung[EMAIL PROTECTED] wrote: On 27.10.2008 16:28 Uhr, Rudá Porto Filgueiras wrote: I will sugest a package called zope.memcached (like zope.sqlalchemy does for SQLAlchemy integration). That way any application who need to talk memcached can do it with out loose atomicit. I don't see a particular reason for creating a new package for here. Extend lovely.memcached and your done. There is not much need for scattering tiny functionalies into two modules here. The module world is already complicated enough. If lovely,memcached alredy is safe when some Exception is raised, discard my sugestion. It's also compatible with zope2? We are using it together with our cache tool I mentioned earlier with Zope 2.8.1. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 26.10.2008 18:43 Uhr, Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. Path indexes and fulltext indexes have a much more complicated implementation compared to field or keyword indexes. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hi Roché, I can see your funkload profile? On Sun, Oct 26, 2008 at 3:43 PM, Roché Compaan [EMAIL PROTECTED]wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Fábio Rizzo Matos ThreePointsWeb [EMAIL PROTECTED] http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto! ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hi Fabio The funkload tests were project specific. I plan to write up my findings and to do benchmarks on a standard Plone instance and blog about it. This will unfortunately have to wait since I'm on holiday this week :-) -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za On Sun, 2008-10-26 at 15:54 -0200, Fabio Rizzo Matos wrote: Hi Roché, I can see your funkload profile? On Sun, Oct 26, 2008 at 3:43 PM, Roché Compaan [EMAIL PROTECTED] wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Fábio Rizzo Matos ThreePointsWeb [EMAIL PROTECTED] http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto! ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sun, 2008-10-26 at 18:50 +0100, Andreas Jung wrote: On 26.10.2008 18:43 Uhr, Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. Path indexes and fulltext indexes have a much more complicated implementation compared to field or keyword indexes. I know, and this alone makes a good argument for caching at catalog level. In our case we used membrane, which makes an excessive amount of catalog queries when looking up users so some level of caching was essential. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. Before (or while) we work on caching, can we try to improve the underlying indexes, and the way that applications use them? I'm pretty sure that there is a lot of room for improvement: - Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around just in case is a sure lose a write time, and may even be expensive at query time (depending on the query). - Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing. - I'm pretty sure that we haven't yet found the best data structure for hierarchy indexes (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies. - I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time. - Other special-purpose indexes (e.g., a recent items index) would be worth a look, especially for applications with large volumes of content. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJBLHb+gerLs4ltQ4RAp59AJwNlfjI0tBv4PdMiDdH4TLKSm5YfwCgu8xB F3u1G0onXKKZ4s7MbLj9B2w= =r0oE -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Very Nice. Have a nice holiday :-) On Sun, Oct 26, 2008 at 3:58 PM, Roché Compaan [EMAIL PROTECTED]wrote: Hi Fabio The funkload tests were project specific. I plan to write up my findings and to do benchmarks on a standard Plone instance and blog about it. This will unfortunately have to wait since I'm on holiday this week :-) -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za On Sun, 2008-10-26 at 15:54 -0200, Fabio Rizzo Matos wrote: Hi Roché, I can see your funkload profile? On Sun, Oct 26, 2008 at 3:43 PM, Roché Compaan [EMAIL PROTECTED] wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Fábio Rizzo Matos ThreePointsWeb [EMAIL PROTECTED] http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto! ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- Fábio Rizzo Matos ThreePointsWeb [EMAIL PROTECTED] http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto! ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 26.10.2008 19:05 Uhr, Roché Compaan wrote: On Sun, 2008-10-26 at 18:50 +0100, Andreas Jung wrote: On 26.10.2008 18:43 Uhr, Roché Compaan wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. Path indexes and fulltext indexes have a much more complicated implementation compared to field or keyword indexes. I know, and this alone makes a good argument for caching at catalog level. In our case we used membrane, which makes an excessive amount of catalog queries when looking up users so some level of caching was essential. First caching is good thing :-) But how about the following issue: CMF/Plone inject additional subqueries for expires/effective/typesAndRoles. At least the security related aubqueries make a cached catalog result very specific to a particular user. That seems to be very ok for a site with lots of anonymous users - it might be an issue with lots of authenticated users. It might be necessary to add some kind of intelligence to decide what to cache and what not. I don't think it does not make sense to cache the result of a fulltext search. I am just thinking if it would make sense to cache on the index level instead of catalog level? So you could for example cache expensive index queries (path index) and combine them with uncached index which are supposed to be fast..however such decisions require detailed mesurements on real systems. One other thing concerning memcached: there is obviously a limit to 1MB for data you can store as a value. We have not found an obvious way for increasing this limit other by patching the memcached sources. We came up with an implementation where data 1MB is split up into individual junks (we have a dedicated set_huge(), get_huge()) implementation. Andreas -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: [EMAIL PROTECTED] - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK E-Publishing, Python, Zope Plone development, Consulting begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most. If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that. It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly. Run a few tests and let me know please. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 25.10.2008 8:48 Uhr, Hedley Roos wrote: The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most. If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that. It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly. Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. As for standard queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now. Since memcached is distributed only a single Zope client needs to perform that query and the result is available to all other Zope clients. And the cache is persistent as long as memcached runs, so you can merrily restart Zope instances and have a warm cache. I didn't even realise this until Roche pointed it out to me. To answer the question: I believe catalogcache will win every time since the return time of a cached query is not dependent on the complexity of the query. We should get a few benchmarks running at query level. I'll have a bit of time next week. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hedley Roos wrote: As for standard queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now. sortable_title is a field index and shouldn't be slower than any other index. This all sounds very cool, by the way. :) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Hi, On Fri, 2008-10-24 at 15:41 +0200, Hedley Roos wrote: The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions. I'd love if this wouldn't be a monkey patch. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Method search just inspects queries and stores results to memcache. Can you give me an example where the cache would deliver non-committed data? ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
In addition, you need to include a serial in your cache keys to avoid dirty reads. The cache invalidation code actively removes items from the cache. Am I understanding you correctly? H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On 25.10.2008 14:53 Uhr, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Using a DataManager is likely the better and more safe choice. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, 2008-10-25 at 14:53 +0200, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. This is probably harmless but will cause unnecessary cache flushes for other clients. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Right. This is the same as clear. Method search just inspects queries and stores results to memcache. That's the issue. If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed. Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, Oct 25, 2008 at 2:57 PM, Andreas Jung [EMAIL PROTECTED] wrote: On 25.10.2008 14:53 Uhr, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Using a DataManager is likely the better and more safe choice. Andreas Thanks Andreas. I'll have a look at your code when available. Christian, I do have a mistake in my reasoning. If an object is added to the catalog in a transaction and I cache that object as result of a query in that same transaction, and then the transaction fails I'll have a bad cache. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed. Kind of like how I came to the same conclusion in parallel to you and stuffed up this thread :) Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. So this is the area I have to focus on right now. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. A possible solution is to keep a module level cache which can be committed to the memcache on transaction boundaries. That way I'll incur no performance penalty. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, 2008-10-25 at 14:55 +0200, Hedley Roos wrote: In addition, you need to include a serial in your cache keys to avoid dirty reads. The cache invalidation code actively removes items from the cache. Am I understanding you correctly? I wasn't even talking about invalidation as your cache wouldn't see 'invalidations' anyways. It's memcached's task to forget stuff: it's a cache anyway. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog sorting issue
--On 10. Juli 2008 15:06:53 +0200 Martijn Jacobs [EMAIL PROTECTED] wrote: Wat denk je hiervan? Hello. In zope 2.10.5 (and probably 2.10.6 and 2.11 and, as I've read, all releases above 2.7) we've encountered a sorting bug in a dtml-in call when querying the catalog. I don't think it's dtml only related, but I'm not sure about that. It is the same bug as found on : https://bugs.launchpad.net/zope2/+bug/143504 seems to correct the problem though and I was wondering why this isn't added in the zope core. Does it break other stuff or should the problem fixed somewhere else? Somebody has some thoughts? Here is the deal: you provide a unittest for the patch and I commit the patch + tests. Andreas pgpjOA2XtXB7c.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog: updateMetadata and comparing string and unicode
Maurits van Rees wrote at 2008-3-5 23:57 +: ... I have an item in the portal_catalog of my Plone site that has some string as description. The real object meanwhile has had a code change so the description field now returns unicode. When I now recatalog that object it throws an error: Module Products.ZCatalog.Catalog, line 359, in catalogObject Module Products.ZCatalog.Catalog, line 318, in updateMetadata UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 159: ordinal not in range(128) /home/maurits/buildout/projectdeploy/parts/zope2/lib/python/Products/ZCatalog/Catalog.py(318)updateMetadata() - if data.get(index, 0) != newDataRecord: You must not mix unicode and str as keys in the same index. If you do, errors as the above are very likely. You can try the following approaches: * if you know the encoding used by your str objects, you can set Python's default encoding to this encoding. Whenever unicode and str come together, the str is converted to unicode using this encoding (which hopefully is the correct one in all such cases). sys.setdefaultencoding is only available at startup. Thus, setting defaultencoding must happen in a sitecustomize or site module. * You completely switch to unicode for the given index and convert the BTrees used be the index. An index usually uses two BTrees: the so called forward index (usually called _index) (it maps the index terms to sets of record ids indexed under this term) and the reverse index (usually called _unindex) (it maps record ids to the values corresponding to these objects). You need to convert the keys of the forward index and the values of the reverse index. For a FieldIndex, the value is the index term, for a KeywordIndex it it a sequence of index terms (all need be converted). The forward index can be converted as follows: self._index = OOBTree(((s.decode(your encoding), v) for (s,v) in self._index.items())) The reverse index uses an IOBTree and is similar to the above. But the details depend on index type. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog: updateMetadata and comparing string and unicode
Dieter Maurer wrote: sys.setdefaultencoding is only available at startup. Thus, setting defaultencoding must happen in a sitecustomize or site module. Or if you're sufficiently devious, it's available any time (not that actually using it is a good idea, but...): import sys sys.setdefaultencoding Traceback (most recent call last): File stdin, line 1, in module AttributeError: 'module' object has no attribute 'setdefaultencoding' del sys.modules['sys'] import sys sys.setdefaultencoding built-in function setdefaultencoding -- Benji York Senior Software Engineer Zope Corporation ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
[EMAIL PROTECTED] wrote at 2006-9-3 18:06 -0400: Thank you so much for following-up with me, here. I think you're right about this having to do with the indexes on the ZCatalog. The catalog has: 1. Seven FieldIndex indexes. 2. A DateIndex index 3. Three ZCTextIndex indexes I have not seen reports that these indexes cause inconsistencies of the type you have described. A long time ago, there have been problems in the BTrees conflict resolution which caused catalog inconsistencies (of a different kind). They affected document lists of length one in the index. When the last remaining document was removed and a new one concurrently inserted, the insertion could be lost. But this was fixed a long time ago. Apart from bugs, inconsistencies are often caused by try: ... except: ... clauses. If the try clause modifies the persistent state and then is aborted by an exception, it is vital that the exception reaches the ZPublisher such that it aborts the transaction. If the except prevents this, then the ZPublisher will commit and you are likely to get inconsistencies. In older Zope versions (before Zope 2.8.1), such a dangerous try: ... except: ... has been in OFS.ObjectManager.ObjectManager.manage_beforeDelete. In more modern versions, the code has been slightly cleaned up but it still is error prone. Skim your log files whether you see manage_beforeDelete () threw ERROR log entries. You can get more hints towards the problem by examining the query that results in your exception: If this query just consists of a subquery to a single index, then you know the problematic index. If it combines several indexes (by an and), then all of these indexes would need to be inconsistent (which is possible but far less likely than a single index). In this case, one would expect that not the index but the catalog primary data got corrupted. In the latter case, the problem would need to happen during initial indexing or final unindexing. Studying the code may exclude some possibilities. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
[EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400: Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): ... - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 This does not look like a catalog corruption. Instead, it looks as if the catalog changed since the search was done. Are you caching your search results? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Dieter Maurer wrote at 2006-9-3 19:22 +0200: [EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400: Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): ... - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 This does not look like a catalog corruption. Instead, it looks as if the catalog changed since the search was done. Are you caching your search results? Sorry! I was wrong! It has nothing to do with a catalog modification after the search. Looks indeed as if an index reported a document which is no longer indexed. Looks like an index bug. Which (types of) indexes are involved in your failing queries? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Thank you so much for following-up with me, here. I think you're right
about this having to do with the indexes on the ZCatalog.
The catalog has:
1. Seven FieldIndex indexes.
2. A DateIndex index
3. Three ZCTextIndex indexes
I actually create these from python in a function that accepts the
catalog as a parameter during the Product Initialization process.
I have a function catalog_add that creates and initializes the a
ZCatalog when this product is created and initialized, so let me
actually post the sourcecode here for that function and the functions it
calls. It's fairly short. There are two functions of note
catalog_add and addIndexes. I'd actually be grateful if you would
take a look and let me know if you see anything. In particular, am I
using an index that you know to have problems, or that I'm using
improperly.
class Newsdesk(Folder,Persistent,Implicit):
meta_type = Newsdesk
# ...
def catalog_add(self):
setup the catalog
self.manage_addProduct[ZCatalog].manage_addZCatalog(Catalog,
Default Catalog)
catalog = self._getOb('Catalog')
self.addLexicon(catalog)
self.addIndexes(catalog)
self.addMetas(catalog)
catalog = self._getOb('Catalog')
return catalog
def addIndexes(self, cat):
cat.addIndex('date_found', 'FieldIndex', 'date_found')
cat.addIndex('deleted_p', 'FieldIndex', 'deleted_p')
cat.addIndex('uri', 'FieldIndex', 'uri')
cat.addIndex('categories_field', 'FieldIndex',
'categories')
cat.addIndex('categorized_p', 'FieldIndex',
'categorized_p')
cat.addIndex('approved_p', 'FieldIndex', 'approved_p')
cat.addIndex('search_id', 'FieldIndex', 'search_id')
extras = Empty()
extras.doc_attr = 'title'
extras.index_type = 'Okapi BM25 Rank'
extras.lexicon_id = 'Lexicon'
cat.addIndex('title', 'ZCTextIndex', extras)
extras2 = Empty()
extras2.doc_attr = 'source_name'
extras2.index_type = 'Okapi BM25 Rank'
extras2.lexicon_id = 'Lexicon'
cat.addIndex('source_name', 'ZCTextIndex', extras2)
extras3 = Empty()
extras3.doc_attr = 'categories'
extras3.index_type = 'Okapi BM25 Rank'
extras3.lexicon_id = 'Lexicon'
cat.addIndex('categories', 'ZCTextIndex', extras3)
cat.addIndex('bobobase_modification_time', 'DateIndex')
def addMetas(self, cat):
create the metadata columns
cat.manage_addColumn(categorized_p)
cat.manage_addColumn(deleted_p)
cat.manage_addColumn(search_id)
cat.manage_addColumn(categories)
cat.manage_addColumn(date_found)
cat.manage_addColumn(source_name)
cat.manage_addColumn(title)
cat.manage_addColumn(uri)
def addLexicon(self, cat):
elem = []
wordSplitter = Empty()
wordSplitter.group = 'Word Splitter'
wordSplitter.name = 'HTML aware splitter'
caseNormalizer = Empty()
caseNormalizer.group = 'Case Normalizer'
caseNormalizer.name = 'Case Normalizer'
stopWords = Empty()
stopWords.group = 'Stop Words'
stopWords.name = 'Remove listed and single char words'
elem.append(wordSplitter)
elem.append(caseNormalizer)
elem.append(stopWords)
cat.manage_addProduct['ZCTextIndex'].manage_addLexicon('Lexicon',
'Default Lexicon', elem)
On Sun, 3 Sep 2006 20:11:33 +0200, Dieter Maurer [EMAIL PROTECTED]
said:
Dieter Maurer wrote at 2006-9-3 19:22 +0200:
[EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400:
Traceback for my Key Error (Dieter, thank you for alerting me on where
to find this):
Traceback (innermost last):
...
- Physical Path: /nephron_links/news_dtml
Module DocumentTemplate.DT_String, line 476, in __call__
Module DocumentTemplate.DT_In, line 465, in renderwb
Module DocumentTemplate.DT_In, line 747, in sort_sequence
Module Products.ZCatalog.Lazy, line 158, in __getitem__
Module Products.ZCatalog.Catalog, line 520, in getScoredResult
KeyError: -464571725
This does not look like a catalog corruption.
Instead, it looks as if the catalog changed since the search was
done. Are you caching your search results?
Sorry! I was wrong!
It has nothing to do with a catalog modification after the search.
Looks indeed as if an index reported a document which is no longer
indexed. Looks like an index bug. Which
Re: [Zope-dev] ZCatalog Key Error question
--On 1. September 2006 02:54:26 -0400 [EMAIL PROTECTED] wrote: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Read or Write conflict errors? Are the objects modified at the same time by other threads? -aj pgpRrhqcwgEtE.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Read or Write conflict errors? I'm getting the Key Error traceback on Reads, but my guess is that the actual error happens sometime before the Key Error, when the ZCatalog itself is corrupted. I am not sure what that is, because it does not generate an error. However, I see the error when I call ZCatalogInstance.Search(...), and only when the Search includes objects that are corrupted in the ZCatalog. Are the objects modified at the same time by other threads? I'm not sure about this. I'm running ZEO, and in general, I think I can cause the error when I am the only user on the system, only issuing one request. However, I'm getting another ConflictError when I try to set an attribute value of a ZODB object during the execution of a script, and I'm told that may be because there is a problem accessing the same object from multiple threads. Is there a good article that you know of that I could find out more about thread-safe programming in ZEO? Best, Brian On Fri, 01 Sep 2006 09:27:41 +0200, Andreas Jung [EMAIL PROTECTED] said: --On 1. September 2006 02:54:26 -0400 [EMAIL PROTECTED] wrote: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Read or Write conflict errors? Are the objects modified at the same time by other threads? -aj Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
[EMAIL PROTECTED] wrote at 2006-9-1 02:54 -0400: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Here are the clues: You forget to provide us with an essential glue: The complete error information (error type, error value and traceback). You find it in your error_log object (in Zope's Root Folder). -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): Module ZPublisher.Publish, line 113, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 40, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.PythonScripts.PythonScript, line 323, in _exec Module None, line 11, in to_javascript - PythonScript at /nephron_links/to_javascript used for /nephron_links/news_dtml - Line 11 Module OFS.DTMLDocument, line 128, in __call__ - DTMLDocument at /nephron_links/news_dtml - URL: http://nephron.org/news_dtml/manage_main - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 On Fri, 1 Sep 2006 19:11:12 +0200, Dieter Maurer [EMAIL PROTECTED] said: [EMAIL PROTECTED] wrote at 2006-9-1 02:54 -0400: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Here are the clues: You forget to provide us with an essential glue: The complete error information (error type, error value and traceback). You find it in your error_log object (in Zope's Root Folder). -- Dieter Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
Guys, Dieter Maurer [EMAIL PROTECTED] wrote: Roché Compaan wrote at 2005-2-25 17:22 +0200: Last year in March the following checkin was made that changed ZCatalog's getObject to use restrictedTraverse instead of unrestrictedTraverse. See: http://mail.zope.org/pipermail/zope-checkins/2004-March/026846.html In my opininion this is wrong, I agree with you! Me also. ... I would propose that getObject does an unrestrictedTraverse of the path and then checks if the user has permission to access that the object. I argued precisely this approach with the person who made the change. I had the impression that I have convinced him -- but apparently, he did not change the code accordingly :-( Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope Roché has added http://www.zope.org/Collectors/Zope/1713 I intend to fix this before 2.7.5 final, probably today or tonight. I feel this is sufficiently important to warrant a fix now. I guess it'll mean an RC2. Please shout if you find problems with this approach. Florent -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
--On Donnerstag, 10. März 2005 12:49 Uhr +0100 Florent Guillaume [EMAIL PROTECTED] wrote: Guys, Dieter Maurer [EMAIL PROTECTED] wrote: Roché Compaan wrote at 2005-2-25 17:22 +0200: Last year in March the following checkin was made that changed ZCatalog's getObject to use restrictedTraverse instead of unrestrictedTraverse. See: http://mail.zope.org/pipermail/zope-checkins/2004-March/026846.html In my opininion this is wrong, I agree with you! Me also. ... I would propose that getObject does an unrestrictedTraverse of the path and then checks if the user has permission to access that the object. I argued precisely this approach with the person who made the change. I had the impression that I have convinced him -- but apparently, he did not change the code accordingly :-( Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope Roché has added http://www.zope.org/Collectors/Zope/1713 I intend to fix this before 2.7.5 final, probably today or tonight. I feel this is sufficiently important to warrant a fix now. I guess it'll mean an RC2. Please see my remark on this issue in the collector. Andreas pgphJkbk8eW1O.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
Roché Compaan wrote at 2005-3-3 22:36 +0200: On Thu, 2005-03-03 at 19:36 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2005-3-3 09:53 +0200: ... -return self.aq_parent.restrictedTraverse(self.getPath(), None) +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): I think this is not correct: validate needs at least a value parameter (this is the forth parameter). I thought this much but what value? And doesn't this make the implementation of restrictedTraverse suspect too? When code is calling getObject on a catalog brain we don't know what attribute or method of that object the calling code will access. Does it then make any sense at all to do security checks in getObject? IMO it doesn't. Value means the accessed value. In your case, this is obj. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
Roché Compaan wrote: +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): +return obj +else: +return None Urm, Roche, doesn't the above seek to do exactly what... return self.aq_parent.restrictedTraverse(self.getPath(), None) ...does? The problem is that an error should be raised, Unauthorized in my opinion, rather than None being returned. None should never be returned in place of a brain, although I'll soften that to say that if it does, it means something weird has happened (used to mean the object the catalog entry mapped to had gone away) I think: self.aq_parent.restrictedTraverse(self.getPath()) ...should be fine, no? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
On Thu, 2005-03-03 at 14:56 +, Chris Withers wrote: Roché Compaan wrote: +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): +return obj +else: +return None Urm, Roche, doesn't the above seek to do exactly what... return self.aq_parent.restrictedTraverse(self.getPath(), None) ...does? No it doesn't, restrictedTraverse fails along the way. If the path is /a/b and the user doesn't have access to /a/ restrictedTraverse will return None even though the user has access to /a/b/. In my code above we only do a security check on the object that the full path resolves to. The problem is that an error should be raised, Unauthorized in my opinion, rather than None being returned. I would be ok with raising Unauthorized but it is not backwards compatible. I suppose changing to 'unrestrictedTraverse' is also not backward compatible but the current 'getObject' seems to suggest that we do not want to raise an exception when the user does not have permission to access the object. Is there some use case for 'getObject' that we are missing here? None should never be returned in place of a brain, although I'll soften that to say that if it does, it means something weird has happened (used to mean the object the catalog entry mapped to had gone away) I agree. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
Roché Compaan wrote at 2005-3-3 09:53 +0200: ... -return self.aq_parent.restrictedTraverse(self.getPath(), None) +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): I think this is not correct: validate needs at least a value parameter (this is the forth parameter). There is a validateValue method (instead of validate) that does what you want. You find it in AccessControl/ImplPython.py. Drawback: it might disappear in Zope 2.8. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
On Thu, 2005-03-03 at 19:36 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2005-3-3 09:53 +0200: ... -return self.aq_parent.restrictedTraverse(self.getPath(), None) +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): I think this is not correct: validate needs at least a value parameter (this is the forth parameter). I thought this much but what value? And doesn't this make the implementation of restrictedTraverse suspect too? When code is calling getObject on a catalog brain we don't know what attribute or method of that object the calling code will access. Does it then make any sense at all to do security checks in getObject? IMO it doesn't. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
--On Mittwoch, 2. März 2005 9:56 Uhr + Chris Withers [EMAIL PROTECTED] wrote: Roché Compaan wrote: Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope Well, I posted just such an issue a few months back. I'm working offline so can't give you the exact number but have a search and you should find it. I seem to remember Andreas rejecting it without thinking, as is his way ;-) I would appreciate it you would help resolve outstanding issues instead of making such statement :-) Andreas pgpGFKGurRtBC.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
On Fri, 2005-02-25 at 21:06 +0100, Andreas Jung wrote: --On Freitag, 25. Februar 2005 20:21 Uhr +0100 Dieter Maurer [EMAIL PROTECTED] wrote: Roché Compaan wrote at 2005-2-25 17:22 +0200: Last year in March the following checkin was made that changed ZCatalog's getObject to use restrictedTraverse instead of unrestrictedTraverse. See: http://mail.zope.org/pipermail/zope-checkins/2004-March/026846.html In my opininion this is wrong, I agree with you! ... I would propose that getObject does an unrestrictedTraverse of the path and then checks if the user has permission to access that the object. I argued precisely this approach with the person who made the change. I had the impression that I have convinced him -- but apparently, he did not change the code accordingly :-( Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope Best to include a patch as well :-) -aj I'm unsure about the security check in the patch below - I copied the way restrictedTraverse does it. I read through validate in the default security policy but it is one of those methods where all the security implications doesn't fit in your head all at once. --- CatalogBrains.py~ 2004-03-23 22:27:23.0 +0200 +++ CatalogBrains.py2005-03-03 09:43:48.0 +0200 @@ -47,7 +47,11 @@ (i.e., it was deleted or moved without recataloging), or if the user is not authorized to access an object along the path. -return self.aq_parent.restrictedTraverse(self.getPath(), None) +obj = self.aq_parent.unrestrictedTraverse(self.getPath(), None) +if obj and securityManager.validate(obj, obj, None, None): +return obj +else: +return None def getRID(self): Return the record ID for this object. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog getObject broken
Roché Compaan wrote at 2005-2-25 17:22 +0200: Last year in March the following checkin was made that changed ZCatalog's getObject to use restrictedTraverse instead of unrestrictedTraverse. See: http://mail.zope.org/pipermail/zope-checkins/2004-March/026846.html In my opininion this is wrong, I agree with you! ... I would propose that getObject does an unrestrictedTraverse of the path and then checks if the user has permission to access that the object. I argued precisely this approach with the person who made the change. I had the impression that I have convinced him -- but apparently, he did not change the code accordingly :-( Maybe, a bug report to the collector will help? http://www.zope.org/Collectors/Zope -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and Zope 2.8
--On Sonntag, 30. Januar 2005 19:17 Uhr +0100 Dieter Maurer [EMAIL PROTECTED] wrote: - some indexes show the number of indexed objects vs the number of indexes terms within the ZMI which is totally inconsistent. I think the ZMI should show the number of indexed objects. Index specific information e.g. the number of indexed terms should be shown within the indexes default view (if necessary). Objections? This was already discussed (-- mailing list archives). I (and others) find it more informative to get a feeling about the size of the index (rather than the number of indexed objects) in the overview. There are two points of view: the normal user is confused if some indexes show up the number of indexed objects and others show the size of the index within the same column. This should be consistent. If would prefer the number of indexed objects within the default ZMI view and put index specific size information into their own default view. - Indexes derived from UnIndex also store informations about objects although they do not index useful information. ... An optimised version of UnIndex would store only values evaluating to non-zero. It would essentially change the number of indexed objects (and make it a bit more informative when one is interested in the size of the index) ;-) right. You must be a bit careful with the non-zero. Some indexes interpret (some) zero values in a special way, e.g. DateRangeIndex. It interprets None as no limit. Yes, but DateRangeIndexes overwrite index_object() so that a change of UnIndex itself would not harm. I would also prefer when None would consistently means: I do not have a value (in the current context). But, this will interfere with some indexes. None could be a problem with other indexes...at least there should be a unique marker saying: I have nothing of interest to be indexed Andreas ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and Zope 2.8
Andreas Jung wrote at 2005-1-31 18:50 +0100: ... [AJ] - some indexes show the number of indexed objects vs the number of indexes terms within the ZMI which is totally inconsistent. I think the ZMI should show the number of indexed objects. Index specific information e.g. the number of indexed terms should be shown within the indexes default view (if necessary). Objections? [DM] This was already discussed (-- mailing list archives). I (and others) find it more informative to get a feeling about the size of the index (rather than the number of indexed objects) in the overview. [AJ] There are two points of view: the normal user is confused if some indexes show up the number of indexed objects and others show the size of the index within the same column. This should be consistent. If would prefer the number of indexed objects within the default ZMI view and put index specific size information into their own default view. [DM] All are with you (including myself) when you strive for consistency. The display should be consistent and correspond to the label in the table head. I am not with you with respect to number of indexed objects versus size of the index. In fact *BOTH* as index specific (otherwise, it would not make any sense to list it in an index specific column). Maybe, a compromize would be to include both numbers? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and Zope 2.8
--On Montag, 31. Januar 2005 20:20 Uhr +0100 Dieter Maurer [EMAIL PROTECTED] wrote: I am not with you with respect to number of indexed objects versus size of the index. In fact *BOTH* as index specific (otherwise, it would not make any sense to list it in an index specific column). Maybe, a compromize would be to include both numbers? I am currently working on a solution for all these issues on a dedicated 2.8 branch. Andreas ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and Zope 2.8
Andreas Jung wrote at 2005-1-30 15:30 +0100: ... Outstanding issues: - some indexes show the number of indexed objects vs the number of indexes terms within the ZMI which is totally inconsistent. I think the ZMI should show the number of indexed objects. Index specific information e.g. the number of indexed terms should be shown within the indexes default view (if necessary). Objections? This was already discussed (-- mailing list archives). I (and others) find it more informative to get a feeling about the size of the index (rather than the number of indexed objects) in the overview. - Indexes derived from UnIndex also store informations about objects although they do not index useful information. ... An optimised version of UnIndex would store only values evaluating to non-zero. It would essentially change the number of indexed objects (and make it a bit more informative when one is interested in the size of the index) ;-) You must be a bit careful with the non-zero. Some indexes interpret (some) zero values in a special way, e.g. DateRangeIndex. It interprets None as no limit. However, I agree with you: at least when an object o does not define a value for index i, then i should not index o. I would also prefer when None would consistently means: I do not have a value (in the current context). But, this will interfere with some indexes. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog fast as admin, dog slow as anonymous and other users
Jason Spisak wrote: Zopistas, My ZCatalog is fast as admin, but dog slow as anonymous and other users. Anyone had this same experience? Details: This is certainly to do with queries which run against FieldIndexes only for anonymous users. This is something the CMF does (e.g. expires and effective dates only applied for anonymous). FieldIndexes are really bad for DateTimes. Delete your date-related FieldIndexes and replace them with DateIndexes. seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog fast as admin, dog slow as anonymous and other users
Seb, thanks! i'll give that a shot. i also thought of adding daterange indexes for the calendar portion or things. does that also garner any speed improvement? Seb Bacon wrote: Jason Spisak wrote: Zopistas, My ZCatalog is fast as admin, but dog slow as anonymous and other users. Anyone had this same experience? Details: This is certainly to do with queries which run against FieldIndexes only for anonymous users. This is something the CMF does (e.g. expires and effective dates only applied for anonymous). FieldIndexes are really bad for DateTimes. Delete your date-related FieldIndexes and replace them with DateIndexes. seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog fast as admin, dog slow as anonymous and other users
Seb, I've replaced the following date indexes that werer FieldIndexes with DateIndexes Date effective expires modified i'll let you know what kind of speed improvement I get. Jason Spisak wrote: Zopistas, My ZCatalog is fast as admin, but dog slow as anonymous and other users. Anyone had this same experience? Details: This is certainly to do with queries which run against FieldIndexes only for anonymous users. This is something the CMF does (e.g. expires and effective dates only applied for anonymous). FieldIndexes are really bad for DateTimes. Delete your date-related FieldIndexes and replace them with DateIndexes. seb Jason Spisak ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Dieter Maurer wrote: But overall, unless you have special (non DC derived) indexes, That can well be the case... Anyway, what are we going to do about this crawling tab? Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Chris Withers wrote at 2003-7-21 08:22 +0100: Dieter Maurer wrote: #objects suggests that it is the number of objects indexed by this index. Who is interested in this information? Well, it's been useful to be on several occasions when I've seen one index has less objects in than another... Unless one has inhomogeous objects, almost all objects are indexed by every index. Thus, #objects is likely to be similar for many indexes. Hmmm... I use ZCatalogs a _lot_ for searching over inhomogenous sets of objects. For example, that's it's primary role in the CMF... CMF's catalog is highly standardized, thanks to Dublin Core. All standard CMF content types define DC attributes. Therefore, each CMF content object is indexed under each DC field index. The Subjects index may lack some objects (because they do not define any Subjects). A text index may lack a few objects (because some objects may have have both an empty Title and an empty Descritpion). But overall, unless you have special (non DC derived) indexes, all #objects should be very similar. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Dieter Maurer wrote: #objects suggests that it is the number of objects indexed by this index. Who is interested in this information? Well, it's been useful to be on several occasions when I've seen one index has less objects in than another... Unless one has inhomogeous objects, almost all objects are indexed by every index. Thus, #objects is likely to be similar for many indexes. Hmmm... I use ZCatalogs a _lot_ for searching over inhomogenous sets of objects. For example, that's it's primary role in the CMF... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Casey Duncan wrote: Actually I regard the current behavior as a feature. Using a stopwatch and a slide-rule I can estimate to within 100 objects, how many values are indexed in a catalog by measuring the time it takes to draw the indexes page. Please do not remove this most valued feature! I see now winks so am scared ;-) Seriously though, it is kinda problematic when you want to get to the ZMI of an index and have to guess the URL 'cos hitting the indexes page cripples the server... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Dieter Maurer wrote: I suggest to change the title to # index terms and revert for the indexes to the old behaviour. If that'll make it quicker, cool :-) Others pointed out, that also the size determination for an index may be expensive. However, it is at most linear in the number (rather than quadratic) and all recently created indexes now use BTrees.Length to maintain their size (which gives constant time). Having a feeling how large an index is is valuable information. Indeed... Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Anthony Baxter wrote: if there's no counter present: create one, do a count of the docs, initialise the counter display counter Sounds good, what needs to happen to make this happen? Since this is a bug fix, can it go on the 2.6 branch? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
--On Freitag, 18. Juli 2003 13:52 Uhr +0100 Chris Withers [EMAIL PROTECTED] wrote: Dieter Maurer wrote: I suggest to change the title to # index terms and revert for the indexes to the old behaviour. If that'll make it quicker, cool :-) I am usually not interested in the number of index terms but in the number of documents that are indexed. This is much more meaningful. -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
--On Freitag, 18. Juli 2003 13:53 Uhr +0100 Chris Withers [EMAIL PROTECTED] wrote: Anthony Baxter wrote: if there's no counter present: create one, do a count of the docs, initialise the counter display counter Sounds good, what needs to happen to make this happen? Since this is a bug fix, can it go on the 2.6 branch? First write the fix and then let's see if it might break something :-) -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
On Friday 18 July 2003 01:29 pm, Dieter Maurer wrote: Anthony Baxter wrote at 2003-7-18 15:14 +1000: Andreas Jung wrote I agree but the current implementation sux. Switching to a counter based solution would solve the problem. The only problem I see is to keep the code fully backward compatible. if there's no counter present: create one, do a count of the docs, initialise the counter We can use the size of the _unindex. However, is it really worth it? #objects suggests that it is the number of objects indexed by this index. Who is interested in this information? Unless one has inhomogeous objects, almost all objects are indexed by every index. Thus, #objects is likely to be similar for many indexes. A much more interesting information would be the size of the index measured by the number of index terms. I agree. and as a plus, its a minor change to the software... -Casey ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Anthony Baxter wrote at 2003-7-18 15:14 +1000: Andreas Jung wrote I agree but the current implementation sux. Switching to a counter based solution would solve the problem. The only problem I see is to keep the code fully backward compatible. if there's no counter present: create one, do a count of the docs, initialise the counter We can use the size of the _unindex. However, is it really worth it? #objects suggests that it is the number of objects indexed by this index. Who is interested in this information? Unless one has inhomogeous objects, almost all objects are indexed by every index. Thus, #objects is likely to be similar for many indexes. A much more interesting information would be the size of the index measured by the number of index terms. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawl reason confirmed
so... would anyone mind? Well, I've often been interested to note the numbers. It gave me a feeling for which indexes are heavily used. Sure, I could figure this out without looking at this page, but the (lack of) speed hasn't bugged me .. -- Jean Jordaan http://www.upfrontsystems.co.za ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawl reason confirmed
--On Donnerstag, 17. Juli 2003 12:26 Uhr +0200 Jean Jordaan [EMAIL PROTECTED] wrote: so... would anyone mind? Well, I've often been interested to note the numbers. It gave me a feeling for which indexes are heavily used. Sure, I could figure this out without looking at this page, but the (lack of) speed hasn't bugged me . The problem is caused by calling len() on the indexes btrees. Instead a counter implemented btree.Length should be used in the future. -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog indexes tab - which Index Types are guilty?
--On Donnerstag, 17. Juli 2003 12:29 Uhr +0100 Chris Withers [EMAIL PROTECTED] wrote: The problem is caused by calling len() on the indexes btrees. Instead a counter implemented btree.Length should be used in the future. Which Index types are currently guilty of this? I think all except ZCTextIndex. How about re-naming the column to Number of Documents Indexed and making sure this is actually what the indexes return. ?? I have a feeling that not all index types actually return the number of objects indexed. Can anyone confirm this? TextIndex in an older version returned the number of indexed words but this is fixed at least since 2.6. -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Chris Withers wrote at 2003-7-17 11:12 +0100: Has anyone noticed that the ZCatalog Indexes tab crawls if you have loads of objects indexed. My guess is that some types of index take way too long to figure out how many objects are indexed. Anyone know which index types those could be? The one that provide the correct number of indexed objects (rather than just the number of indexed terms). Because the same object can be indexed under several terms, determining the number of indexed objects requires to build the union of all the index values. This almost surely has quadratic (worst case) runtime characteristics. BTW, would anyone object if I removed that object count, since it's not often very useful... You probably should replace it with the size of the index (i.e. the number of index terms). Formerly, the index overview displayed this information but under a buggy # objects title. Someone fixed this for most indexes, they now show the number of objects but at a high price. I suggest to change the title to # index terms and revert for the indexes to the old behaviour. Others pointed out, that also the size determination for an index may be expensive. However, it is at most linear in the number (rather than quadratic) and all recently created indexes now use BTrees.Length to maintain their size (which gives constant time). Having a feeling how large an index is is valuable information. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Actually I regard the current behavior as a feature. Using a stopwatch and a slide-rule I can estimate to within 100 objects, how many values are indexed in a catalog by measuring the time it takes to draw the indexes page. Please do not remove this most valued feature! -Casey On Thursday 17 July 2003 04:35 pm, Dieter Maurer wrote: Chris Withers wrote at 2003-7-17 11:12 +0100: Has anyone noticed that the ZCatalog Indexes tab crawls if you have loads of objects indexed. My guess is that some types of index take way too long to figure out how many objects are indexed. Anyone know which index types those could be? The one that provide the correct number of indexed objects (rather than just the number of indexed terms). Because the same object can be indexed under several terms, determining the number of indexed objects requires to build the union of all the index values. This almost surely has quadratic (worst case) runtime characteristics. BTW, would anyone object if I removed that object count, since it's not often very useful... You probably should replace it with the size of the index (i.e. the number of index terms). Formerly, the index overview displayed this information but under a buggy # objects title. Someone fixed this for most indexes, they now show the number of objects but at a high price. I suggest to change the title to # index terms and revert for the indexes to the old behaviour. Others pointed out, that also the size determination for an index may be expensive. However, it is at most linear in the number (rather than quadratic) and all recently created indexes now use BTrees.Length to maintain their size (which gives constant time). Having a feeling how large an index is is valuable information. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
--On Donnerstag, 17. Juli 2003 18:22 Uhr -0400 Casey Duncan [EMAIL PROTECTED] wrote: Actually I regard the current behavior as a feature. Using a stopwatch and a slide-rule I can estimate to within 100 objects, how many values are indexed in a catalog by measuring the time it takes to draw the indexes page. Please do not remove this most valued feature! I agree but the current implementation sux. Switching to a counter based solution would solve the problem. The only problem I see is to keep the code fully backward compatible. -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Indexes tab crawls...
Andreas Jung wrote I agree but the current implementation sux. Switching to a counter based solution would solve the problem. The only problem I see is to keep the code fully backward compatible. if there's no counter present: create one, do a count of the docs, initialise the counter display counter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and not registered clasees
On Tue, 2002-06-04 at 15:38, Vincenzo Di Somma wrote: Hi all, I have a product with a main class (registeredin the __init__.py) and some utility classes I don`t want to register, does the ZCatalog have problem finding istances of not registered classes ? I`m not able to index them. I`ve found the solution, is not a registration but a traverse problem. -- Vincenzo Di Somma - Responsabile Ricerca e Sviluppo - Icube S.r.l. Sede: Via Ridolfi 15 - 56124 Pisa (PI), Italia E-mail: [EMAIL PROTECTED] WWW: www.icube.it Tel:(+39) 050 97 02 07 Fax: (+39) 050 31 36 588 signature.asc Description: This is a digitally signed message part
Re: [Zope-dev] ZCatalog: hiding search results from unauthorized users - hack
The problem with this solution is that you must wake up every object found, thus negating the performance/memory advantaged of ZCatalog's lazy result sets. Since you said (in your other message) that the restriction is by role, couldn't you just index the roles allowed to view a given object and then filter on that index. The caveat with that is that you would need to reindex whenever the access settings changed. So, this would need to be done in such a way in your application so that the objects could be indexed when the security settings changed. This would make implicit (acquired) security changes difficult to handle. On the bright side, you wouldn't need to subclass ZCatalog, or bring all the results into memory and validate each one. Thats *extremely* expensive. All you would need to do is create a method or python script that returned a list of roles allowed to view an object, and then create a keyword index on this method/script. hth, Casey Igor Stroh wrote: Hi all, in case someone have same problem as me (see ZCatalog - hiding query results thread for more info) - I got a solution: - create a product that subclasses ZCatalog - in this product overwrite ZCatalog.getobject with def getobject(self, rid): foo obj = self.aq_parent.unrestrictedTraverse(self.getpath(rid),\ restricted=1) return obj - create a python script in your catalog (e.g. filterCat) with a parameter that reoresents the catalog itself (brains here): retval = [] for brain in brains: try: obj = container.getobject(brain.getRID()) retval.append(brain) except: pass return retval - adjust your catalog query reports, so that they call filterCat(_['catalogname']) instead of catalogname Now your users see only those hits in a query which apply to objects they are allowed to View. greetings, Igor ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog - hiding query results
Igor Stroh writes: That means, users that don't have the permission to View or Access Content Information can see the brains as well... You can look how the CMF (Content Management Framework) solves this problem. Look for allowedRolesAndUsers and the implicit query extension for users without special privileges (in the CatalogTool). Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ZCatalog
what do you mean by fuzzy? Do you mean with wildcards, NL, or something else? -Original Message- From: Dirk Datzert [mailto:[EMAIL PROTECTED]] Sent: Thursday, December 06, 2001 1:03 PM To: [EMAIL PROTECTED] Subject: [Zope-dev] ZCatalog Hi, I'm looking for a howto or a hint for fuzzy search in german and english with ZCatalog / CatalogQuery Can anybody sent me a URL ? Thanks Dirk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog
the search engine htdig has a fuzzy feature which automatically search for different endings or alternative spelling of a word, based on the ispell algorithm [EMAIL PROTECTED] schrieb: what do you mean by fuzzy? Do you mean with wildcards, NL, or something else? -Original Message- From: Dirk Datzert [mailto:[EMAIL PROTECTED]] Sent: Thursday, December 06, 2001 1:03 PM To: [EMAIL PROTECTED] Subject: [Zope-dev] ZCatalog Hi, I'm looking for a howto or a hint for fuzzy search in german and english with ZCatalog / CatalogQuery Can anybody sent me a URL ? Thanks Dirk ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ZCatalog
Personally, I am of the opinion that sending a Catalog raw queries is not
very useful for really powerful searching... hopefully this changes in the
future, but for now...
Since catalog doesn't have anything built-in that does this, other than
globbing vocabularies, you have to do this within your appliction. What you
want to do is hijack and rewrite user queries before sending them through a
catalog query... for example:
word = 'dancing'
newword = re.sub('ing$', '*', word)
#newword is now 'danc*' which
This would work with plural words as well, as long as you have a useful set
of patterns for word endings...
For alternate spellings, abbreviations, and synonyms you would need to
create lookup tables and indexed methods that took advantage of them for
text indexes. This works only for text indexes though...
In summary, use query rewrites for word endings and auto-wildcarding (the
work is done at search time)... and use methods that weight and translate
your text in your product code (the work is done at index time)...
Sean
-Original Message-
From: Dirk Datzert [mailto:[EMAIL PROTECTED]]
Sent: Thursday, December 06, 2001 1:38 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: [Zope-dev] ZCatalog
the search engine htdig has a fuzzy feature which automatically search for
different endings or alternative spelling of a word, based on the ispell
algorithm
[EMAIL PROTECTED] schrieb:
what do you mean by fuzzy? Do you mean with wildcards, NL, or something
else?
-Original Message-
From: Dirk Datzert [mailto:[EMAIL PROTECTED]]
Sent: Thursday, December 06, 2001 1:03 PM
To: [EMAIL PROTECTED]
Subject: [Zope-dev] ZCatalog
Hi,
I'm looking for a howto or a hint for fuzzy search in german and english
with ZCatalog / CatalogQuery
Can anybody sent me a URL ?
Thanks
Dirk
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ZCatalog becoming empty when using Update in conjunction with ZPatterns
Are you using any virtual hosting setup of any kind? If so, switching from
using a base class of CatalogAware.CatalogAware to
CatalogPathAware.CatalogAware might solve this problem...
Sean
-Original Message-
From: Jean Jordaan [mailto:[EMAIL PROTECTED]]
Sent: Monday, December 03, 2001 1:59 AM
To: [EMAIL PROTECTED]
Subject: [Zope-dev] ZCatalog becoming empty when using Update in
conjunction with ZPatterns
Hi all
I had a ZCatalog with about 3500 DataSkin instances indexed. Then
I started to index instances of a regular ZClass as well. Then I
added another index, to catalog an additional property of this
ZClass, and hit Update Catalog.
When Zope returned from updating, there were no records in the
Catalog, though the Vocabulary was intact. I see that I cannot
search for the DataSkin metatypes using the Find Objects
ZCatalog tab: they don't show up in the Find objects of type
field. So perhaps this is why updating failed.
However, now the Catalog is not accepting any entries. I originally
did the cataloging using this kind of loop on all the classes I
wanted indexed::
ids = container.Roundup.Issues.defaultRack.getPersistentItemIDs()
for id in ids:
object = container.Roundup.Issues.defaultRack.getItem(id)
Catalog.catalog_object( object
, string.join( object.getPhysicalPath()
, '/'))
When I try to run that again, I still get There are no objects in
the Catalog. upon return. When I run it using a newly created catalog
('testCatalog') instead, it works fine.
Could anyone perhaps explain what's happening here?
Regards,
Jean
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog: hiding search results from unauthorized users - hack
On Fri, 16 Nov 2001 13:19:20 + Steve Alexander [EMAIL PROTECTED] wrote: Now your users see only those hits in a query which apply to objects they are allowed to View. However, you will be loading each object that is a potential query result into memory for every query. That could be a lot of objects. I didn't say it's perfect, it works however... BTW, it's not the View permission but Access content information... obviously :) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog - hiding query results
Igor Stroh wrote:
Hi all,
I don't know if it's the right list to post to, but I have the following
problem:
I have several objects (documents, folders etc) that are accessible only
by a certain user role, this objects are cataloged. Now if I query the
catalog the brains of these objects are returned correctly, but to _all_
users that issue a query. That means, users that don't have the permission
to View or Access Content Information can see the brains as well...
I tried to filter the result set by converting the brains into real
objects (brain.getObject) in an external method (I thought, this way I
should be able to exclude unauthorized users by adding the
skip_unauthorized to the dtml-in), but that doesn't work
since there are brains that are actually NoBrainer instances...
Does anyone have an idea of how I could solve this problem?
Actually I thought this kind of information hiding is supported by basic
ZCatalog machinery, but now it looks like I'll have to hack a
workaround...
Any help greatly appreciated.
Igor,
I had exactly the same problem and solved it this way:
1. define a method 'catalog_permission' in the classes of the objects
that will be indexed:
from AccessControl.PermissionRole import rolesForPermissionOn
class someClass(Folder):
def catalog_permission(self):
return: Liste der roles, die die permissions 'View',
'Access Content Information sowie view archivDoc haben
l1 = rolesForPermissionOn('View', self)
if type(l1) == type(''):
l1 = [l1, ]
l2 = rolesForPermissionOn('Access contents information', self)
if type(l2) == type(''):
l2 = [l2, ]
res = []
for x in l1:
if x in l2:
res.append(x)
return res
2. define a new Catalog class, with a newly defined method
searchResults:
from Products.ZCatalog.ZCatalog import ZCatalog
from AccessControl import getSecurityManager
class ACatalog(ZCatalog):
def searchResults(self, REQUEST=None, used=None, **kw):
roles = getSecurityManager().getUser().getRoles()
if REQUEST is not None:
REQUEST['catalog_permission'] = roles
elif kw != {}:
kw['catalog_permission'] = roles
else:
self.REQUEST['catalog_permission'] = roles
return ZCatalog.searchResults(self, REQUEST, used, **kw)
3. Add a keyword index 'catalog_permission' to the ACatalog instance.
(Ok, that could be done automatically in ACatalog.__init__ , but I was
too lazy to write that...)
A more reliable implemetation should make sure that only those objects
are indexed, which define a method catalog_permission. Or
Catalog.catalogObject could be overloaded to automatically build the
information to be thrown into the catalog_permission index.
Abel
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog searching for missing values
We've installed Kavio's CatalogQuery product and are very happy with it. Haven't looked at the ZOQLMethod from iuveno yet, but both look like great steps in the right direction. Thanks. I actually need to look at Casey's code and see what I can reuse. The efficiency of my version is not that great yet, but much more flexible. So I want to take Casey's code and optimize mine a little. BTW, I just added a mailing list and a poll for ZOQL. Please go to http://demo.iuveno-net.de/iuveno/Products/ZOQLMethod and vote in the poll, so I can get an estimate of the general interest. I have one question: is it possible using a normal catalog query or Kavio's catalog query to check if a value is not set? The representation returned is Missing.Value, but I seem to be unable to say, e.g, 'Description == Missing.Value' etc. How hard is it to add such functionality? You mean, whether the system checks, if a property exists at all or is set to None? I think both would be no problem. 1. Property exists or not: I could support a function, like: exists(Property) Example: SELECT * FROM Test WHERE exists(Property); 2. Property is set to None: Simply use equal: Property == None Note: That might work already? I have to check... Example: SELECT * FROM Test WHERE Property == None; Regards, Stephan -- Stephan Richter CBU - Physics and Chemistry Student Web2k - Web Design/Development Technical Project Management ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ZCatalog searching for missing values
Having an exists function would be great. I was trying to do the same using CatalogQuery, but I found no way to check for Missing.Value, which is the repr() of what's in the catalog's metadata field for objects that don't have that attribute/function. I also ran into problems doing more complex or'ing and and'ing, and I assume we'll have better luck with ZOQLMethod for that. I just tried out ZOQLMethod as well. Couldn't get it to work, but it looks like really solid craftsmanship. I like the user-friendly way of selecting base object and the complete syntax. :) Some questions: 1. I couldn't get any query to work. I keep getting exceptions.TypeError argument 2 to map() must be a sequence object Here's an example query that I've tried SELECT id WHERE title == '' RECURSIVE; 2. Is the ZCatalog searching implemented? How do I activate it? 3. Do you have any plans to implement JOIN? The exists function would be very cool. Keep up the great work. Hope this helps flush out some features :) Bye, -- Bjorn Stephan wrote: [...] I actually need to look at Casey's code and see what I can reuse. The efficiency of my version is not that great yet, but much more flexible. So I want to take Casey's code and optimize mine a little. BTW, I just added a mailing list and a poll for ZOQL. Please go to http://demo.iuveno-net.de/iuveno/Products/ZOQLMethod and vote in the poll, so I can get an estimate of the general interest. I have one question: is it possible using a normal catalog query or Kavio's catalog query to check if a value is not set? The representation returned is Missing.Value, but I seem to be unable to say, e.g, 'Description == Missing.Value' etc. How hard is it to add such functionality? You mean, whether the system checks, if a property exists at all or is set to None? I think both would be no problem. 1. Property exists or not: I could support a function, like: exists(Property) Example: SELECT * FROM Test WHERE exists(Property); 2. Property is set to None: Simply use equal: Property == None Note: That might work already? I have to check... Example: SELECT * FROM Test WHERE Property == None; Regards, Stephan -- Stephan Richter CBU - Physics and Chemistry Student Web2k - Web Design/Development Technical Project Management ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ZCatalog searching for missing values
Ahhh... Got it working now. Thanks. :) Not sure I know how to activate ZCatalog searching though, or if it is on all the time; the example only mentions ZCatalog once when it deletes it. What determines if a search is a raw object-database search and when it uses the ZCatalog? Always returning the objects is going to a bit slow, though. Couldn't it just return the Pluggable Brains if you search the ZCatalog; or at least have an option to only return them? Bye, -- Bjorn -Original Message- From: Stephan Richter [mailto:[EMAIL PROTECTED]] Sent: Monday, October 08, 2001 23:33 To: Bjorn Stabell Cc: [EMAIL PROTECTED] Subject: RE: [Zope-dev] ZCatalog searching for missing values I just tried out ZOQLMethod as well. Couldn't get it to work, but it looks like really solid craftsmanship. I like the user-friendly way of selecting base object and the complete syntax. :) Some questions: 1. I couldn't get any query to work. I keep getting exceptions.TypeError argument 2 to map() must be a sequence object Solved. See below. Here's an example query that I've tried SELECT id WHERE title == '' RECURSIVE; You can only select meta types (this is by design; I just don't want to deal with brains and all that stuff right now.)!! Did you see the help? It has a long example in there as well. But there is another bug. You have to specify FROM right now. I will fix that. Okay, is fixed for the next release. So, SELECT DTML Method FROM . WHERE title == '' RECURSIVE; should work. 2. Is the ZCatalog searching implemented? How do I activate it? Yes, see the Rather Lengthy Example. 3. Do you have any plans to implement JOIN? No, for the reasons given above. You select only objects, not attributes. BUT, patches are always welcomed!!! :-) The exists function would be very cool. I will look into this. Hopefully it will be not too hard to implement. Keep up the great work. Hope this helps flush out some features :) I hope I will. :-) Yes it does. If you and all the other Zope people could write me some arguments to support attribute selection, then I will look into it (but it will be not that easy). Regards, Stephan -- Stephan Richter CBU - Physics and Chemistry Student Web2k - Web Design/Development Technical Project Management ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog API
E. Seifert wrote: Hi list, as I'm not very familiar with the internals of ZCatalog I have a question to all you API experts: Why does ZCatalog.py (under Zope 2.4.1) specify a 'addIndex' permission for a method called 'manage_delIndexes' although there is no such method? The only method I could find to delete Indexes is 'manage_deleteIndex'. Can anyone comment on this? Thanks for pointing this out. There was some cruft in ZCatalog.py, which has now has been cleaned up. The changes are in CVS, and will be in the next release of Zope. The method manage_deleteIndex is still there, but is deprecated, and will issue a warning the first time it is called. You can use manage_delIndex instead. Both are protected by the addIndex permission. -- Steve Alexander Software Engineer Cat-Box limited ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog: path summary indices not generated
- Original Message - From: Shane Rowatt [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 03, 2001 00:54 Subject: [Zope-dev] ZCatalog: path summary indices not generated Unfortunately I tried the same with the 'path' index by adding the following to DTMLMethod.py def getPath(self): Get path return getPath(self) def path(self): Get path return join(self.getPhysicalPath(), /) but the 'path' index only works it is a FieldIndex. When it's a PathIndex I get the value of None for all cataloged items. Shane, you don't have to provide special path() to your objects. The PathIndex works a bit different from the other indexes because it does not look for an attribute or method with a name equal to the name of your PathIndex. So how do PathIndexes work ? - ZCatalog calls PathIndex.index_object() for all objects to be cataloged. - index_object() determines the physical path the object and indexes this result inside the PathIndex data structure. We have not seen necessity to provide support for a user-defined hook. If you have some use cases let me know. Hope this helps ;-) Andreas ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog: path summary indices not generated
On Wednesday 03 October 2001 12:54 am, Shane Rowatt allegedly wrote: Zope Version: 2.4.1 on linux. When I tried to add a ZCatalog followed by finding objects to index using the default indices provided (path, summary, id, title etc), the 'path' and 'summary' indices are never generated. The 'summary' index is always an empty string and the 'path' index is 'None'. Summary is not defined (as you found) for most plain objects. CatalogAware defines it, but unfortunately without stripping tags. You might check out my DTMLDocumentExt code which renders the document and strips the html tags. It is an extension of some code Dieter posted online a while back. You can find it at: http://www.zope.org/Members/Kaivo/DTMLDocumentExt [snip] Surely someone else has come across these problems before using this basic catalog stuff?? Yup 8^) Shane Rowatt Astracon Inc. hth /---\ Casey Duncan, Sr. Web Developer National Legal Aid and Defender Association [EMAIL PROTECTED] \---/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog API
E. Seifert wrote: Hi list, as I'm not very familiar with the internals of ZCatalog I have a question to all you API experts: Why does ZCatalog.py (under Zope 2.4.1) specify a 'addIndex' permission for a method called 'manage_delIndexes' although there is no such method? The only method I could find to delete Indexes is 'manage_deleteIndex'. Can anyone comment on this? Well, considering that there's these methods in ZCatalog: manage_addColumn manage_delColumns manage_addIndex manage_deleteIndex ... manage_deleteIndex is obviously the odd one out. Especially as it allows you to delete several indexes in one go. I suggest it be renamed manage_delIndices or manage del_Indexes. manage_clearIndex also operates on multiple indexes, so that should become manage_clearIndexes. Funnily enough, the ZCatalog regression test at ./regressiontests/regressionCatalogTiming.py uses this call: manage_delIndexes([x]) which supports renaming manage_deleteIndex to manage_delIndexes. However, while dtml/catalogIndexes.dtml uses manage_delIndexes, ./dtml/manageIndex.dtml uses manage_deleteIndex. manage_reindexIndex should also become manage_reindexIndexes, as it requires a sequence of ids. Also, manage_reindexIndex doesn't have an associated permission. There's a few more place this touches, although all the changes are in ZCatalog.py and manageIndex.dtml. Congratulations, you've found a bug! This would have been more obvious if ZCatalog were using the new declarative security assertions, as the declaration would be right next to the method definition. Here's a first-draft patch, for comments etc. I haven't looked at the documentation of the API, so that might need changing too. -- Steve Alexander Software Engineer Cat-Box limited *** ZCatalog.py.orig --- ZCatalog.py *** *** 196,202 'manage_catalogView', 'manage_catalogFind', 'manage_catalogSchema', 'manage_catalogIndexes', 'manage_catalogAdvanced', 'manage_objectInformation', ! 'manage_catalogReindex', 'manage_catalogFoundItems', 'manage_catalogClear', 'manage_addColumn', 'manage_delColumns', 'manage_addIndex', 'manage_delIndexes', 'manage_main', --- 196,202 'manage_catalogView', 'manage_catalogFind', 'manage_catalogSchema', 'manage_catalogIndexes', 'manage_catalogAdvanced', 'manage_objectInformation', ! 'manage_reindexIndexes', 'manage_clearIndexes', 'manage_catalogReindex', 'manage_catalogFoundItems', 'manage_catalogClear', 'manage_addColumn', 'manage_delColumns', 'manage_addIndex', 'manage_delIndexes', 'manage_main', *** *** 422,430 RESPONSE.redirect(URL1 + '/manage_main?manage_tabs_message=Index%20Added') ! def manage_deleteIndex(self, ids=None, REQUEST=None, RESPONSE=None, URL1=None): ! del an index if not ids: return MessageDialog(title='No items specified', message='No items were specified!', --- 422,430 RESPONSE.redirect(URL1 + '/manage_main?manage_tabs_message=Index%20Added') ! def manage_delIndexes(self, ids=None, REQUEST=None, RESPONSE=None, URL1=None): ! del indexes if not ids: return MessageDialog(title='No items specified', message='No items were specified!', *** *** 436,444 if REQUEST and RESPONSE: RESPONSE.redirect(URL1 + '/manage_main?manage_tabs_message=Index%20Deleted') ! def manage_clearIndex(self, ids=None, REQUEST=None, RESPONSE=None, URL1=None): ! del an index if not ids: return MessageDialog(title='No items specified', message='No items were specified!', --- 436,444 if REQUEST and RESPONSE: RESPONSE.redirect(URL1 + '/manage_main?manage_tabs_message=Index%20Deleted') ! def manage_clearIndexes(self, ids=None, REQUEST=None, RESPONSE=None, URL1=None): ! clear indexes if not ids: return MessageDialog(title='No items specified', message='No items were specified!', *** *** 462,468 if obj is not None: self.catalog_object(obj, p, idxs=[name]) ! def manage_reindexIndex(self, ids=None, REQUEST=None, RESPONSE=None, URL1=None): Reindex indexes from a ZCatalog if not ids: return MessageDialog(title='No items specified', --- 462,468 if obj is not None: self.catalog_object(obj, p, idxs=[name]) ! def manage_reindexIndexes(self, ids=None, REQUEST=None, RESPONSE=None, URL1=None): Reindex indexes from a ZCatalog if not ids:
Re: [Zope-dev] zcatalog and versions
Thanks for the fast reply Casey. Casey Duncan wrote: On Thursday 27 September 2001 12:48 pm, Oliver Bleutgen allegedly wrote: Hi, I'm resending this to zope-dev because on zope nobody answered, it would be very nice if someone could step up with a small hint. Can somenone briefly explain what exactly gets locked in zope 2.3.3's catalog when it tries to index an object which is hold in a version? The whole catalog? Any Btree buckets (in the indexes) which get changed get locked, which can effectively lock out other changes to the catalog. This is a limitation of the way the indexes are implemented, for which, sadly, there is no easy fix. I found some answers which indicate that it has to do with the catalog when we see version lock errors where there shouldn't be any (from a naive POV). I would like to know how far reaching these problems are, because I'm currently rewriting an application, and I might be able to work around that. I would suggest that if any objects are reindexed in a version, that no cataloged objects should be reindexed in any other version until that version is saved. I hope I understand you correctly, but I'd say that if there's already a locked bucket, I've lost. There's no guarantee how long this particular version will stay uncommitted. I would also have to check anyhow whether there is a lock somewhere in the catalog's index. You could also get around that by deferring the indexing until the version is commited, but this will take some coding on your part. This seems easier - if I'm a little bit lax about when the indexing occurs. Like making the object only index/reindex/unindex itself if it's not in a version, and combining that with a nightly cronjob which reindexes all (non-versioned) documents. With some coding on your part you mean making Version.py more intelligent? Like instead of just doing commitVersion(s,'') doing the following: 1) search for objects which have been deleted in the version 2) search for objects which want to catalog themselves and are locked in version s 3) unindex the objects found in 1) 4) commit version 5) index/reindex objects found in 2) Where it's not clear to me how to prevent that 3) and 5) will not itself get versioned. Ultimately I think that ZCatalog should do this for you, or at least somehow let indexes have concurrent versioning (any volunteers?) I should say that I really don't grasp this ZODB voodoo, but I suspect that this will also be not too easy. AFAIK, the decision to write in a version (and in which) is taken deep down in zope's innards. I am thinking about writing a fishbowl proposal for ZCatalog upgrades sometime next month, and this is one potential problem areas to address there, especially as things like the CMF make it more ubiquitous If I don't misunderstand how versions work (not unlikely), it might be necessary in the end rather to improve versions than the zcatalog in order to remedy this specific problem. thanks again, oliver ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog problem: sort_on bug
Steve Alexander wrote: I'm getting a strange problem with ZCatalog, using python 2.1, Zope from CVS. I get all the results I expect with this: ul dtml-in Catalog(process_step=['start','mailed']) li dtml-subject_name; -- dtml-relationship_name; - dtml-rater_name; /dtml-in /ul But, I only get one subject_name's worth of results with this: ul dtml-in Catalog(process_step=['start','mailed'], sort_on='subject_name') li dtml-subject_name; -- dtml-relationship_name; - dtml-rater_name; /dtml-in /ul This smells like a BTrees bug to me, but I'm not sure. I'm looking into this closely now, but if anyone's seen this before, please speak up! More data: This gives partial results: dtml-in Catalog(process_step=['start','mailed'], sort_on='subject_name') This gives full results: dtml-in Catalog(process_step=['start','mailed'], sort_on='subject_name')[:] This gives full results: dtml-in Catalog(process_step=['start','mailed'], sort_on='subject_name') sort=subject_name -- Steve Alexander Software Engineer Cat-Box limited ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog problem: sort_on bug
Steve Alexander wrote: This gives partial results: dtml-in Catalog(process_step=['start','mailed'], sort_on='subject_name') ...because this returns a LazyCat instance, for which len() is broken. This gives full results: dtml-in Catalog(process_step=['start','mailed'], sort_on='subject_name')[:] ...because this is a list. This gives full results: dtml-in Catalog(process_step=['start','mailed'], sort_on='subject_name') sort=subject_name ...because this is a LazyMap instance, for which len() works. Patch coming up soon... -- Steve Alexander Software Engineer Cat-Box limited ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog problem: PATCH
Steve Alexander wrote:
Patch coming up soon...
Patch against Catalog.py, from CVS:
*** lib/python/Products/ZCatalog/Catalog.py.original
--- lib/python/Products/ZCatalog/Catalog.py.patched
***
*** 673,679
if (type(so) is type('') and
lower(so) in ('reverse', 'descending')):
r.reverse()
! r=LazyCat(map(lambda i: i[1], r), len(r))
return r
--- 673,681
if (type(so) is type('') and
lower(so) in ('reverse', 'descending')):
r.reverse()
! r=map(lambda i: i[1], r)
! r=LazyCat(r, reduce(lambda x,y: x+len(y), r, 0))
!
return r
I'd use a list comprehension instead of a map(lambda...) if I thought it
would get past Jim ;-)
--
Steve Alexander
Software Engineer
Cat-Box limited
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog out of Zope
Paulo M. Goncalves wrote: I'm a Zope and Python newbie and I'm trying to work with Zope's ZCatalog in a Product coded in python. Can anyone give some intro or point to some links with info in this subject. Maybe have a look at the SquishSite.py file in Squishdot? The search method of the SquishSite class does some ZCatalog searching... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog out of Zope
Hi Paulo.. I can't give you any real specific advice on where to get info besides the Zope Book, but you may want to take a look at the ZCatalog.py and Catalog.py source code (in the ZCatalog product). Also, the (always forgotten) help system has documentation on the Catalog's interfaces. - C On Sat, 30 Jun 2001 20:47:02 +0100 Paulo M. Goncalves [EMAIL PROTECTED] wrote: Hi, I'm a Zope and Python newbie and I'm trying to work with Zope's ZCatalog in a Product coded in python. Can anyone give some intro or point to some links with info in this subject. Thanks, Paulo --- Paulo Marques Goncalves CDFtel, Lda Ph. No: +351.22.3389843 Rua Goncalo Cristovao, 347 - Sala 205 FAX No: +351.22.3389845 4000-270 PORTO E-mail: mailto:[EMAIL PROTECTED]Portugal --- ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog out of Zope
On Sat, 30 Jun 2001 20:47:02 +0100, you wrote: I'm trying to work with Zope's ZCatalog in a Product coded in python. Can anyone give some intro or point to some links with info in this subject. There is a ZCatalog tutorial on http://www.zope.org/Documentation/How-To/ZCatalogTutorial/ The ZCatalog class is documented in the API Reference on http://www.zope.org/Members/michel/ZB/AppendixB.dtml Perhaps these are useful examples: http://www.zope.org/Members/Bill/Products/KnowledgeKit http://www.zope.org/Members/NIP/ZMailIn I found these using the search engine and I didn't check if they even use ZCatalog. Here is some python coding that shows how to use the catalog class outside of Zope: http://www.zope.org/Members/kelcmab3/catalog_out_of_zope This is a message with some code snippets from someone who tried to do the same: http://lists.zope.org/pipermail/zope/2000-April/023705.html Regards, René Pijlman ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope] Re: [Zope-dev] Zcatalog bloat problem (berkeleydb is a solution?)
Chris McDonough wrote: This purpose aligns well with those of the ArmoredCatalog proposal as well.. see http://dev.zope.org/Wikis/DevSite/Proposals/ArmoredCatalog . But even using such a lazy catalog awareness, you might get into trouble. Using the ZCatalog's find objects function, I hit the limits of my Linux box: 640 MB RAM were not enough... This should not happen. :-( Just to add another data point, we're still having issues if we catalog-as-you go when trying to recreate our mailing list archives in Zope. As I understand it, the guys managed to get it to work by importing 28,000 odd message and then indexing, rather than indexing as each one was added. This was using Zope 2.3.2, should it be expected? cheers, Chris PS: Andy D was going to post this but he went home ill, I don't think that was ZCatalog related ;-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
