Re: [Zope-dev] catalog performance: query plan

2008-11-10 Thread Lennart Regebro
On Sun, Nov 9, 2008 at 19:58, Roché Compaan [EMAIL PROTECTED] wrote: Since I'm in full agreement that we need to fix indexes that are problematic, I started doing some benchmarks on the large data set that gave us so many headaches. It is probably not surprising that the more complex indexes

Re: [Zope-dev] catalog performance: query plan

2008-11-10 Thread Lennart Regebro
On Sun, Nov 9, 2008 at 22:29, Matt Hamilton [EMAIL PROTECTED] wrote: Lennart Regebro regebro at gmail.com writes: I would be interested in seeing a bunch of Gurus sit down at some sprint and trying to come up with a catalog engine that is incremental and uses query plans. There is no reason

Re: [Zope-dev] catalog performance: query plan

2008-11-10 Thread Martin Aspeli
Hi Tres, Index Name |Type |Avg Time |Calls/second == object_implements|KeywordIndex |0.2172234| 4.6 This is clearly not the same issue as the other KeywordIndexes: in fact, I am astonished

Re: [Zope-dev] catalog performance: query plan

2008-11-10 Thread Roché Compaan
On Mon, 2008-11-10 at 18:38 +0200, Hedley Roos wrote: Kinda pointless for me to continue since this is turning into a Plone-specific discussion on zope-dev. But at least the whole exercise has forced us to look in detail into how all these indexes affect performance with a zodb with many many

Re: [Zope-dev] catalog performance: query plan

2008-11-10 Thread Roché Compaan
On Mon, 2008-11-10 at 11:08 -0500, Tres Seaver wrote: Index Name |Type |Avg Time |Calls/second == object_implements|KeywordIndex |0.2172234| 4.6 This is clearly not the same issue as the

Re: [Zope-dev] catalog performance: query plan

2008-11-10 Thread Laurence Rowe
Lennart Regebro wrote: On Sun, Nov 9, 2008 at 19:58, Roché Compaan [EMAIL PROTECTED] wrote: Since I'm in full agreement that we need to fix indexes that are problematic, I started doing some benchmarks on the large data set that gave us so many headaches. It is probably not surprising that the

Re: [Zope-dev] catalog performance: query plan

2008-11-09 Thread Matt Hamilton
Lennart Regebro regebro at gmail.com writes: I would be interested in seeing a bunch of Gurus sit down at some sprint and trying to come up with a catalog engine that is incremental and uses query plans. There is no reason that would not be stupidly fast. :) We can then make a new catalog

Re: [Zope-dev] catalog performance: query plan

2008-11-09 Thread Roché Compaan
On Mon, 2008-10-27 at 11:32 -0500, Alan Runyan wrote: I agree with Tres. A lot more can be done with Indexes and Catalog without caching. The most exiciting development in Catalog optimizations comes out Jarn. Helge Tesdal (iirc) did a buncha work at a RDBMS company when he was in

Re: [Zope-dev] catalog performance: query plan

2008-11-06 Thread Lennart Regebro
On Mon, Oct 27, 2008 at 17:32, Alan Runyan [EMAIL PROTECTED] wrote: I agree with Tres. A lot more can be done with Indexes and Catalog without caching. The most exiciting development in Catalog optimizations comes out Jarn. Helge Tesdal (iirc) did a buncha work at a RDBMS company when he

[Zope-dev] catalog performance: query plan

2008-10-27 Thread Alan Runyan
I agree with Tres. A lot more can be done with Indexes and Catalog without caching. The most exiciting development in Catalog optimizations comes out Jarn. Helge Tesdal (iirc) did a buncha work at a RDBMS company when he was in college. He has a protoype of a query plan for ZCatalog.

Re: [Zope-dev] Catalog performance

2003-09-21 Thread John Barratt
[EMAIL PROTECTED] wrote: Just my 2 cents observation... I ran this code and monitored the page Cache extreme detail in ZMI ControlPanel DebugInfo. With this method, the object was not loaded. However the intermediate objects that the unrestrictedTraverse() passed by were loaded into memory. e.g.

[Zope-dev] Catalog performance

2003-09-18 Thread zhimin
John Barratt wrote: If you can't use catalog metadata as Seb suggests (eg. you are actually accessing many attributes, large values, etc.) and if indeeed memory is the problem (which seems likely) then you can ghostify the objects that were ghosts to begin with, and it will save memory

Re: [Zope-dev] Catalog performance

2003-09-12 Thread Chris Withers
John Barratt wrote: Chris Withers wrote: John Barratt wrote: docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.aq_parent.unrestrictedTraverse(doc.getPath()) was_ghost = obj._p_changed is None value = obj.attr if was_ghost:obj._p_deactivate()

Re: [Zope-dev] Catalog performance

2003-09-11 Thread John Barratt
Chris Withers wrote: John Barratt wrote: docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.aq_parent.unrestrictedTraverse(doc.getPath()) was_ghost = obj._p_changed is None value = obj.attr if was_ghost:obj._p_deactivate() Bear in mind though,

[Zope-dev] Catalog performance

2003-09-10 Thread Nguyen Quan Son
Hi, I have a problem with performance and memory consumption when trying to do some statistics, using following code: ... docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.getObject() value = obj.attr ... With about 10.000 documents this Python

Re: [Zope-dev] Catalog performance

2003-09-10 Thread Romain Slootmaekers
Nguyen Quan Son wrote: Hi, I have a problem with performance and memory consumption when trying to do some statistics, using following code: ... docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.getObject() value = obj.attr ... With about 10.000

Re: [Zope-dev] Catalog performance

2003-09-10 Thread John Barratt
Max M wrote: Nguyen Quan Son wrote: Hi, I have a problem with performance and memory consumption when trying to do some statistics, using following code: ... docs = container.portal_catalog(meta_type='Document', ...) for doc in docs: obj = doc.getObject() value = obj.attr

Re: [Zope-dev] Catalog performance - SOLVED

2003-09-10 Thread Nguyen Quan Son
with 1GB RAM. What's wrong with this code? Any suggestion is appreciated. From: John Barratt [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, September 10, 2003 6:41 PM Subject: Re: [Zope-dev] Catalog performance If you can't use catalog metadata as Seb suggests (eg. you are actually