Re: [Zope] "Pausing" ZCatalog and re-indexing later

2006-08-11 Thread Jonathan


- Original Message - 
From: <[EMAIL PROTECTED]>

To: 
Sent: Friday, August 11, 2006 1:10 PM
Subject: [Zope] "Pausing" ZCatalog and re-indexing later




Hi :-)

The system I am working on is using a ZCatalog with some indexes (over 10) 
for faster searching.


Unfortunatelly there are some use cases, where I have to add really many 
new objects to ZODB. It works, but very slowly. I suspect that this is 
because when I add an object to Folder (with _setObject) it gets indexed - 
couple thousands of objects multiplied by over 10 indexes = long.


Am I guessing right or am I completely lost? ;-)

If this is the probable reason - is there some way I could disable 
indexing temporarily and after adding all of the objects make it re-index 
the new ones? I think that even a full re-index would work faster than it 
works now...


I would be very grateful for any suggestions.


Another approach is to build/update a new ZCatalog in another .fs file (you 
can do this offline, on another box, etc), and then just swap in the new .fs 
file.  You can also create a mount point for a folder that contains nothing 
but the ZCatalog and then just swap in that (instead of swapping the entire 
Data.fs file).



hth

Jonathan


___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] "Pausing" ZCatalog and re-indexing later

2006-08-11 Thread Andreas Jung



--On 11. August 2006 17:10:29 + [EMAIL PROTECTED] wrote:



Hi :-)

The system I am working on is using a ZCatalog with some indexes (over
10) for faster searching.

Unfortunatelly there are some use cases, where I have to add really many
new objects to ZODB. It works, but very slowly.


What kind of objects?


I suspect that this is
because when I add an object to Folder (with _setObject) it gets indexed
- couple thousands of objects multiplied by over 10 indexes = long.


Consider using CatalogQueue to defer indexing.



Am I guessing right or am I completely lost? ;-)


No idea, you might investigate your bottleneck(s) using ZopeProfiler.


-aj

pgpD2fQSIOfZn.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


[Zope] "Pausing" ZCatalog and re-indexing later

2006-08-11 Thread krokodylek

Hi :-)

The system I am working on is using a ZCatalog with some indexes (over 10) for 
faster searching.

Unfortunatelly there are some use cases, where I have to add really many new 
objects to ZODB. It works, but very slowly. I suspect that this is because when 
I add an object to Folder (with _setObject) it gets indexed - couple thousands 
of objects multiplied by over 10 indexes = long.

Am I guessing right or am I completely lost? ;-)

If this is the probable reason - is there some way I could disable indexing 
temporarily and after adding all of the objects make it re-index the new ones? 
I think that even a full re-index would work faster than it works now...

I would be very grateful for any suggestions.

Best regards,
Maciek




___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )