Re: [Zope-dev] 27 million objects.

2001-04-27 Thread Erik Enge
On Thu, 26 Apr 2001, Dieter Maurer wrote: Are the imported object CatalogAware? They are. The old (pre 2.3.1) catalog implementation was know not to be very storage friendly. If a significant portion of the catalog indexes would be affected by imports, then you would see a quadratic

Re: [Zope-dev] 27 million objects.

2001-04-27 Thread Erik Enge
On Thu, 26 Apr 2001, Chris McDonough wrote: This level of growth doesn't seem like a sane level of growth... what Zope version are you using? Zope 2.3.1b1 Someone told me that ZEO and bulk-adding could be a thing to look at... Isn't bulk-adding what you're doing now? It is, but I'm

Re: [Zope-dev] 27 million objects.

2001-04-27 Thread Chris Withers
Erik Enge wrote: The old (pre 2.3.1) catalog implementation was know not to be very storage friendly. If a significant portion of the catalog indexes would be affected by imports, then you would see a quadratic storage increase. I'm using Zope 2.3.1b1 so that shouldn't be a problem?

Re: [Zope-dev] 27 million objects.

2001-04-27 Thread Erik Enge
On Fri, 27 Apr 2001, Chris Withers wrote: Erik Enge wrote: I'm using Zope 2.3.1b1 so that shouldn't be a problem? Yes, it will be. [...] So the bug in Zope 2.3.1b1 which makes the ZODB grow dramatically is gone in Zope 2.3.2b2? ___

Re: [Zope-dev] 27 million objects.

2001-04-27 Thread Chris Withers
Erik Enge wrote: On Fri, 27 Apr 2001, Chris Withers wrote: Erik Enge wrote: I'm using Zope 2.3.1b1 so that shouldn't be a problem? Yes, it will be. [...] So the bug in Zope 2.3.1b1 which makes the ZODB grow dramatically is gone in Zope 2.3.2b2? Not so much a bug as a

Re: [Zope-dev] 27 million objects.

2001-04-27 Thread Chris McDonough
Erik Enge wrote: On Fri, 27 Apr 2001, Chris Withers wrote: Erik Enge wrote: I'm using Zope 2.3.1b1 so that shouldn't be a problem? Yes, it will be. [...] So the bug in Zope 2.3.1b1 which makes the ZODB grow dramatically is gone in Zope 2.3.2b2? Not so much a bug

Re: [Zope-dev] 27 million objects.

2001-04-26 Thread Erik Enge
On Thu, 5 Apr 2001, Michael R. Bernstein wrote: I'm trying to find out of there is a point where you start getting non-linear performance penalties for additional objects (storing, retreiving, or indexing). I've just finished adding a somewhat small number of objects: 5000. For every 1000th

Re: [Zope-dev] 27 million objects.

2001-04-26 Thread Chris McDonough
Erik Enge wrote: On Thu, 5 Apr 2001, Michael R. Bernstein wrote: I'm trying to find out of there is a point where you start getting non-linear performance penalties for additional objects (storing, retreiving, or indexing). I've just finished adding a somewhat small number of

Re: [Zope-dev] 27 million objects.

2001-04-26 Thread Dieter Maurer
Erik Enge writes: I've just finished adding a somewhat small number of objects: 5000. For every 1000th object, the Data.fs seemed to grow to about 900MB; that's when things started going slow, in a non-linear fashion (this is more a hunch than something I payed much attention to). I

Re: [Zope-dev] 27 million objects.

2001-04-09 Thread Erik Enge
On Thu, 5 Apr 2001, Michael R. Bernstein wrote: I'm trying to find out of there is a point where you start getting non-linear performance penalties for additional objects (storing, retreiving, or indexing). I don't know, but I feel that is the case. Actually, I know it is the case, but I

Re: [Zope-dev] 27 million objects.

2001-04-09 Thread Chris Withers
Zopista wrote: Hear, hear. The cost of the incremental cataloguing is horrific. One of our developers when we started with Zope a year ago wrote a great script that imported a bunch, restarted zope, packed it, restarted it, imported a bunch more to get optimal performance. We dont do it

Re: [Zope-dev] 27 million objects.

2001-04-09 Thread Andy McKay
ernstein" [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Monday, April 09, 2001 7:31 AM Subject: Re: [Zope-dev] 27 million objects. Zopista wrote: Hear, hear. The cost of the incremental cataloguing is horrific. One of our developers when we started with Zope a year ago wrote a great script that

Re: [Zope-dev] 27 million objects.

2001-04-09 Thread Andy McKay
hael R. Bernstein" [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Monday, April 09, 2001 10:52 AM Subject: Re: [Zope-dev] 27 million objects. Andy McKay wrote: As I said this was a year ago... but still incremental cataloging is very expensive. How come? I always thought this was one of

Re: [Zope-dev] 27 million objects.

2001-04-09 Thread Chris Withers
Andy McKay wrote: Any cataloguing and un-cataloguing of an object is expensive, c'mon you are changing all the indices, vocabulary and so on. Yup... You never notice it normally for 1 - 10 things, but run an import script of 1 and catalog each object as it gets added (rather than

Re: [Zope-dev] 27 million objects.

2001-04-09 Thread Michael R. Bernstein
Andy McKay wrote: Any cataloguing and un-cataloguing of an object is expensive, c'mon you are changing all the indices, vocabulary and so on. You never notice it normally for 1 - 10 things, but run an import script of 1 and catalog each object as it gets added (rather than all of them

Re: [Zope-dev] 27 million objects.

2001-04-09 Thread Andy McKay
Andy McKay wrote: Any cataloguing and un-cataloguing of an object is expensive, c'mon you are changing all the indices, vocabulary and so on. You never notice it normally for 1 - 10 things, but run an import script of 1 and catalog each object as it gets added (rather than all of

Re: [Zope-dev] 27 million objects.

2001-04-09 Thread Sin Hang Kin
Concerning Zcatalog: 1. Search of vocabulary is fully matched, partial search, trancate search or fuzzy search will be useful for searcher. 2. If we catalog all things in one catalog, then we need to make filter when we want to search on one particular portation. 3. I would like to suggest :

[Zope-dev] 27 million objects.

2001-04-05 Thread Erik Enge
Hi, Michael. Sorry for taking so long getting back to you. The programmer solving our problems with the post codes has solved it in a different way than what I would've done (his method is way superior), so we're not ending up adding all addresses as Zope Objects. Therefore, I don't have any

Re: [Zope-dev] 27 million objects.

2001-04-05 Thread Michael R. Bernstein
Erik Enge wrote: The programmer solving our problems with the post codes has solved it in a different way than what I would've done (his method is way superior), so we're not ending up adding all addresses as Zope Objects. Oh well. Does anyone else have any setups that store truly massive

Re: [Zope-dev] 27 million objects.

2001-04-05 Thread Andy McKay
ik Enge" [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Thursday, April 05, 2001 7:24 AM Subject: Re: [Zope-dev] 27 million objects. Erik Enge wrote: The programmer solving our problems with the post codes has solved it in a different way than what I would've done (his method is way s