Re: [Zope-dev] ZCatalog, REQUEST, misc.
> > Well, that's good, except I thought you couldn't get rid of objects? > > Muhahaha. I got those little bastards... :) That's one way to fix it, I suppose. ;-) > > Yes, but not with 1,000,000 objects (see > > lib/python/ZCatalog/tests/testCatalog.py). It would be nice to have > > such a report. > > If I have time, I'll give you such a report this weekend. No promises, > though. No expectations... > Have you read 1984 by George Orwell? Maybe we could have a Hate Week for > the Catalog once in a while. Just absolutely find everything one hated > about it. Hate the Catalog Week ;) That's just about every week for me, although its gotten a lot less virulent lately. ;-) - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
On Thu, 17 May 2001, Chris McDonough wrote: > Well, that's good, except I thought you couldn't get rid of objects? Muhahaha. I got those little bastards... :) > Yes, but not with 1,000,000 objects (see > lib/python/ZCatalog/tests/testCatalog.py). It would be nice to have > such a report. If I have time, I'll give you such a report this weekend. No promises, though. > That sounds good! At least for the Catalog. Want to be a tester? Sure! Just let me know a couple of days (I'd love a weeks notice, but that might be stretching it?) notice and I'll have a go at it and give plenty of feedback (if there is any, of course). Have you read 1984 by George Orwell? Maybe we could have a Hate Week for the Catalog once in a while. Just absolutely find everything one hated about it. Hate the Catalog Week ;) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
Chris McDonough wrote: > > Well, some revision of 2.4 alpha will ship with drop-in indexes, so using it > would be wonderful. Lemme know as soon as it's in CVS and I'll see if I can get my Zope from source going on WinNT, I gave up the last tiem my need came close but Brian has solved the problem in the meantime (although I think I still owe him a How-To ;-) cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
Well, some revision of 2.4 alpha will ship with drop-in indexes, so using it would be wonderful. - C - Original Message - From: "Chris Withers" <[EMAIL PROTECTED]> To: "Chris McDonough" <[EMAIL PROTECTED]> Cc: "Erik Enge" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Thursday, May 17, 2001 11:38 AM Subject: Re: [Zope-dev] ZCatalog, REQUEST, misc. > Chris McDonough wrote: > > > > > > That sounds good! At least for the Catalog. Want to be a tester? ;-) > > I would, especially for drop-in indexes and AND keyword indexes :-) > > cheers, > > Chris > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
Chris McDonough wrote: > > > That sounds good! At least for the Catalog. Want to be a tester? ;-) I would, especially for drop-in indexes and AND keyword indexes :-) cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
> On Thu, 17 May 2001, Chris McDonough wrote: > > > I'd be curious to know how long a query that involves only a single > > field index takes, and how long a query that involves only a single text > > index takes... does each take a roughly equivalent amount of time? > > I might be able to check that for you later tonight or during the > weekend. What I did to solve my problems was to throw a couple of objects > around so that I now only have about 50k objects in the Catalog. > > Boy it's fast. I like! :) Well, that's good, except I thought you couldn't get rid of objects? > > Or is one much faster than the other? If one is not much faster than > > the other, it's a Catalog issue. If one *is* much faster than the > > other, it's an Index issue. > > Have you guys at DC not done testing on this? Yes, but not with 1,000,000 objects (see lib/python/ZCatalog/tests/testCatalog.py). It would be nice to have such a report. > Maybe we could set up a > "test-group" for each module in Zope, say Catalog/ZCatalog/SearchIndex, > that before each release, really beat the living shit about of it? A bit > more coordinated. That sounds good! At least for the Catalog. Want to be a tester? ;-) > > > What's an acceptable query time for your application? Are you sorting > > the results via sort_on? > > Well, actually, the query time isn't that bad (now with only 50k objects), > it's all the problems a big result-set drags along with it that really > slows things down. Still curious whether it's index dependent. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
On Thu, 17 May 2001, Chris McDonough wrote: > I'd be curious to know how long a query that involves only a single > field index takes, and how long a query that involves only a single text > index takes... does each take a roughly equivalent amount of time? I might be able to check that for you later tonight or during the weekend. What I did to solve my problems was to throw a couple of objects around so that I now only have about 50k objects in the Catalog. Boy it's fast. I like! :) > Or is one much faster than the other? If one is not much faster than > the other, it's a Catalog issue. If one *is* much faster than the > other, it's an Index issue. Have you guys at DC not done testing on this? Maybe we could set up a "test-group" for each module in Zope, say Catalog/ZCatalog/SearchIndex, that before each release, really beat the living shit about of it? A bit more coordinated. > What's an acceptable query time for your application? Are you sorting > the results via sort_on? Well, actually, the query time isn't that bad (now with only 50k objects), it's all the problems a big result-set drags along with it that really slows things down. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
Erik Enge wrote: > I've indexed about 410.000 objects now. A plain query with 'meta_type' > and 'firstname' to searchResults takes about 3-4 seconds. Not too bad, > but not that good either. I assume meta_type is a field index and 'firstname' is a text index. I'd be curious to know how long a query that involves only a single field index takes, and how long a query that involves only a single text index takes... does each take a roughly equivalent amount of time? Or is one much faster than the other? If one is not much faster than the other, it's a Catalog issue. If one *is* much faster than the other, it's an Index issue. > I'm sure I can so something to make it faster, but other than index fewer > objects (which I can't do, since I have more objects that needs to be > indexed) I don't see what else I can do. What's an acceptable query time for your application? Are you sorting the results via sort_on? > Maybe one could design a framework of scalable Catalogs? Just like ZEO > does for ZODB? There are specific incremental improvements that can be made to the Catalog, especially via: 1) extending its query language 2) Making it less expensive to do incremental indexing -- need to queue up index requests There is a proposal for 1 on dev.zope.org named UnionAndIntersectionOperations. 2 has no proposal behind it. There are additionally no proposals to address lacking query time speed (of which there haven't been too many reports, but I can imagine problems, especially in conjunction with sorting). 2.4 will have "drop in" indexes which will make it possible to write your own type of index (such as, for example, a DateIndex, that stores document ids presorted in reverse chronological order, so you don't always need to sort the entire result set and reverse it for batch-type operations). This will have an impact on query speed to the extent that you'll be able to use an appropriate type of index for your data rather than stuffing it in to one of the default three. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
On Tue, 15 May 2001, Chris McDonough wrote: > I'll be curious to see the results. Hopefully you'll have better luck > under 2.3.1b2. I've indexed about 410.000 objects now. A plain query with 'meta_type' and 'firstname' to searchResults takes about 3-4 seconds. Not too bad, but not that good either. I'm sure I can so something to make it faster, but other than index fewer objects (which I can't do, since I have more objects that needs to be indexed) I don't see what else I can do. Maybe one could design a framework of scalable Catalogs? Just like ZEO does for ZODB? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
> > (it's fairly DWIM-ish as well) > > DWIM? Do What I Mean. ;-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
On Tue, 15 May 2001, Chris McDonough wrote: > YourCatalog.catalog_object(newobject) > > as opposed from inheriting from CatalogAware and relying on manage_afterAdd > or calling object.index_item() manually. That's really it. Well, if you put it that way :) *removing CatalogAwareness* > (it's fairly DWIM-ish as well) DWIM? > Both CatalogAware and CatalogPathAware need to be fixed somewhat (they > need to be replaced with an observer-based cataloger). This is > probably my fault at this point, but don't say I didn't tell you not > to use them later. ;-) Hehe, ok :) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
> > Why CatalogAware? You do know that the only thing CatalogAware does > > is add/remove/reindex objects in one particular Catalog when they're > > added, removed, or changed? > > Yes, and this is all I need. Where is the overhead with CatalogAware > objects, then? Any time a parent object is deleted or moved (or the object itself is moved), all the objects inside of it which are CatalogAware will then blissfully reindex themselves with the catalog. If you move a folder full of these things, you'll potentially wait hours for the copy to happen. Additionally, each of the manage_afterAdd/manage_afterClone/manage_beforeDelete methods of a CatalogAware objects iterates over the results of self.objectValues() and calls the respective methods on each of their subobjects (if you don't define objectValues for your noncontainerish object or if you don't inherit simpleitem, you'll be in a world of hurt). Additionally, the reindex_object method of CatalogAware (if you ever call it) is completely redundant and badly inefficient at this point, because it just calls unindex_object and index_object of the cataloged object. Versions of the Catalog as of late make this a very bad thing, because they're quite careful about only reindexing things which have changed, thus only catalog_object should be called. Also, the current CatalogAware uses nonvirtualhost-aware URLs as Catalog uids (absolute_url as opposed to getPhysicalPath) which might bite you if you're using virtual hosting. If this is important to you, you could try using "CatalogPathAware" which is CatalogAware updated to use virtual-host aware paths. All you need to do as a part of your code when not using CatalogAware is: YourCatalog.catalog_object(newobject) as opposed from inheriting from CatalogAware and relying on manage_afterAdd or calling object.index_item() manually. That's really it. The incremental-change stuff and remove-from-catalog-on-delete stuff provided by CatalogAware is currently inappropriate for large numbers of obj ects when moving, deleting, or renaming containers (it's fairly DWIM-ish as well) because of the assumptions it makes about time-to-unindex/reindex. This will hurt you in the long run. Both CatalogAware and CatalogPathAware need to be fixed somewhat (they need to be replaced with an observer-based cataloger). This is probably my fault at this point, but don't say I didn't tell you not to use them later. ;-) - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
On Tue, 15 May 2001, Chris McDonough wrote: > Why CatalogAware? You do know that the only thing CatalogAware does > is add/remove/reindex objects in one particular Catalog when they're > added, removed, or changed? Yes, and this is all I need. Where is the overhead with CatalogAware objects, then? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
> > I'd either make my own CatalogAware-alike mixin class that did things > > a bit differently than CatalogAware (perhaps didn't index on add, and > > didn't unindex on delete), or I'd just manage the whole lot completely > > manually. (How often will each of these million objects change?) > > Not often. But new objects will be added to them quite > frequently. Catalog aware ones. Why CatalogAware? You do know that the only thing CatalogAware does is add/remove/reindex objects in one particular Catalog when they're added, removed, or changed? You can likely afford to do that yourself if you've got the serious business of having a million similar objects to deal with. It's a trivial amount of code, and you'll likely discover that you didn't really want CatalogAware in the first place if they're not going to be changed on a regular basis. CatalogAware is something that is useful if you have a relatively small number of frequently-changing objects that often need their data reindexed on change. It certainly doesn't sound like you need this. > > > CatalogAware is fairly ancient code at this point, and I don't think > > it was written with the mindset that there would be a million objects > > in any particular catalog. > > Hm... I'll give it some thought. If it works the way it is now, I'm > sufficiently happy. If not: late nights making ErikAwareness ;). One late 1/2 hour more likely... ;-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
On Tue, 15 May 2001, Chris McDonough wrote: > Probably not much difference on bare bulk indexing speed, but I'll bet > that it finishes this time. ;-) We'll see :) > I'd either make my own CatalogAware-alike mixin class that did things > a bit differently than CatalogAware (perhaps didn't index on add, and > didn't unindex on delete), or I'd just manage the whole lot completely > manually. (How often will each of these million objects change?) Not often. But new objects will be added to them quite frequently. Catalog aware ones. > CatalogAware is fairly ancient code at this point, and I don't think > it was written with the mindset that there would be a million objects > in any particular catalog. Hm... I'll give it some thought. If it works the way it is now, I'm sufficiently happy. If not: late nights making ErikAwareness ;). :-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
On Tue, 15 May 2001, Chris Withers wrote: > Why nto go for 2.3.2 final? IIRC, 2.3.2b2 had some nasty ZCatalog bugs > in it... Heh, that's what I get for not keeping up; I didn't even know 2.3.2 final was out. Bleh. :-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
> Thanks for the fast reply! > > On Tue, 15 May 2001, Chris McDonough wrote: > > > Have you read > > http://www.zope.org/Members/mcdonc/HowTos/UpgradeToNewCatalog/index_html > > ? I suspect there will be improvement. > > Surely there will be improvement, but not of factors two or three, or > more? Probably not much difference on bare bulk indexing speed, but I'll bet that it finishes this time. ;-) > > And, I can do the who adding process again, so I don't need to convert the > Catalog. That's good... > > > If I do not uncomment those things in CatalogAwareness.py I get an error > > > on SERVER_URL. I think it is resolve_url() that tries to make a run for > > > it. > > > > What are you referring to when you say "those things" in the sentence above? > > Sorry, the lines 181-184 of CatalogAwareness.py: > > def index_object(self): > """A common method to allow Findables to index themselves.""" > #if hasattr(self, self.default_catalog): > # getattr(self,self.default_catalog).catalog_object(self,self.url()\ > ) I'd probably refrain from using a million CatalogAware objects. Instead, I'd either make my own CatalogAware-alike mixin class that did things a bit differently than CatalogAware (perhaps didn't index on add, and didn't unindex on delete), or I'd just manage the whole lot completely manually. (How often will each of these million objects change?) CatalogAware is fairly ancient code at this point, and I don't think it was written with the mindset that there would be a million objects in any particular catalog. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
Erik Enge wrote: > > This is Zope 2.3.1b1, by the way. I'm changing to Zope 2.3.2b2 as we > speak, but I don't think it will improve performance that much. Why nto go for 2.3.2 final? IIRC, 2.3.2b2 had some nasty ZCatalog bugs in it... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
Thanks for the fast reply! On Tue, 15 May 2001, Chris McDonough wrote: > Have you read > http://www.zope.org/Members/mcdonc/HowTos/UpgradeToNewCatalog/index_html > ? I suspect there will be improvement. Surely there will be improvement, but not of factors two or three, or more? And, I can do the who adding process again, so I don't need to convert the Catalog. > > If I do not uncomment those things in CatalogAwareness.py I get an error > > on SERVER_URL. I think it is resolve_url() that tries to make a run for > > it. > > What are you referring to when you say "those things" in the sentence above? Sorry, the lines 181-184 of CatalogAwareness.py: def index_object(self): """A common method to allow Findables to index themselves.""" #if hasattr(self, self.default_catalog): #getattr(self,self.default_catalog).catalog_object(self,self.url()\ ) > Have you seen the "makerequest.py" script inside of the > lib/python/Testing directory? It does what you're trying to do when > you set applic.REQUEST=REQUEST (the major difference being that it > works ;-). I'll check that out. :-) > If you read the Zope Developer's Guide testing and debugging chapter > from its CVS repository, I believe it's documented in there. If you > don't have the lib/python/Testing/makerequest.py file, look on > cvs.zope.org for it. Will do :) > I'll be curious to see the results. Hopefully you'll have better luck > under 2.3.1b2. So I hope. I'll let you know when the results come through; probably tomorrow morning at the earliest. Thanks so far :) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog, REQUEST, misc.
> I have a 1GHz Pentium with 1GB RAM and 1GB swap. After
> I added all the objects with a little script (that took about 12 hours), I
> was going to index them to the Catalog I have. (I had to uncomment the
> index_object method's innards in CatalogAwareness.py because of a problem
> I mention further down.)
>
> After using up all my RAM and swap, taking five hours and making nothing
> happen, I gave up trying to index the objects. (This was _with_
> subtransactions enabled; set to "1".)
>
> This is Zope 2.3.1b1, by the way. I'm changing to Zope 2.3.2b2 as we
> speak, but I don't think it will improve performance that much.
Have you read
http://www.zope.org/Members/mcdonc/HowTos/UpgradeToNewCatalog/index_html ?
I suspect there will be improvement.
>
> So, to the origin of one of the problems.
>
> This is part of my script to add all these objects:
>
> """
> from AccessControl.SecurityManagement import newSecurityManager
> from ZPublisher.Request import Request
> from ZPublisher.Response import Response
>
> import sys
> import Zope
> import os
>
> os.environ['SERVER_NAME'] = 'localhost'
> os.environ['SERVER_PORT'] = '80'
>
> response=Response()
> REQUEST = Request(sys.stdin, os.environ, response)
>
> applic=Zope.app()
> #applic.REQUEST = REQUEST
>
> uf=applic.acl_users
> user=uf.getUser('sysadm').__of__(uf)
> newSecurityManager(None, user)
> print applic
> """
>
> Nice, that works. What happends from now on in the script is quite
> boring; it just adds object. Kinda like:
>
> applic['object'].manage_add_something()
>
> If I do not uncomment those things in CatalogAwareness.py I get an error
> on SERVER_URL. I think it is resolve_url() that tries to make a run for
> it.
What are you referring to when you say "those things" in the sentence above?
Have you seen the "makerequest.py" script inside of the lib/python/Testing
directory? It does what you're trying to do when you set
applic.REQUEST=REQUEST (the major difference being that it works ;-). If
you read the Zope Developer's Guide testing and debugging chapter from its
CVS repository, I believe it's documented in there. If you don't have the
lib/python/Testing/makerequest.py file, look on cvs.zope.org for it.
> Clever as I thought I was, I added the "applic.REQUEST = REQUEST" line
> (which, as you can see, is now commented out) and thought that it would
> fix my problems. Initially, it looked good. Adding the objects (with the
> CatalogAwareness.py stuff not commented out) was working.
>
> However, when I tried to access any of the newly added objects, Zope died,
> restarted and pretended like nothing had happend. This is reproducable.
I don't know why this is, but I'd suggest you use makerequest.
> What I would really like, is to avoid this after-add-reindex procedure
> (since I can't make it happen). Is there any way I can produce a "good
> enough" REQUEST for Zope? So that it doesn't do that die-repeat cycle on
> me?
>
> Any hints greatly appretiated! :-)
>
> (I haven't got much confidence that the Catalog will be able to respond to
> any queries with over a million objects indexed, but time will show I
> guess. I tried a 2.3.1b1 Catalog with 250.000 odd objects, and it wasn't
> very talkative.)
I'll be curious to see the results. Hopefully you'll have better luck under
2.3.1b2.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog, REQUEST, misc.
Hi.
I was adding a couple of objects to my system. Turns out, it's over a
million of them. I have a 1GHz Pentium with 1GB RAM and 1GB swap. After
I added all the objects with a little script (that took about 12 hours), I
was going to index them to the Catalog I have. (I had to uncomment the
index_object method's innards in CatalogAwareness.py because of a problem
I mention further down.)
After using up all my RAM and swap, taking five hours and making nothing
happen, I gave up trying to index the objects. (This was _with_
subtransactions enabled; set to "1".)
This is Zope 2.3.1b1, by the way. I'm changing to Zope 2.3.2b2 as we
speak, but I don't think it will improve performance that much.
So, to the origin of one of the problems.
This is part of my script to add all these objects:
"""
from AccessControl.SecurityManagement import newSecurityManager
from ZPublisher.Request import Request
from ZPublisher.Response import Response
import sys
import Zope
import os
os.environ['SERVER_NAME'] = 'localhost'
os.environ['SERVER_PORT'] = '80'
response=Response()
REQUEST = Request(sys.stdin, os.environ, response)
applic=Zope.app()
#applic.REQUEST = REQUEST
uf=applic.acl_users
user=uf.getUser('sysadm').__of__(uf)
newSecurityManager(None, user)
print applic
"""
Nice, that works. What happends from now on in the script is quite
boring; it just adds object. Kinda like:
applic['object'].manage_add_something()
If I do not uncomment those things in CatalogAwareness.py I get an error
on SERVER_URL. I think it is resolve_url() that tries to make a run for
it.
Clever as I thought I was, I added the "applic.REQUEST = REQUEST" line
(which, as you can see, is now commented out) and thought that it would
fix my problems. Initially, it looked good. Adding the objects (with the
CatalogAwareness.py stuff not commented out) was working.
However, when I tried to access any of the newly added objects, Zope died,
restarted and pretended like nothing had happend. This is reproducable.
What I would really like, is to avoid this after-add-reindex procedure
(since I can't make it happen). Is there any way I can produce a "good
enough" REQUEST for Zope? So that it doesn't do that die-repeat cycle on
me?
Any hints greatly appretiated! :-)
(I haven't got much confidence that the Catalog will be able to respond to
any queries with over a million objects indexed, but time will show I
guess. I tried a 2.3.1b1 Catalog with 250.000 odd objects, and it wasn't
very talkative.)
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
