Re: [Zope-dev] skinscript and URL traversal question
Hi all, A solution has been found. woohoo. A not found error was produced with direct URL traversal. Steve Spicklemire sent this solution off list. The process is split into two steps: Do the query to 'load the object on traversal': WITH QUERY searchBy_Name(name=self.id) COMPUTE name Do the query to 'load the attributes needed for rendering a particular document': WITH QUERY searchByNameCheck(self.id) COMPUTE username,coupon_text,categorylist,categoryheader,expirationdate,status,couponterms During traversal AUTHENTICATED_USER is not yet defined, so it's no good trying to get to it. However, once authentication has happened (after Zope has sorted out what object you're after an imposed security on the object..) then it should be OK. Since the Storage tab uses 'name' to load the object, we can't access AUTHENTICATED_USER before loading 'name', but the other attributes are OK. -Aaron ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Specialist/Rack scalability
After comsidering the fedback I got from the previous 'Massive scalability thread, I decided to split my queries into two areas: Rack scalability and ZCatalog scalability. This email deals with the former. It seems clear that indexing and searching are more of a botleneck than storage/retreival. Nevertheless, so far I have not heard of anyone trying to store more than 60,000 objects in a rack. I need to know if there is any reason to suspect that storage (in the ZODB) or retreival performance would suffer if the number of objects was in the hundreds of thousands or even millions. Does anyone have anectodal or benchmark data that would suggest what happens with that many objects? Thanks, Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog scalability
After comsidering the feedback I got from the previous 'Massive scalability' thread, I decided to split my queries into two areas: Rack scalability and ZCatalog scalability. This email deals with the latter. Partial match (wildcard) searches have already been identified as a resource hog, depending on the size of the result list. I am more than willing to give up wildcards in my application for performance. What I am interested in for my application are two things: - ZTopics populated using one or more keyword indexes - Full text search on a single computed attribute that concatenates several fields including the aforementioned keyword index fields and a few simple string attributes (title, caption, description, etc.) I need to know how far the ZCatalog will scale using this indexing and search strategy. Does anyone have anectodal or benchmark data to suggest if (and when) I will hit a 'wall' regarding the number of objects being indexed and searched? Some anectodal data suggests that single field indexing will scale easily to 60,000 objects, what about hundreds of thousands or even millions of objects? Also, is there a way to disable wildcards in full text searches? Thanks, Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Specialist/Rack scalability
Hi Michael, Michael Bernstein wrote: It seems clear that indexing and searching are more of a botleneck than storage/retreival. Nevertheless, so far I have not heard of anyone trying to store more than 60,000 objects in a rack. I need to know if there is any reason to suspect that storage (in the ZODB) or retreival performance would suffer if the number of objects was in the hundreds of thousands or even millions. I can't answer your question; however, I may be able to help clarify the question. The ZODB is really just a transaction manager, and an interface and contract of behaviour, for an object database. You can plug a variety of Storages into the ZODB. The default storage the Zope comes with is FileStorage -- Data.fs. There are also BerkeleyStorage, OracleStorage, DBMStorage, and others, in varying states of finishedness. So, storing things in a Rack happens in a number of stages: Your application interacts with the Rack The Rack (perhaps) stores the object persistently in its BTree The BTree is a collection of persistent ZODB objects The ZODB objects are stored as Python Pickles in a FileStorage We can consider what the effect of storing 60 000 objects is at each of these interfaces. The Rack shouldn't have a problem with 60 000 objects. I doubt a BTree would have a problem. The ZODB might not like accessing many large objects during a single transaction, as all those objects need to be in memory at once. A FileStorage should have no problem reading 60 000 stored objects. However, if these objects are changing much, your Data.fs will grow fast. In any case, you may find undo and history screens take a long time to appear. However, if you are using a Rack, you have a lot of choice about where you put your data. You can put frequently changed aspects of your data on the filesystem, and the rest in FileStorage for example. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Michael Bernstein wrote: Also, is there a way to disable wildcards in full text searches? Do not allow direct queries to search the catalog. Instead, make searches go through an external method (or a PythonScript with Proxy permissions) that uses string.replace to change '*' and '?' to ''. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Steve Alexander wrote: Michael Bernstein wrote: Also, is there a way to disable wildcards in full text searches? Do not allow direct queries to search the catalog. Instead, make searches go through an external method (or a PythonScript with Proxy permissions) that uses string.replace to change '*' and '?' to ''. A *very* handy suggestion. You might want to add that as a Tip to Zope.org. Thanks, Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Erik Enge wrote: [Michael Bernstein] | I need to know how far the ZCatalog will scale using this indexing | and search strategy. Does anyone have anectodal or benchmark data to | suggest if (and when) I will hit a 'wall' regarding the number of | objects being indexed and searched? I'm going to try to stuff 27 million objects into ZODB sometime in the next week or the week after that (all post addresses in England). I haven't got a clue as to whether this will work or just... well not work. I haven't come up with a strategy for segmenting the data, but that shouldn't be a problem at all. This isn't actually much data, so I don't expect the Data.fs file to more than 500 MB. I'm quite confident that ZODB, ZCatalog and BTree will scale very nicely for this. I have a plan ;). I'll let you know how it goes. (And please, do poke at me if it takes too long. Will do, Thanks! Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Specialist/Rack scalability
Steve Alexander wrote: Hi Michael, Michael Bernstein wrote: It seems clear that indexing and searching are more of a botleneck than storage/retreival. Nevertheless, so far I have not heard of anyone trying to store more than 60,000 objects in a rack. I need to know if there is any reason to suspect that storage (in the ZODB) or retreival performance would suffer if the number of objects was in the hundreds of thousands or even millions. [snip] So, storing things in a Rack happens in a number of stages: Your application interacts with the Rack The Rack (perhaps) stores the object persistently in its BTree The BTree is a collection of persistent ZODB objects The ZODB objects are stored as Python Pickles in a FileStorage We can consider what the effect of storing 60 000 objects is at each of these interfaces. Are there any differences if you scale the number of objects up to the hundreds of thousands or even into the millions? The Rack shouldn't have a problem with 60 000 objects. I doubt a BTree would have a problem. The ZODB might not like accessing many large objects during a single transaction, as all those objects need to be in memory at once. Neither of my applications require batch adds to the DB, however, one of them (the image archive) has objects (Photos) with several images as attributes. This results in a fairly large object. There is some question in my mind if accessing any attribute (such as the thumbnail version) causes all attributes to be loaded into memory. If so, displaying a list of images with thumbnails may result in many large objects being loaded into memory. A FileStorage should have no problem reading 60 000 stored objects. However, if these objects are changing much, your Data.fs will grow fast. In any case, you may find undo and history screens take a long time to appear. No. Once added, I don't expect the data to change frequently. Thanks for the feedback. Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Specialist/Rack scalability
Michael Bernstein wrote: There is some question in my mind if accessing any attribute (such as the thumbnail version) causes all attributes to be loaded into memory. If so, displaying a list of images with thumbnails may result in many large objects being loaded into memory. Make sure that each large attribute is an instance of a class that derives from Persistent. Of course, if this is a ZPatterns application, you'd probably want to have the images in their own Rack, and use an Attribute Provider on your Photo objects that gets the images for a Photo as needed. The Photo (with meta-data) and the images are entirely different objects, accessed via different Racks, via different Specialists. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
[Michael Bernstein] | I need to know how far the ZCatalog will scale using this indexing | and search strategy. Does anyone have anectodal or benchmark data to | suggest if (and when) I will hit a 'wall' regarding the number of | objects being indexed and searched? I'm going to try to stuff 27 million objects into ZODB sometime in the next week or the week after that (all post addresses in England). I haven't got a clue as to whether this will work or just... well not work. I haven't come up with a strategy for segmenting the data, but that shouldn't be a problem at all. This isn't actually much data, so I don't expect the Data.fs file to more than 500 MB. I'm quite confident that ZODB, ZCatalog and BTree will scale very nicely for this. I have a plan ;). I'll let you know how it goes. (And please, do poke at me if it takes too long.) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Specialist/Rack scalability
Steve Alexander wrote: Michael Bernstein wrote: There is some question in my mind if accessing any attribute (such as the thumbnail version) causes all attributes to be loaded into memory. If so, displaying a list of images with thumbnails may result in many large objects being loaded into memory. Make sure that each large attribute is an instance of a class that derives from Persistent. Ok, I'll give that a try. Since Photo is a Python Product, what will happen to current instances if I make this (and only this) change? Of course, if this is a ZPatterns application, you'd probably want to have the images in their own Rack, and use an Attribute Provider on your Photo objects that gets the images for a Photo as needed. The Photo (with meta-data) and the images are entirely different objects, accessed via different Racks, via different Specialists. I'm not certain that that makes sense, since the Images are really cached 'views' of the Photo object. When a new image is uploded to replace an existing one, *all* versions (thumbnails, small, medium, large, etc) are regenerated. But assuming that I went so far as to break out the Images to their own Rack, would you reccomend that each image size have a dedicated Rack, or would you suggest that all images be stored in the same Rack? Thanks, Michael. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Specialist/Rack scalability
Michael Bernstein wrote: Make sure that each large attribute is an instance of a class that derives from Persistent. Ok, I'll give that a try. Since Photo is a Python Product, what will happen to current instances if I make this (and only this) change? I don't know. I can think of reasons that it might be ok. I can also rationalize why it would cause badness. :-) [[ put images in their own specialist ] I'm not certain that that makes sense, since the Images are really cached 'views' of the Photo object. When a new image is uploded to replace an existing one, *all* versions (thumbnails, small, medium, large, etc) are regenerated. Makes sense to me. You're not generating them on the fly; you're storing them persistently. If you put them in their own Specialist and Rack or Racks, you get to say how they are stored entirely independently of how the Photo objects are stored. I would have just one Images specialist, and then probably store them in different racks, but expose them to the rest of the application as all being of the same class of Image, but with a different image_size attribute; either "thumbnail", "small", "medium" or "large". That way, I could make the small rack generate thumbnails from the medium rack if, for example, the small size was rarely requested. There are many ways to design that though, and it depends on how you want things to work. (Obviously :-) ) But assuming that I went so far as to break out the Images to their own Rack, would you reccomend that each image size have a dedicated Rack, or would you suggest that all images be stored in the same Rack? There are advantages and disadvantages to each approach. However, you should be hiding the details of what Racks exist behind the facade of the Specialist. The Specialist will have a getItem method, which will get an Image from the appropriate rack, and probably some methods like listImagesFor(photo) and getImageFor(image_type, photo) so you can get all the images for a particular photo. Perhaps also storeImageFor(photo, original_image), which would end up processing and storing images derived from the original image. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] CSS file serving
Casey Duncan wrote: [EMAIL PROTECTED] wrote: I've ended up hacking a version of special_dtml / ClassicHTMLFile that serves up CSS. Has anyone got a better way of doing it (that detects changes in the file and serves it up with the correct Content-Type)? Check out: http://www.zope.org/Members/haqa/ZStyleSheet Hrm - that's a little heavy for what I want - which is just an analog of HTMLFile but for CSS files. That is, something I can add to my Python product as an attribute. Definitely no management or other sorts of trickiness required. Richard -- Richard Jones [EMAIL PROTECTED] Senior Software Developer, Bizar Software (www.bizarsoftware.com.au) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Michael Bernstein wrote: Also, is there a way to disable wildcards in full text searches? Do not allow direct queries to search the catalog. Instead, make searches go through an external method (or a PythonScript with Proxy permissions) that uses string.replace to change '*' and '?' to ''. Wouldn't using a normal vocabulary as opposed to a globbing vocabulary prevent this as well? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Chris Withers wrote: Wouldn't using a normal vocabulary as opposed to a globbing vocabulary prevent this as well? That would stop globbing searches for everyone. While I might want to stop users of a site making wildcard searches, I still want to keep that facility for myself :-) -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] dtml-in batching
The dtml-in batching mechanisms are quite difficult to debug - the DTML documentation I have has examples which break it and there's no indication of why. It turns out the DT_In renderwb() code gobbles all exceptions from the rendering of the previous and next blocks. I've hacked my code so that there's no try/except clause any more. I'm not sure what exception it's trying to catch, but could it perhaps be made a little more picky? Richard -- Richard Jones [EMAIL PROTECTED] Senior Software Developer, Bizar Software (www.bizarsoftware.com.au) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] dtml-in batching
[EMAIL PROTECTED] wrote: The dtml-in batching mechanisms are quite difficult to debug - the DTML documentation I have has examples which break it and there's no indication of why. It turns out the DT_In renderwb() code gobbles all exceptions from the rendering of the previous and next blocks. I've hacked my code so that there's no try/except clause any more. I'm not sure what exception it's trying to catch, but could it perhaps be made a little more picky? Re-glancing at the code shows me that this is only the case for the next block. Sorry about that. -- Richard Jones [EMAIL PROTECTED] Senior Software Developer, Bizar Software (www.bizarsoftware.com.au) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] dtml-in batching
[EMAIL PROTECTED] wrote: The dtml-in batching mechanisms are quite difficult to debug - the DTML documentation I have has examples which break it and there's no indication of why. It turns out the DT_In renderwb() code gobbles all exceptions from the rendering of the previous and next blocks. I've hacked my code so that there's no try/except clause any more. I'm not sure what exception it's trying to catch, but could it perhaps be made a little more picky? And here's a diff - I'm pretty sure this was the intended behaviour... *** /tmp/DT_In.py.orig Mon Jan 22 12:00:53 2001 --- DT_In.pyMon Jan 22 12:00:58 2001 *** *** 561,566 --- 561,570 # there are more items, without actually # computing a length: sequence[end] + except IndexError: + if self.elses: result=render(self.elses, md) + else: result='' + else: pstart,pend,psize=opt(end+1-overlap,0, sz,orphan,sequence) kw['next-sequence']=1 *** *** 568,576 kw['next-sequence-end-index']=pend-1 kw['next-sequence-size']=pend+1-pstart result=render(section,md) - except: - if self.elses: result=render(self.elses, md) - else: result='' else: result = [] append=result.append --- 572,577 -- Richard Jones [EMAIL PROTECTED] Senior Software Developer, Bizar Software (www.bizarsoftware.com.au) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Specialist/Rack scalability
At 07:12 PM 1/21/01 +, Steve Alexander wrote: So, storing things in a Rack happens in a number of stages: Your application interacts with the Rack The Rack (perhaps) stores the object persistently in its BTree The BTree is a collection of persistent ZODB objects The ZODB objects are stored as Python Pickles in a FileStorage We can consider what the effect of storing 60 000 objects is at each of these interfaces. The Rack shouldn't have a problem with 60 000 objects. I doubt a BTree would have a problem. The ZODB might not like accessing many large objects during a single transaction, as all those objects need to be in memory at once. A FileStorage should have no problem reading 60 000 stored objects. However, if these objects are changing much, your Data.fs will grow fast. In any case, you may find undo and history screens take a long time to appear. However, if you are using a Rack, you have a lot of choice about where you put your data. You can put frequently changed aspects of your data on the filesystem, and the rest in FileStorage for example. Just to expand a little on the abov... Racks should scale at least as well, if not larger than a ZCatalog, given the same storage backing for the ZODB. This is because ZCatalog has to manage a minimum of one forward and reverse BTree for *each* index, plus another few BTrees for overall storage and housekeeping. Also, keyword and full text indexes store multiple BTree entries per object, so that's a factor as well. So don't worry about the Rack. If you're using a Rack, you can store the data anywhere, and you can index it in an RDBMS, LDAP directory, ZCatalog, or some combination thereof, using triggers to keep the data in sync. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
