Re: [Zope] Big folders
Jimmie Houchin wrote: > My apologies if this covered somewhere. no, but it is talked about: http://www.zope.org/Wikis/zope-dev/ReallyBigFolders ~ethan ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Big folders
- Original Message - From: "Albert Langer" <[EMAIL PROTECTED]> To: "'Jimmie Houchin'" <[EMAIL PROTECTED]>; "'Kevin Dangoor'" <[EMAIL PROTECTED]> Cc: "'Zope Mailing List'" <[EMAIL PROTECTED]> Sent: Saturday, June 17, 2000 6:33 PM Subject: RE: [Zope] Big folders > [AL] As well as an ETA for Btrees, in the absence of ZPatterns > documentation, could somebody please post URLs for specific references to > further information about Btrees in particular and Zpatterns in eneral, - > eg email list archives and other Wickis particular thread URLs etc. The BTrees are included with Zope itself (ZCatalog uses them). I'm not sure if there are docs online for the btrees themselves, though I did see a message on the mailing list (zope-dev, I believe) a few weeks back that gave a brief run down of what you can do with the BTree module. (The message was from Michel Pelletier). > I've been studying the ZPatterns Wicki and code with a view to following up > on Kevin's earlier email reference to it which I posted to Zcommerce > recently. Didn't notice any explicit reference to Btree folders within > ZPatterns. ZPatterns does not come with a "BTreeFolder"... Basically, ZPatterns includes a "Rack" object which contains "RackMountable" objects. The pattern defines a standard set of methods (like getItem, newItem) that can be implemented to pull information from different sources in a way that is transparent to other Zope components. If you look through the code in Racks.py, you'll see that the default implementation stores the objects in a BTree in the ZODB. Racks don't have the friendly user interface that a normal Folder does. But, you don't necessarily want that interface, because you wouldn't want to wait for a listing of 10,000 objects. > As far as I can make out, Kevin is saying, and appears to be > right, that by designing for "Zpatterns" it should be feasible use either > ZODB or an RDBM without having to rewrite the whole application, and also > feasible to rewrite just the parts that are affected by different facilities > on different RDBMs to support more than one RDBM option. (Of course if a > Zope application is NOT designed from the start to do that, but simply > assumes the ZODB will be the only data source, this would not apply. > Unfortunately the two currently available Zope ecommerce products appear to > have taken that route). Correct. Using ZPatterns, you can make some parts of an object come from an RDBMS if you wish, other parts from the ZODB. And, yes, the existing ecommerce products are designed to use the ZODB alone, as far as I've seen. > However I got a strong impression ZPatterns may not be as useful as I > thought, in a recent exchange with Chris McDonough in an OpenACS bboard > (attached, with link to discussion). Any comments and references from Kevin > and other people more familiar with ZPatterns would be appreciated. As Chris mentioned in his message there, ZPatterns is not well documented and there is no collection of example programs to work from. So far, for many people, ZPatterns probably seems like more trouble than it's worth. But, ZPatterns 0.4 provides a bunch of new functionality that is very useful. The main intention of ZPatterns is to allow you to create very customizable Zope products. If you're developing something in-house that will be used for one product, you don't really need ZPatterns (well, you might want it because requirements *always* change over time, but you certainly don't need it to get your app going.) If you're developing some code that can be applied to many different scenarios with some small amount of customization, ZPatterns is great. LoginManager is the best living example of that. Here is a product that provides Zope authentication. Your users can login using cookies or basic auth (or even session info in the URL, if you feel like writing that code). User objects can get their data from the ZODB, an RDBMS, or LDAP. Or all three at once. There is a lot of code reuse going on there... > BTW While Btrees might make ZODB usable with very large folders I can't see > how it would make it better than using an RDBM in that situation (hopefully > under Zope, using ZPatterns). Once you have a lot more data than fits in RAM > you are likely to also have a higher ratio of writes to reads than ZODB is > "aggressively optimized" for, and need an application design that explicitly > optimizes for relational operations on subsets of data rather than > navigating from one linked object to another as OODBMs are designed to do. > Using an OODBM like ZODB would mean a full disk seek for every item, which > RDBMs are designed to avoid. Likewise if you
RE: [Zope] Big folders
[AL] As well as an ETA for Btrees, in the absence of ZPatterns documentation, could somebody please post URLs for specific references to further information about Btrees in particular and Zpatterns in general, - eg email list archives and other Wickis particular thread URLs etc. I've been studying the ZPatterns Wicki and code with a view to following up on Kevin's earlier email reference to it which I posted to Zcommerce recently. Didn't notice any explicit reference to Btree folders within ZPatterns. As far as I can make out, Kevin is saying, and appears to be right, that by designing for "Zpatterns" it should be feasible use either ZODB or an RDBM without having to rewrite the whole application, and also feasible to rewrite just the parts that are affected by different facilities on different RDBMs to support more than one RDBM option. (Of course if a Zope application is NOT designed from the start to do that, but simply assumes the ZODB will be the only data source, this would not apply. Unfortunately the two currently available Zope ecommerce products appear to have taken that route). However I got a strong impression ZPatterns may not be as useful as I thought, in a recent exchange with Chris McDonough in an OpenACS bboard (attached, with link to discussion). Any comments and references from Kevin and other people more familiar with ZPatterns would be appreciated. BTW While Btrees might make ZODB usable with very large folders I can't see how it would make it better than using an RDBM in that situation (hopefully under Zope, using ZPatterns). Once you have a lot more data than fits in RAM you are likely to also have a higher ratio of writes to reads than ZODB is "aggressively optimized" for, and need an application design that explicitly optimizes for relational operations on subsets of data rather than navigating from one linked object to another as OODBMs are designed to do. Using an OODBM like ZODB would mean a full disk seek for every item, which RDBMs are designed to avoid. Likewise if you need concurrent external access to the same data via ODBC I don't see how ZODB Btrees would help (eg for ecommerce links to General Ledger and fulfilment etc). What worries me is that in the absence of actual working examples of how to handle RDBMs with ZPatterns, I have had to recommend that someone needing an RDBM based application (Ecommerce) use the OpenACS ecommerce module and just kludge a way to also use Zope for other things in parallel. I'll be discussing the specific implications for ecommerce in Zcommerce, but perhaps here is better for info on ZPatterns and RDBMs generally? Also, although still not sufficiently well informed, I have a strong feeling that the features of Postgresql as an ORDBM as opposed to an ODBM or an RDBM are particularly relevant to Zope. As well as a normal external Python API, Postgresql has the capability to run scripting code as well as C functions inside the database engine as a stored procedure language and also for dynamically loaded user defined data types and comparison operators etc. It is a true "Object" Relational DBMS rather than a conventional RDBMs. Already implemented for Tcl. If some Zope/Python code was put inside it, designed to work with attribute providers and especially indexing agents and rule agents for ZPatterns, it strikes me that the combination could be incredibly powerful. Has there been any discussion of that, or any thoughts now? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Jimmie Houchin Sent: Sunday, June 18, 2000 7:12 AM To: Kevin Dangoor Cc: Zope Mailing List Subject: Re: [Zope] Big folders [JH] Thanks, this is great info. I was planning on developing with ZPatterns but haven't downloaded or began yet. I wasn't aware that a Rack could do that. Cool. I don't mind going to a RDBMS if necessary, but if possible I like sticking with Zope/ZODB. Does anyone have an eta on BTrees for ZODB? Jimmie Houchin Kevin Dangoor wrote: > > [snip] [JH] > > My databases will have millions of objects in what would/will be single > > table/folders. From what I understand ZODB doesn't currently do big > > folders well. > > > > So I guess my question is, should I continue with the SQL and the RDBMS > > or is there a solution that I don't know about or understand with ZODB? > [KD] > As Shane mentioned, there's a BTreeFolder in the works. Additionally, you > can use the "Rack" from ZPatterns. If you design for a Rack, you can switch > to an RDBMS later on without having to change your application code. But, > you might not need to switch to an RDBMS, because the current Rack > implementation can store objects using BTrees within the ZODB. So, > theoretically, you can have great big piles of objects within a single Rack. > > Unfortunately,
Re: [Zope] Big folders
Thanks, this is great info. I was planning on developing with ZPatterns but haven't downloaded or began yet. I wasn't aware that a Rack could do that. Cool. I don't mind going to a RDBMS if necessary, but if possible I like sticking with Zope/ZODB. Does anyone have an eta on BTrees for ZODB? Jimmie Houchin Kevin Dangoor wrote: > > [snip] > > My databases will have millions of objects in what would/will be single > > table/folders. From what I understand ZODB doesn't currently do big > > folders well. > > > > So I guess my question is, should I continue with the SQL and the RDBMS > > or is there a solution that I don't know about or understand with ZODB? > > As Shane mentioned, there's a BTreeFolder in the works. Additionally, you > can use the "Rack" from ZPatterns. If you design for a Rack, you can switch > to an RDBMS later on without having to change your application code. But, > you might not need to switch to an RDBMS, because the current Rack > implementation can store objects using BTrees within the ZODB. So, > theoretically, you can have great big piles of objects within a single Rack. > > Unfortunately, ZPatterns is still very new and there is not a lot of > documentation. But, when you start playing with it, things become a bit > clearer. > > Kevin > > ___ > Zope maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Big folders
[snip] > My databases will have millions of objects in what would/will be single > table/folders. From what I understand ZODB doesn't currently do big > folders well. > > So I guess my question is, should I continue with the SQL and the RDBMS > or is there a solution that I don't know about or understand with ZODB? As Shane mentioned, there's a BTreeFolder in the works. Additionally, you can use the "Rack" from ZPatterns. If you design for a Rack, you can switch to an RDBMS later on without having to change your application code. But, you might not need to switch to an RDBMS, because the current Rack implementation can store objects using BTrees within the ZODB. So, theoretically, you can have great big piles of objects within a single Rack. Unfortunately, ZPatterns is still very new and there is not a lot of documentation. But, when you start playing with it, things become a bit clearer. Kevin ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Big folders
Jimmie Houchin wrote: > > I've been studying SQL this week because I've been thinking that I'll > probably have to use a RDBMS to develop my site. Ideally and > philosophically I like ZODB. Mountable ZODBs give me much of what I need > with the possible exception of a big folder problems. > > My databases will have millions of objects in what would/will be single > table/folders. From what I understand ZODB doesn't currently do big > folders well. It's more accurate to say that the current Folder implementation in Zope does not handle a large number of subobjects well. As I understand it, to load any subobject, it must load *all* subobjects from ZODB. A "BTreeFolder", which promises to solve part of the problem, is in the works. ZODB itself, however, is and always was a great way to store just about any kind of data. Shane ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
[Zope] Big folders
I've been studying SQL this week because I've been thinking that I'll probably have to use a RDBMS to develop my site. Ideally and philosophically I like ZODB. Mountable ZODBs give me much of what I need with the possible exception of a big folder problems. My databases will have millions of objects in what would/will be single table/folders. From what I understand ZODB doesn't currently do big folders well. So I guess my question is, should I continue with the SQL and the RDBMS or is there a solution that I don't know about or understand with ZODB? My apologies if this covered somewhere. Jimmie Houchin ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
