Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
Terry Jones wrote: That's 3 times where I somehow get to be in the wrong for taking the time to point out that a piece of documentation in the project is quite misleading. You didn't say you thought it was misleading, you worded it as if you expected it to still be true. A small difference maybe, but we're all busy people ;-) That's one of the aims. It works against that aim if the documentation is misleading. And, it works against the aim of having more people involved if someone new and well-intentioned shows up, runs into what looks like a clear (and, yes, very minor) problem, takes the time to report it, and is then repeatedly told that they're in the wrong! Well, you didn't say "this is wrong, I want to fix it". You said "doing this thing that's obviously out of date should work, why doesn't it", which is a slightly different kettle of fish ;-) Sure. But when it's the only thing you have to go on, It's not. There's plenty of coverage both in the 2.6 book on zope.org and the 2.7 book on plope.org... when the document is sitting on the official zope.org web site, zope.org is a joke, despite many people's efforts. Get over it and move on to enjoying zope despite this ;-) and when it says something is separately downloadable, experience dictates that you ignore the mentioned version number and go looking for the latest version of that thing. Which is exactly what I did. I don't think that's correct. If you see an ancient version of any software mentioned in a doc, you might want to consider the validity of the document as a whole... original smiley). I was trying to help yesterday. I'm trying to now too, when I say that I don't think it's in the best interests of the project to tell newcomers who run across problems that they're in the wrong because they were naive enough to read and rely on the documentation. Well, okay, I'll try and make the point again: you need to think hard when reading docs... if it's ancient, it might well be out of date and wholely inaccurate to the point of being totally misleading, as in this case... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
> "Chris" == Chris Withers <[EMAIL PROTECTED]> writes: Chris> Terry Jones wrote: >> Usually you find database records by means of the record's key. >> However, the key that you use for your record will not always contain >> the information required to provide you with rapid access to the data >> that you want to retrieve. Chris> Ah, okay, you do this with multiple BTrees in Zope, unless you want Chris> to swallow the whole ZCatalog... I think I'll roll my own, at least for now. I don't want to use ZCatalog for the indexing part, at least not yet. The BaseIndex class in src/zope/app/catalog/README.txt of the Zope3 SVN tree illustrates pretty much all I need right now. See, I'm still relying on reading the docs :-) | >> For example, suppose your database contains | >> records related to users. The key might be a string that is some unique | >> identifier for the person, such as a user ID. | | Chris> from BTrees.IOBTree import IOBTree | Chris> chris = object() | | Chris> id2user = IOBTree() | Chris> id2user[1234] = chris | | >> is, by the information stored in the key), it may also on occasion want | >> to location people by, say, their name. | | Chris> from BTrees.OOBTree import OOBTree, OOSet | Chris> name2user = OOBTree() | Chris> name2user['chris'] = chris | | Chris> Now, if you have more than one value, you use a set: | | Chris> dan = object() | Chris> age2user = OOBTree() | Chris> age2user[27] = OOSet(chris,dan) Thanks for the code too. Terry ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
Hi Chris Yesterday you said: Chris> Not everything is as hard as some people manage to make it ;-) And today this | >> | $ cd Zope-2.0.0a3-src/ | Chris> And what version of Zope are we on now? and | Chris> But really, with Open Source, it's up to you to be the judge. If a That's 3 times where I somehow get to be in the wrong for taking the time to point out that a piece of documentation in the project is quite misleading. I do well understand that the principle of caveat emptor applies to open source. But, many people have worked for a long time on zope and they've spent so much time in packaging it up and making it available because they also want it to be used by others. That's one of the aims. It works against that aim if the documentation is misleading. And, it works against the aim of having more people involved if someone new and well-intentioned shows up, runs into what looks like a clear (and, yes, very minor) problem, takes the time to report it, and is then repeatedly told that they're in the wrong! | Chris> Well, by all means report this anachronism to zope-web@zope.org, and | Chris> someone will hopefully delete it. Ok, will do. | Chris> If a document is over 3 years old and mentions a version of Zope | Chris> that's *nine* point releases earlier than the one you're using, d'ya | Chris> not think it might be out of date? ;-) Sure. But when it's the only thing you have to go on, when the document is sitting on the official zope.org web site, and when it says something is separately downloadable, experience dictates that you ignore the mentioned version number and go looking for the latest version of that thing. Which is exactly what I did. BTW, I don't personally mind your reaction at all (and I did see the original smiley). I was trying to help yesterday. I'm trying to now too, when I say that I don't think it's in the best interests of the project to tell newcomers who run across problems that they're in the wrong because they were naive enough to read and rely on the documentation. Anyway, no hard feelings, and thank for your replies. Apologies for so many words. Regards, Terry ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
Terry Jones wrote: |ZCatalog can be downloaded from the Zope download area and is also a |module in the public CVS for Zope. Untar it while in the root directory |of your Zope installation: | | $ cd Zope-2.0.0a3-src/ And what version of Zope are we on now? | $ tar xzf ../ZCatalog-x.x.tgz Or, do you really want to shoot the messenger? :-) Well, by all means report this anachronism to zope-web@zope.org, and someone will hopefully delete it. But really, with Open Source, it's up to you to be the judge. If a document is over 3 years old and mentions a version of Zope that's *nine* point releases earlier than the one you're using, d'ya not think it might be out of date? ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
Terry Jones wrote: Usually you find database records by means of the record's key. However, the key that you use for your record will not always contain the information required to provide you with rapid access to the data that you want to retrieve. Ah, okay, you do this with multiple BTrees in Zope, unless you want to swallow the whole ZCatalog... For example, suppose your database contains records related to users. The key might be a string that is some unique identifier for the person, such as a user ID. from BTrees.IOBTree import IOBTree chris = object() id2user = IOBTree() id2user[1234] = chris is, by the information stored in the key), it may also on occasion want to location people by, say, their name. from BTrees.OOBTree import OOBTree, OOSet name2user = OOBTree() name2user['chris'] = chris Now, if you have more than one value, you use a set: dan = object() age2user = OOBTree() age2user[27] = OOSet(chris,dan) Although, to be honest, if you're doing all this, you'll likely get more milleage out of the zcatalog than rolling it all yourself... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
[Following up on my original posting, with a summary of resolution.] Thanks for the help & pointers to ZCatalog. I actually did find a standalone version of ZCatalog, created by Kevin Dangoor, at http://www.blazingthings.com/dev/zcatalog.html His pulls code out of Zope3, and rewrites some of it. Kevin's code and one example each requires one trivial change to work on the current Zope3 SVN version (email me if you need these). I read the plope chapter on ZCatalog, but its emphasis is on getting at ZCatalog from zope, whereas I want to do everything from python, with no zope instance. Meanwhile, I had decided that my option #3 | 3. Redesign what I'm doing so as to only need ZODB BTrees. This is quite | far from optimal (in terms of space and time), but might be the | quickest to implement. | | However, this may not solve the problem. If a fundamental need is to | (e.g.) find all values >= X, where duplicates are allowed but you're | using trees that do not allow duplicate keys, this is difficult. Hence | the question about byValue(). actually wasn't too hard, and that I should simply store values as keys and let values be a list of objects that had that value. (I hope that makes sense) Then, finally reading the Zope3 src/zope/app/catalog/README.txt I saw that that is basically what ZCatalog already does (except with a BTrees.IFBTree.IFTreeSet not a list), as well as adding indexing. That solution wastes space, as values are stored twice. How much of a waste depends on the multiplicity of value values, but it could about a 2x waste (i.e., when all values are distinct and their size is large compared to the size of an docid). And there's more pickling, and more traffic to the storage. But that's life I guess. I'd rather the simpler, easier solution that lies wholly within ZODB/ZEO and which lets me have just a single transaction and recovery type to worry about than to hack a hybrid ZODB/bsddb just to save space and have to deal with those issues in a robust way. So that's it, problem solved. Thanks again for the pointer to ZCatalog. Terry ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
> "Chris" == Chris Withers <[EMAIL PROTECTED]> writes: Chris> ZCatalog ships as a standard part of Zope, no need to download. Chris> It's covered extensively in the zope book on Plope.org... Chris> Not everything is as hard as some people manage to make it ;-) OK, thanks. I hope you'll agree though that the three excerpts below from http://www.zope.org/Documentation/How-To/ZCatalogTutorial do give the clear impression that ZCatalog is available as a separate standalone entity and is available somewhere as ZCatalog-x-x.tgz for download. |* Usable outside of Zope. The software is broken into a Python Catalog | which wrapped by a ZCatalog. The Python Catalog can be used in any | Python program; all it requires is the Z object database and the | indexing machinery from Zope. |* The ZCatalog is a free, Open Source part of the Zope software | repository and thus is covered under the same license as Zope. It is | being developed in conjunction with the Zope Portal Toolkit | effort. However, the ZCatalog product is managed as its own module in | CVS. |Installing ZCatalog | |ZCatalog can be downloaded from the Zope download area and is also a |module in the public CVS for Zope. Untar it while in the root directory |of your Zope installation: | | $ cd Zope-2.0.0a3-src/ | $ tar xzf ../ZCatalog-x.x.tgz Or, do you really want to shoot the messenger? :-) BTW, I did see the ZCatalog stuff in the Zope2 tarball, but I assumed that was the Zope Product, not the implied standalone. Anyway, I'll get on with it. Thanks. Terry ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
Terry Jones wrote: I've just spent 20 minutes trying to do this, without success. So if someone has a URL for a ZCatalog download, I'd be happy to receive it. ZCatalog ships as a standard part of Zope, no need to download. It's covered extensively in the zope book on Plope.org... Not everything is as hard as some people manage to make it ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
Hi Jonathan > "Jonathan" == Jonathan <[EMAIL PROTECTED]> writes: Jonathan> This usage scenario can easily be implemented using ZCatalog (as Jonathan> Chris mentioned) - check the ZopeBook for info. Thanks, that sounds good. Without meaning to whine, I hope it's easier to use than it is to actually find and download ZCatalog :-) I've just spent 20 minutes trying to do this, without success. So if someone has a URL for a ZCatalog download, I'd be happy to receive it. Here are the things I tried: - Google(zope ZCatalog). This got me to http://www.zope.org/Documentation/How-To/ZCatalogTutorial which has an overview (that sounds good), but does not tell me where to download. - I read one-by-one through each of 15 really poorly formatted pages summarizing 300 Zope products. These were claimed to be sorted alphabetically, but they certainly are not. Clicking on the title column did rearrange them, but not into sorted or reverse-sorted order. - I used the search box on the www.zope.org site and the results were dismal. - I looked in the source tarball of recent Zope 2 & 3 releases. - Add "download" to the above google, followed a few links, gave up :-) Anyway, thanks again. Terry ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
From the Berkeley DB docs: Usually you find database records by means of the record's key. However, the key that you use for your record will not always contain the information required to provide you with rapid access to the data that you want to retrieve. For example, suppose your database contains records related to users. The key might be a string that is some unique identifier for the person, such as a user ID. Each record's data, however, would likely contain a complex object containing details about people such as names, addresses, phone numbers, and so forth. While your application may frequently want to query a person by user ID (that is, by the information stored in the key), it may also on occasion want to location people by, say, their name. Rather than iterate through all of the records in your database, examining each in turn for a given person's name, you create indexes based on names and then just search that index for the name that you want. You can do this using secondary databases. In DB, the database that contains your data is called a primary database. A database that provides an alternative set of keys to access that data is called a secondary database. In a secondary database, the keys are your alternative (or secondary) index, and the data corresponds to a primary record's key. This usage scenario can easily be implemented using ZCatalog (as Chris mentioned) - check the ZopeBook for info. Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
Hi Chris > "Chris" == Chris Withers <[EMAIL PROTECTED]> writes: You may Chris> have more joy with these questions on the zodb-dev@zope.org mailing Chris> list... Thanks, I've re-sent. Thanks for the other comments & pointers too. Regards Terry. >> - ZODB BTrees do not allow you to set up secondary keys (a la bsddb). Chris> Dunno what that means. I think you're just use two BTrees for this... [Sorry if the following is off-topic for the list. It's presumably good for the archives, and, I suppose, is the end of the thread.] >From the Berkeley DB docs: Usually you find database records by means of the record's key. However, the key that you use for your record will not always contain the information required to provide you with rapid access to the data that you want to retrieve. For example, suppose your database contains records related to users. The key might be a string that is some unique identifier for the person, such as a user ID. Each record's data, however, would likely contain a complex object containing details about people such as names, addresses, phone numbers, and so forth. While your application may frequently want to query a person by user ID (that is, by the information stored in the key), it may also on occasion want to location people by, say, their name. Rather than iterate through all of the records in your database, examining each in turn for a given person's name, you create indexes based on names and then just search that index for the name that you want. You can do this using secondary databases. In DB, the database that contains your data is called a primary database. A database that provides an alternative set of keys to access that data is called a secondary database. In a secondary database, the keys are your alternative (or secondary) index, and the data corresponds to a primary record's key. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Questions on ZODB BTrees versus bsddb BTrees
Terry Jones wrote: I've just read the ZODB/ZEO Programming Guide (Release 3.6.0, A.M. Kuchling, January 5, 2006) and I'd like to ask a few questions. Are the following all correct? You may have more joy with these questions on the zodb-dev@zope.org mailing list... - The BTrees supported by ZODB have nothing to do with Berkeley DB Btrees (apart from the fact that they're both implementations of BTrees). Correct. And I think the ZODB ones are a variant of BTree... - ZODB BTrees do not allow duplicate keys (a la bsddb). Correct. - ZODB BTrees do not allow you to set up secondary keys (a la bsddb). Dunno what that means. I think you're just use two BTrees for this... - Is the byValue() ZODB BTree method officially deprecated? Scheduled for deprecation / removal? Never used it, no idea... Assuming the first three above are correct, and that I would like both duplicate keys What do you mean by duplicate keys? This sounds like a fairly screwy concept for a mapping... Do you mean you want to map one key to a set of values or that you want to map more than one key to the same value? and secondary keys Again, you need to explain what you mean by secondary keys... 1. Use both ZODB and bsddb in the same application, dealing with things like locking and transactions across both object types in a somewhat manual way. Oooo.. feel the pain... This would have the additional major downside of making it harder to use ZEO for storage, right? (Yes, I could pickle the entire bsddb databases and hand them to ZEO, but that's not really what I had in mind.) That would be insane ;-) 2. Try to add support for bsddb BTrees (with duplicate keys) to ZODB. Doubt you'd want to do that ;-) 3. Redesign what I'm doing so as to only need ZODB BTrees. This is quite far from optimal (in terms of space and time), but might be the quickest to implement. I think there are things that you can do with BTrees and their Set objects that will solve any problems you have. You need to explain what it is you're trying to do and we can help you... If someone with more knowledge of ZODB and the BTrees therein and bsddb feels like confirming, correcting, or commenting on the above summary, and options, that would be great. Take a look at the code in ZCatalog and PluginIndexes. I think you'll find everything you're looking for... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
