Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
On Jun 3, 2008, at 10:52 AM, Martijn Faassen wrote: Hi there, On Tue, Jun 3, 2008 at 3:28 PM, Sean Allen [EMAIL PROTECTED] wrote: [snip] i dont think you can compare ZODB to gemstone's products. i've looked at both over the last few months with a decent level of depth and the gemstone from a programmer's standpoint is infintately easier to use. no special classes, no nothing really. just put anything persistent in a global dictionary and blam its done. The ZODB can be used that way; the root object *is* a global dictionary. Admittedly Persistent is necessary to make sure that attributes changes are persisted; this may be nicer in Gemstone. Or do you mean any global dictionaries are persisted? That part is the same. What is different is that persistence is built into their vm. You just put anything into any global and its persisted. How does Gemstone implement efficient querying or indexing? under the hood or application level? there are two on the application level. you can use standard collection methods on the persistent instances, this isnt indexed OR you can use their custom block statement syntax ( i'm talking just the smalltalk now ) to work specifically on indexes. indexes are established by sending a message to the class of whatever you want to index. under the hood? i haven't looked into the specific details of how they wire it altogether but it comes down to, gemstone is a fullstack. whether you are using the smalltalk, java or eventual ruby... they write the vm which has primitives to make their ops fast, has built in persistence so you just dont think about it at all. in fact, you have to ask for a class to not be persistent. I know the PyPy people have a demo where multiple interpreters share objects transparently, so perhaps this is closer to what Gemstone does. that would be closer yeah on some level. it goes back to that full stack thing. gemstone isnt just an oodb. its an oodb, its a scaling solution, its a vm. theres an object store shared by all. there are multiple vms instances running whatever code and the entire thing can run across multiple machines... need to scale, add more machines in. i'm still digging into it all, its only been 3 weeks so i still have a lot of the terminology wrong etc, but it really is a very cool product. not having to think about the data store is just real nice. its all just objects and you dont have to change anything about how you code for them unless you want to use indexes and then the changes are very minor. pull it back out and there it is again, object pointers fully intact. store in 2 different directories, modify in one, blam! modified in the other. I'm not sure how this is different than using the root object to store objects and ZEO? if i have customer A who has order B and i store customer A to customer dictionary and order B to order dictionary then later access order B from order dictionary, modify and update it does ZEO update the instance of order pointed to by customer A? I cant get it to do it. My understanding is it cant. Well, it could but it isnt 'right out of the box' seamless. If you do that in gemstone, there is only one copy of Order B, no matter what variable in what dictionary you come at it from. And its drop dead simple. I looked at implementing that with zodb and moved along. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
On Tue, Jun 3, 2008 at 5:28 PM, Sean Allen [EMAIL PROTECTED] wrote: What is different is that persistence is built into their vm. You just put anything into any global and its persisted. Yeah, that's nice. I want the whole operating system to work like that. ;) -- Lennart Regebro: Zope and Plone consulting. http://www.colliberty.com/ +33 661 58 14 64 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
Hi there, On Tue, Jun 3, 2008 at 3:28 PM, Sean Allen [EMAIL PROTECTED] wrote: [snip] i dont think you can compare ZODB to gemstone's products. i've looked at both over the last few months with a decent level of depth and the gemstone from a programmer's standpoint is infintately easier to use. no special classes, no nothing really. just put anything persistent in a global dictionary and blam its done. The ZODB can be used that way; the root object *is* a global dictionary. Admittedly Persistent is necessary to make sure that attributes changes are persisted; this may be nicer in Gemstone. Or do you mean any global dictionaries are persisted? How does Gemstone implement efficient querying or indexing? I know the PyPy people have a demo where multiple interpreters share objects transparently, so perhaps this is closer to what Gemstone does. pull it back out and there it is again, object pointers fully intact. store in 2 different directories, modify in one, blam! modified in the other. I'm not sure how this is different than using the root object to store objects and ZEO? Regards, Martijn ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
Hey, On Tue, Jun 3, 2008 at 6:09 PM, Tim Cook [EMAIL PROTECTED] wrote: [snip] But that's a feature and not a limitation. :-) If I store patient A in demographics and a clinical entry B in ehr When I edit the clinical entry B my DBM had better NOT update B. It is supposed to create a new clinical entry that is reference as a later version of B. I don't get it what you're saying here. The ZODB has transparent references. An OODB should be like a normal collection of Python objects, and references are transparent there. Regards, Martijn ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
How does Gemstone implement efficient querying or indexing? [snip] Okay, this sounds like an indexing framework built into the database layer, something the ZODB doesn't have, but of course has been built on top with the catalog. pretty much. i haven't looked into the specific details of how they wire it altogether but it comes down to, gemstone is a fullstack. whether you are using the smalltalk, java or eventual ruby... they write the vm which has primitives to make their ops fast, has built in persistence so you just dont think about it at all. in fact, you have to ask for a class to not be persistent. Fullstack has its advantages, though also disadvantages. It means they need to reimplement compliant interpreters for any language they want to support, and that's going to hurt their library support. (as I doubt arbitrary CPython extensions would work with a hypothetical Python version of this) indeed they would have to. disadvantage for them. as an end user, i dont see that as really hurting me. [snip] theres an object store shared by all. there are multiple vms instances running whatever code and the entire thing can run across multiple machines... need to scale, add more machines in. This is something ZEO also provides, as far as I can grasp from your description. indeed it does. i'm still digging into it all, its only been 3 weeks so i still have a lot of the terminology wrong etc, but it really is a very cool product. not having to think about the data store is just real nice. its all just objects and you dont have to change anything about how you code for them unless you want to use indexes and then the changes are very minor. I'd say that the ZODB by itself also doesn't put heavy requirements on your code. The main thing is the subclassing from Persistent, and _p_changed flags if you use non-persistent subobjects you still want to persist. For indexing, a framework like Zope 3 requires zero changes to the classes themselves. pull it back out and there it is again, object pointers fully intact. store in 2 different directories, modify in one, blam! modified in the other. I'm not sure how this is different than using the root object to store objects and ZEO? if i have customer A who has order B and i store customer A to customer dictionary and order B to order dictionary then later access order B from order dictionary, modify and update it does ZEO update the instance of order pointed to by customer A? I cant get it to do it. My understanding is it cant. Well, it could but it isnt 'right out of the box' seamless. ZEO should do just that. I understand you have an object A which has a reference to B. You also have a dictionary that has a reference to A, and a dictionary that has a reference to B. Both A and the dictionary will be pointing to the same instance of B. (if A and B are both subclasses of Persistent. If not, it might be both serialize separately, I'm not sure). If you do that in gemstone, there is only one copy of Order B, no matter what variable in what dictionary you come at it from. And its drop dead simple. I looked at implementing that with zodb and moved along. I'm confused. This has been the way the ZODB worked for a long time, unless I'm really missing something in your description. i tried to do this: create customer that has order so that i can have different extents type situations... store customer in one dictionary. store order in another. if i pulled the order back out from the order dictionary and modified it then pulled the customer out, the customers order was no longer in sync with what came out of the order dictionary. the reference was lost on serialization. original in memory objects were fine, those that came back out from zodb werent. i'm going to quote the initial email i sent with the idea in general and the followup i got and i then tried it out to make sure i hadnt asked the question wrong, and yeah... what i wanted to do, wasnt easily done. the quotes: The biggest concern I have is how do to the layout/storage so that this slightly contrived example works: Product has a brand. There are many brands. How do I store so that I can find all products simply and all brands simply and also so that changes in a brand instance are reflected when the product instance is deserialized. By 'simply' I mean that it doesnt really work on our end to have to walk all Products looking for unique brands. Should just be able to go directly in and get said brands ( using keys() or similar call ). If I create 'brand' and 'product' as btrees, then if i do something like some_product.brand.name = 'something entirely different' and that brand already exists in 'brand', would it be updated? are references maintained in that fashion? do we have to handle manually on update and creation? Note that we would just be using ZODB not Zope in this scenario. Back
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
On Tue, Jun 3, 2008 at 1:13 PM, Gary Poster [EMAIL PROTECTED] wrote: What you *can't* do out of the box is ask hey, what products have an attribute that points to this brand?. That's a back-reference, and that needs solutions like the ones to which I was referring. In other words: if you want an index of references (or any attribute for that matter), you have to set it up (just as you would for a RDBMS). -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
Uh-oh, I'm implicated! (see below) On Jun 3, 2008, at 12:53 PM, Sean Allen wrote: ... If you do that in gemstone, there is only one copy of Order B, no matter what variable in what dictionary you come at it from. And its drop dead simple. I looked at implementing that with zodb and moved along. I'm confused. This has been the way the ZODB worked for a long time, unless I'm really missing something in your description. i tried to do this: create customer that has order so that i can have different extents type situations... store customer in one dictionary. store order in another. if i pulled the order back out from the order dictionary and modified it then pulled the customer out, the customers order was no longer in sync with what came out of the order dictionary. the reference was lost on serialization. original in memory objects were fine, those that came back out from zodb werent. i'm going to quote the initial email i sent with the idea in general and the followup i got and i then tried it out to make sure i hadnt asked the question wrong, and yeah... what i wanted to do, wasnt easily done. the quotes: The biggest concern I have is how do to the layout/storage so that this slightly contrived example works: Product has a brand. There are many brands. How do I store so that I can find all products simply and all brands simply and also so that changes in a brand instance are reflected when the product instance is deserialized. By 'simply' I mean that it doesnt really work on our end to have to walk all Products looking for unique brands. Should just be able to go directly in and get said brands ( using keys() or similar call ). If I create 'brand' and 'product' as btrees, then if i do something like some_product.brand.name = 'something entirely different' and that brand already exists in 'brand', would it be updated? are references maintained in that fashion? do we have to handle manually on update and creation? Note that we would just be using ZODB not Zope in this scenario. Back references are not maintained automatically. I'd identify two classic solutions to this sort of thing. One is to make a custom mapping (using a BTree as the inner data structure) that maintains back-references when objects are placed in them or removed. zope.app(.container? .folder? I'd have to look) has code that does this, along with firing events. For simple stories like the one you describe here, that's what I'd probably recommend. It works to the strengths of the ZODB, which particularly shines in terms of readability when you just need to walk a tree of attributes to get what you want. The other is to keep an external index, a la zc.extrinsicreference or zc.relation. zc.extrinsicreference does not have too many dependencies beyond ZODB, and as long as zope.app.keyreference doesn't drag much along with it, might be usable as a library. That said, it's also very simple, and could be used as a model for you, even if you don't use it directly. It would also be a reasonable choice for a simple situation like the one you describe. It relies on events to update its data structures. zc.relation an almost-released-revision of zc.relationship that drastically reduces dependencies--actually, it has no additional dependencies to ZODB, as you can see at http://svn.zope.org/zc.relation/trunk/setup.py?view=markup . It's also a bit overwhelming and low-level: see the README:http://svn.zope.org/zc.relation/trunk/src/zc/relation/README.txt?view=auto . It doesn't hook anything up for you: you set the relationship catalog up and you arrange for it to be updated, via events or direct messages. That said, if you need its power, it is well- tested and would be a good choice for some jobs from at least some perspectives (caveat read-or: I'm the author). Now in the context of this discussion, I see that I misread you. I apologize. This works out of the box: You have a Product class and a Brand class. Both inherit from persistent.Persistent. You have a persistent-aware mapping such as a BTree. In the root of your ZODB, put two BTrees, one for your products and one for your brands. Create some Brand instances and put them in the brand mapping. Create some Product instances and put them in the product mapping. Assign some of the brands you have made as attributes of the products. Now, product.brand.foo = 'bar' (in any thread or any ZEO client) will be changing the same effective object (let's call it a record) as the one in the brand mapping you have in the root of your db. I believe that this is what you are talking about. Again, I apologize for not reading your original question closely enough. What you *can't* do out of the box is ask hey, what products have an attribute that points to this brand?. That's a back-reference, and that needs solutions like the ones to which I was
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
On Jun 3, 2008, at 1:32 PM, Benji York wrote: On Tue, Jun 3, 2008 at 1:13 PM, Gary Poster [EMAIL PROTECTED] wrote: What you *can't* do out of the box is ask hey, what products have an attribute that points to this brand?. That's a back-reference, and that needs solutions like the ones to which I was referring. In other words: if you want an index of references (or any attribute for that matter), you have to set it up (just as you would for a RDBMS). but i could just loop over all the products checking the brand. and if that statement doesnt set off alarm bells then i feel comfortable in my understanding if not how i managed to 1. misinterpret something and 2. how i managed to write code that actually supported that misinterpretation. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
On Jun 3, 2008, at 4:06 PM, Benji York wrote: On Tue, Jun 3, 2008 at 3:45 PM, Sean Allen [EMAIL PROTECTED] wrote: On Jun 3, 2008, at 1:32 PM, Benji York wrote: On Tue, Jun 3, 2008 at 1:13 PM, Gary Poster [EMAIL PROTECTED] wrote: What you *can't* do out of the box is ask hey, what products have an attribute that points to this brand?. That's a back-reference, and that needs solutions like the ones to which I was referring. In other words: if you want an index of references (or any attribute for that matter), you have to set it up (just as you would for a RDBMS). but i could just loop over all the products checking the brand. You could... and if that statement doesnt set off alarm bells then i feel comfortable in my understanding ...but you'd kill kittens, er no, you'd take an inordinate amount of time to find things. That's what indexes are for, after all. unless it was a rather limited list of items... if not how i managed to 1. misinterpret something and 2. how i managed to write code that actually supported that misinterpretation. I can't tell from here. ;) neither can i til i dig up the code. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
Martijn Faassen wrote: Hi there, Christian Theune wrote: this might be interesting to ZODB users and developers: http://rss.slashdot.org/~r/Slashdot/slashdot/~3/302177093/article.pl What I find interesting is that Python has had such a thing for about a decade (the ZODB), and a mostly vaporware announcement in the Ruby world makes such a splash. That's because most things Zope are not very good with PR. The future is already here folks. So, what's the status of the zodb.zope.org project to actually promote it better? It's easy to know what to write on the homepage, just go to the Ruby buzz, translate the hype in terms of the ZODB, tone it down some, and add that it's been battle-tested for a decade. I assume you mean http://new.zope.org/projects/zodb ? It's there, waiting for the rest of that site to be finished. Wichert. -- Wichert Akkerman[EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
Hi there, On Mon, Jun 2, 2008 at 3:19 PM, Wichert Akkerman [EMAIL PROTECTED] wrote: Martijn Faassen wrote: Christian Theune wrote: this might be interesting to ZODB users and developers: http://rss.slashdot.org/~r/Slashdot/slashdot/~3/302177093/article.pl What I find interesting is that Python has had such a thing for about a decade (the ZODB), and a mostly vaporware announcement in the Ruby world makes such a splash. That's because most things Zope are not very good with PR. I know what the reason is. I just want to point it out again. Not being good at it is not an excuse not to get better. It can be done, it's just that it's not been a priority in people's minds. The future is already here folks. So, what's the status of the zodb.zope.org project to actually promote it better? It's easy to know what to write on the homepage, just go to the Ruby buzz, translate the hype inI terms of the ZODB, tone it down some, and add that it's been battle-tested for a decade. I assume you mean http://new.zope.org/projects/zodb ? It's there, waiting for the rest of that site to be finished. Progress! Good. Some criticisms next: The site is nothing compared to what it could be concerning documentation. It's currently consists of mostly a huge amount of empty pages of documentation that will intimidate anyone who wants to work on this site. I strongly recommend doing it the other way around: mine the existing documentation and assemble this into a site, and don't intimidate people with a lot of empty structure. It also holds back the whole new.zope.org project itself, as lots of empty pages will make people wait to publish the site until it's all filled, meaning it will never actually happen. Is anyone actually approaching ZODB developers to fill in pieces? The only way any ZODB site is going to improve if people on this very mailing list will actually do something about it. It won't be me. Finally, I still think this project is big enough to warrant its own site, independent from the rest of zope.org. If you did this, you wouldn't need the red letters that this can be used independently from Zope, it'd be implicit, and people would really trust that this is indeed the case instead of the nagging suspicion I think they'll end up with now. What I would do (but I'm not in charge of this and don't want to be) is to put up a zodb.zope.org with what's there now, give everybody who wants to add documentation access to add content, and then nag until it contains some. Probably the current set of two real pages would be enough to embaress people into adding more. Regards, Martijn P.S. Wichert, I know you're doing good work. I'm just hoping someone on this mailing list is actually listening and wants to coordinate this effort (and then actually does it). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
Wichert Akkerman wrote: Martijn Faassen wrote: What I find interesting is that Python has had such a thing for about a decade (the ZODB), and a mostly vaporware announcement in the Ruby world makes such a splash. That's because most things Zope are not very good with PR. Smart people are usually very demanding with themselves, which leads to the bad pattern of not advertizing our good stuff because it is not perfect. Time to change our patterns, and advertize stuff before they are perfect ;-) -- Godefroid Chapelle (aka __gotcha) http://bubblenet.be ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
On Mon, 2008-06-02 at 15:45 +0200, Martijn Faassen wrote: Lots documentation on the ZODB is available, scattered around the web. Let's gather it into one place for starters. So, please, someone actually do something about this rotten situation? Finally? Please? Please? Regards, Martijn P.S. This is not a complaint to all the people who are doing something. We just need a bit of effort from some more people to get this done. I'm feeling some guilt pangs here. :-) I benefit from Zope and the ZODB especially. I know that I am a little better at PR than I am at the deep technical aspects (not very good) so I will initially gather (I already have lots of bookmarks) all the info I can find and start building something coherent out of it. Can I have a few minutes of attention to emails now and then from experts? I am NOT good at the flashy layout and graphics stuff that attracts the attention of people. I will needs some help in that respect. But I can organize information into something that new-comers can learn from. Regards, Tim -- Timothy Cook, MSc Health Informatics Research Development Services LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook Skype ID == timothy.cook ** *You may get my Public GPG key from popular keyservers or * *from this link http://timothywayne.cook.googlepages.com/home* ** signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
Tim Cook wrote: On Mon, 2008-06-02 at 15:45 +0200, Martijn Faassen wrote: Lots documentation on the ZODB is available, scattered around the web. Let's gather it into one place for starters. So, please, someone actually do something about this rotten situation? Finally? Please? Please? Regards, Martijn P.S. This is not a complaint to all the people who are doing something. We just need a bit of effort from some more people to get this done. I'm feeling some guilt pangs here. :-) I benefit from Zope and the ZODB especially. I know that I am a little better at PR than I am at the deep technical aspects (not very good) so I will initially gather (I already have lots of bookmarks) all the info I can find and start building something coherent out of it. Can I have a few minutes of attention to emails now and then from experts? Of course. Tech people are good at finding faults in documentation, but not good at writing documentation. I am NOT good at the flashy layout and graphics stuff that attracts the attention of people. I will needs some help in that respect. But I can organize information into something that new-comers can learn from. We already have a complete design with proper layouts on new.zope.org. We just need more content. Wichert. -- Wichert Akkerman[EMAIL PROTECTED]It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Ruby/Smalltalk OODB
On Mon, Jun 2, 2008 at 12:31 PM, Tim Cook [EMAIL PROTECTED] wrote: I benefit from Zope and the ZODB especially. I know that I am a little better at PR than I am at the deep technical aspects (not very good) so I will initially gather (I already have lots of bookmarks) all the info I can find and start building something coherent out of it. Great! Can I have a few minutes of attention to emails now and then from experts? I'm not a ZODB expert, but I'll do what I can to help you. I am NOT good at the flashy layout and graphics stuff that attracts the attention of people. I will needs some help in that respect. organize information into something that new-comers can lear Hopefully the already underway design effort will help there. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
