Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Sun, 2008-03-23 at 17:25 -0400, Sean Allen wrote: I have not as it seemed rather Zope specific. Are there particular parts I could zero in on as being particularly relevant? I don't know enough about your application to say. If it is a web application then quite a lot is relevant especially the form and schema stuff. In general though I would not want to build an application without a component architecture and that pretty much means I'll use Zope. I have read the zodb/zeo guide as well as several other things my googling turned up plus some misc other documents from zope.org that seemed applicable. I can see that you are trying hard to find answers so if you don't please continue sending your questions to the list. The big problem I have is that all the zodb stuff i have found is either very trivial or tied up zope. Zope by this point in time, is a highly advanced and complicated system from the outside. Having to slog through zope specific stuff trying to pick out zodb details is frustrating. I agree that having a basic ZODB example that doesn't require Zope will help. So let me know if there are still gaps in your understanding and if you still need help with the indexing and searching of objects. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Mar 24, 2008, at 3:40 AM, Roché Compaan wrote: On Sun, 2008-03-23 at 17:25 -0400, Sean Allen wrote: I have not as it seemed rather Zope specific. Are there particular parts I could zero in on as being particularly relevant? I don't know enough about your application to say. If it is a web application then quite a lot is relevant especially the form and schema stuff. In general though I would not want to build an application without a component architecture and that pretty much means I'll use Zope. I have read the zodb/zeo guide as well as several other things my googling turned up plus some misc other documents from zope.org that seemed applicable. I can see that you are trying hard to find answers so if you don't please continue sending your questions to the list. The big problem I have is that all the zodb stuff i have found is either very trivial or tied up zope. Zope by this point in time, is a highly advanced and complicated system from the outside. Having to slog through zope specific stuff trying to pick out zodb details is frustrating. I agree that having a basic ZODB example that doesn't require Zope will help. So let me know if there are still gaps in your understanding and if you still need help with the indexing and searching of objects. I think I have enough know to start playing with, the searching and indexing wouldnt come for a while anyway. I think what I am going to do right now is spend some time mocking up the basic chain of objects for part of the application, moving all the data over from the rdbms for those parts and start working out our most complicated queries to see if there are going to be serious roadblocks. Thanks for your help so far, I'm sure I shall return at some point. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Sat, 2008-03-22 at 15:49 -0400, Sean Allen wrote: Ha! Ok. Questions then from there. Why not store each object type, customer, order etc in their own folders? You could, and I understand that you want to do so coming from a RDBMS world, but if you are working with objects then you have the luxury of organising content in a way that is closer to reality. I think it is rather convenient to store a customer's orders inside it. I can see the convenience. Ok, so here is a big question... Assuming you did store each object in its own folder and you had Customer has many orders At what point do you have to hook into zodb so that if you say did this: customer = Customer( ... ) customer.addOrder( Order( ... ) ) that the order in question when customer is saved, gets saved to its own folder and then customer gets a reference to that now saved persistent order before it itself is saved. You first create the order inside the orders folder, then you reference it from the customer: order = Order() orderfolder['order123'] = order # assuming you have a folder for orders in each customer, the following # will create a reference since the above code already persisted the # object. customer.orders['order123'] = order I don't see why you need the reference though. Why not just create the order directly in the order folder for the relevant customer: order = Order() customer.orders['order123'] = order There is another good reason - it makes a lot more sense to distribute object creation to different containers in the ZODB to avoid write conflict errors. I think I understand this but can you elaborate? By 'distribute object creation to different containers' do you mean store each object in own folders? No that would lead to an unnecessary proliferation of folders and it wouldn't actually remove conflict errors. Conflicts errors occur when two concurrent transactions modify the same attribute. When you are putting objects in a folder you are essentially modifying the same attribute (unless you are using a btree). A good application design strategy for the ZODB is to ensure that concurrent transactions are modifying attributes on different objects. If you are putting all your orders in one folder you are increasing the chances of conflict errors. Like I mentioned above you could use a btree folder since it has a fairly good conflict resolution strategy. But given a high enough insertion rate even a btree will not protect you against conflict errors. I would recommend you create orders inside a btree-based order folder inside each customer. This will significantly reduce the likeliness of conflict errors. is there a reason to have a folder that contains a more complete graph? what advantages would that have? speed of access? What do you mean with more complete graph? storing line items inside orders inside customers type scenario Access and insertion times for items in folders with fewer items are faster and like I mentioned above, distributing objects writes reduces conflict errors -- the database i am looking at moving over has 1.2 million entries in a transactions table 980,000 orders 775,000 customers 1.5 million order items I think these numbers are quite manageable. But think carefully what kind of queries you want to do on the data. You have a very rich query language in SQL that allows the construction of complex queries and it will come naturally to you. To do the same in the ZODB will take careful planning. Are there any mistakes that people usually make when doing this sort of mental context switch that you can make me aware of now? Any good reading on the subject? Either there isnt a ton of information on this on the net or my google skills are slipping. The most valuable resource is the mailing list. Unfortunately nobody has documented their experience elsewhere before. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Mar 23, 2008, at 3:59 AM, Roché Compaan wrote: On Sat, 2008-03-22 at 15:49 -0400, Sean Allen wrote: Ha! Ok. Questions then from there. Why not store each object type, customer, order etc in their own folders? You could, and I understand that you want to do so coming from a RDBMS world, but if you are working with objects then you have the luxury of organising content in a way that is closer to reality. I think it is rather convenient to store a customer's orders inside it. I can see the convenience. Ok, so here is a big question... Assuming you did store each object in its own folder and you had Customer has many orders At what point do you have to hook into zodb so that if you say did this: customer = Customer( ... ) customer.addOrder( Order( ... ) ) that the order in question when customer is saved, gets saved to its own folder and then customer gets a reference to that now saved persistent order before it itself is saved. You first create the order inside the orders folder, then you reference it from the customer: order = Order() orderfolder['order123'] = order # assuming you have a folder for orders in each customer, the following # will create a reference since the above code already persisted the # object. customer.orders['order123'] = order I don't see why you need the reference though. Why not just create the order directly in the order folder for the relevant customer: order = Order() customer.orders['order123'] = order Can that be wrapped in a transaction so that if something goes wrong with customer creation, the order gets rolled back as well? Or put higher level, do transactions only work within a single folder or across folders? There is another good reason - it makes a lot more sense to distribute object creation to different containers in the ZODB to avoid write conflict errors. I think I understand this but can you elaborate? By 'distribute object creation to different containers' do you mean store each object in own folders? No that would lead to an unnecessary proliferation of folders and it wouldn't actually remove conflict errors. Conflicts errors occur when two concurrent transactions modify the same attribute. When you are putting objects in a folder you are essentially modifying the same attribute (unless you are using a btree). A good application design strategy for the ZODB is to ensure that concurrent transactions are modifying attributes on different objects. If you are putting all your orders in one folder you are increasing the chances of conflict errors. If you dont put all orders in a single folder, when it comes time to search orders, how would you know where to find? Do you sequentially search the multiple folders for a given object? Like I mentioned above you could use a btree folder since it has a fairly good conflict resolution strategy. But given a high enough insertion rate even a btree will not protect you against conflict errors. I would recommend you create orders inside a btree-based order folder inside each customer. This will significantly reduce the likeliness of conflict errors. So if this occurs, you get an exception ( basically optimistic locking ) which you then deal with on a higher level in your application? -- the database i am looking at moving over has 1.2 million entries in a transactions table 980,000 orders 775,000 customers 1.5 million order items I think these numbers are quite manageable. But think carefully what kind of queries you want to do on the data. You have a very rich query language in SQL that allows the construction of complex queries and it will come naturally to you. To do the same in the ZODB will take careful planning. Are there any mistakes that people usually make when doing this sort of mental context switch that you can make me aware of now? Any good reading on the subject? Either there isnt a ton of information on this on the net or my google skills are slipping. The most valuable resource is the mailing list. Unfortunately nobody has documented their experience elsewhere before. Well, if I go with zeo/zodb and python, I'll be sure to document the entire thing. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Sun, 2008-03-23 at 11:38 -0400, Sean Allen wrote: Can that be wrapped in a transaction so that if something goes wrong with customer creation, the order gets rolled back as well? Or put higher level, do transactions only work within a single folder or across folders? You can modify as many objects as you like in a transaction. Have you read the ZODB/ZEO programming guide yet? http://www.zope.org/Wikis/ZODB/guide/zodb.html There is another good reason - it makes a lot more sense to distribute object creation to different containers in the ZODB to avoid write conflict errors. I think I understand this but can you elaborate? By 'distribute object creation to different containers' do you mean store each object in own folders? No that would lead to an unnecessary proliferation of folders and it wouldn't actually remove conflict errors. Conflicts errors occur when two concurrent transactions modify the same attribute. When you are putting objects in a folder you are essentially modifying the same attribute (unless you are using a btree). A good application design strategy for the ZODB is to ensure that concurrent transactions are modifying attributes on different objects. If you are putting all your orders in one folder you are increasing the chances of conflict errors. If you dont put all orders in a single folder, when it comes time to search orders, how would you know where to find? Do you sequentially search the multiple folders for a given object? No. You index the order attributes that you want to search on and perform a search on the indexes. Have a look at zc.index and zc.catalog http://svn.zope.org/zc.index/ http://svn.zope.org/zc.catalog/ I think you will also benefit a lot by looking at Zope as development platform. Have a look at the Zope3Book: http://wiki.zope.org/zope3/Zope3Book Like I mentioned above you could use a btree folder since it has a fairly good conflict resolution strategy. But given a high enough insertion rate even a btree will not protect you against conflict errors. I would recommend you create orders inside a btree-based order folder inside each customer. This will significantly reduce the likeliness of conflict errors. So if this occurs, you get an exception ( basically optimistic locking ) which you then deal with on a higher level in your application? Yes. The most valuable resource is the mailing list. Unfortunately nobody has documented their experience elsewhere before. Well, if I go with zeo/zodb and python, I'll be sure to document the entire thing. Cool! We really need that! -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Mar 23, 2008, at 1:30 PM, Roché Compaan wrote: On Sun, 2008-03-23 at 11:38 -0400, Sean Allen wrote: Can that be wrapped in a transaction so that if something goes wrong with customer creation, the order gets rolled back as well? Or put higher level, do transactions only work within a single folder or across folders? You can modify as many objects as you like in a transaction. Have you read the ZODB/ZEO programming guide yet? http://www.zope.org/Wikis/ZODB/guide/zodb.html yes but the examples are always about objects in a single folder. wasnt sure if different folders would have any impact. i assumed not, but you know where assumptions get you. There is another good reason - it makes a lot more sense to distribute object creation to different containers in the ZODB to avoid write conflict errors. I think I understand this but can you elaborate? By 'distribute object creation to different containers' do you mean store each object in own folders? No that would lead to an unnecessary proliferation of folders and it wouldn't actually remove conflict errors. Conflicts errors occur when two concurrent transactions modify the same attribute. When you are putting objects in a folder you are essentially modifying the same attribute (unless you are using a btree). A good application design strategy for the ZODB is to ensure that concurrent transactions are modifying attributes on different objects. If you are putting all your orders in one folder you are increasing the chances of conflict errors. If you dont put all orders in a single folder, when it comes time to search orders, how would you know where to find? Do you sequentially search the multiple folders for a given object? No. You index the order attributes that you want to search on and perform a search on the indexes. Have a look at zc.index and zc.catalog http://svn.zope.org/zc.index/ http://svn.zope.org/zc.catalog/ I think you will also benefit a lot by looking at Zope as development platform. Have a look at the Zope3Book: http://wiki.zope.org/zope3/Zope3Book I have not as it seemed rather Zope specific. Are there particular parts I could zero in on as being particularly relevant? When I review the index, there isnt anything that jumps out and says, o read me. I have read the zodb/zeo guide as well as several other things my googling turned up plus some misc other documents from zope.org that seemed applicable. The big problem I have is that all the zodb stuff i have found is either very trivial or tied up zope. Zope by this point in time, is a highly advanced and complicated system from the outside. Having to slog through zope specific stuff trying to pick out zodb details is frustrating. If I go with zodb as a backend, I would intend to help others who come in the future by providing more zodb specific docs that arent at all tied to zope. ( I've actually come across many blog point that make that specific point ). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Fri, 2008-03-21 at 22:49 -0400, Sean Allen wrote: And we want to change the the description of Widget B and have that change appear for everything, how do I do that? basically, I want all LineItems for Widget A to all refer to a single instance of Product. If you simply assign the product to an attribute on the line item, it will create a reference in the ZODB. And you can use zc.relation to index and query the relation. # get the product from your products container: product = productfolder['ProductXYZ'] # assign it to the line item instance lineitem.product = product It's really that simple! If you change the description attribute of the referenced product, all line items referencing it will show the new description. In a relational database, that is easy... there are 3 entries in the Products table and there is a foreign key in LineItems. Here, I have no idea how to handle. Which makes me think that I'm missing some fundamental point. This is what zc.relation is for. On your line item instance, you assign to a attribute that holds the product reference and you use zc.relation to index and query it. I'm sure the answer to this will start a whole slew of new questions, so thanks in advance for the help with this and going forward. Its very frustrating to keep beating my head against these conceptual points that I'm just not getting. Usually, I have eureka moments after spending this much time on something, this time, its just not coming. Don't worry you'll get it soon :-) ZODB programming is easier than people expect so they don't easily accept it ;-) Stepping back somewhat, I'll structure the app like this. Have containers in the root of your app for customers and products. Store customers in your customer container, orders inside the customer and line items in the order: Customers | +--- Customer | +- Order | +--- Line item Products | + Product Hope that helps. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Sat, 2008-03-22 at 04:55 -0400, Sean Allen wrote: On Mar 22, 2008, at 3:50 AM, Roché Compaan wrote: On Fri, 2008-03-21 at 22:49 -0400, Sean Allen wrote: And we want to change the the description of Widget B and have that change appear for everything, how do I do that? basically, I want all LineItems for Widget A to all refer to a single instance of Product. If you simply assign the product to an attribute on the line item, it will create a reference in the ZODB. And you can use zc.relation to index and query the relation. # get the product from your products container: product = productfolder['ProductXYZ'] # assign it to the line item instance lineitem.product = product It's really that simple! If you change the description attribute of the referenced product, all line items referencing it will show the new description. Ha! Ok. Questions then from there. Why not store each object type, customer, order etc in their own folders? You could, and I understand that you want to do so coming from a RDBMS world, but if you are working with objects then you have the luxury of organising content in a way that is closer to reality. I think it is rather convenient to store a customer's orders inside it. There is another good reason - it makes a lot more sense to distribute object creation to different containers in the ZODB to avoid write conflict errors. is there a reason to have a folder that contains a more complete graph? what advantages would that have? speed of access? What do you mean with more complete graph? -- the database i am looking at moving over has 1.2 million entries in a transactions table 980,000 orders 775,000 customers 1.5 million order items I think these numbers are quite manageable. But think carefully what kind of queries you want to do on the data. You have a very rich query language in SQL that allows the construction of complex queries and it will come naturally to you. To do the same in the ZODB will take careful planning. and probably another 600,000 other assorted entires that would be needed in an oo scenario ( tons of join table entries get ditched tho, nice! ) what type of server setup etc would be needed to handle that load considering that currently, 40,000 new orders go in a month along with the corresponding numbers of customers etc. It depends a lot on the concurrency. It is usually advisable to not compromise on server architecture, but be sure to get a server with enough RAM (nothing less than 4GB) and very fast disks. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Mar 20, 2008, at 11:04 PM, Sean Allen wrote: I've been playing with this and I'm still having a disconnect 3 week later. 1. I love the zc.relation stuff it answers one area of stuff I hadn't even gotten to yet. cool :-) 2. I'm just totally missing something less cool. :-/ If I have Customers and Orders and I want to be able update all of them independently of their relationships ( so that if an order is updated, when i get the customer at some later time, it has the updated order amongst that relation ) but I can't figure it out. I tried looking at the zope.app.folder stuff but I keep getting lost in the zope aspects of it and am having a hard time seeing the forest for the trees. Is there some more general ready on the patterns used that you know of? I think if I understood the idea more in an abstract sense, I could get a lot more out of the folder implementation. Here's a simple, dumb example that parallels the folder stuff. See the Dict class in http://svn.zope.org/zc.async/branches/dev/src/zc/async/utils.py?rev=84657view=auto . You put something in the dict, and __setitem__ slams a name and a parent. The other mutating methods should do the right thing as well in terms of updating the back reference. So, completely randomly and arbitrarily, but to try and make a parallel, what if customers were a dict of orders, and when you made an order it was associated with only one customer, and you put the order in the customer. This may be bizarre--what's the key? can more than one customer be associated with an order? But that would mean that customer.values() would get all of the customer's orders, and order.parent would be the associated customer. This example subclasses zc.dict, a super simple package that only depends on ZODB, btw. It does not have full dict behavior, as Jim likes to point out, because items are stored by BTree sorting, not hashes, but it looks like a dict otherwise. Maybe it can help you out. If you want to try out this example, copy the code out and get the zc.dict egg and give it a try. Anyway, this pattern of directly manipulating back-references is good for intrinsic relationships like customers and orders. Hope that helps a bit, Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Mar 21, 2008, at 8:41 PM, Gary Poster wrote: On Mar 20, 2008, at 11:04 PM, Sean Allen wrote: I've been playing with this and I'm still having a disconnect 3 week later. 1. I love the zc.relation stuff it answers one area of stuff I hadn't even gotten to yet. cool :-) 2. I'm just totally missing something less cool. :-/ If I have Customers and Orders and I want to be able update all of them independently of their relationships ( so that if an order is updated, when i get the customer at some later time, it has the updated order amongst that relation ) but I can't figure it out. I tried looking at the zope.app.folder stuff but I keep getting lost in the zope aspects of it and am having a hard time seeing the forest for the trees. Is there some more general ready on the patterns used that you know of? I think if I understood the idea more in an abstract sense, I could get a lot more out of the folder implementation. Here's a simple, dumb example that parallels the folder stuff. See the Dict class in http://svn.zope.org/zc.async/branches/dev/src/zc/async/utils.py?rev=84657view=auto . You put something in the dict, and __setitem__ slams a name and a parent. The other mutating methods should do the right thing as well in terms of updating the back reference. So, completely randomly and arbitrarily, but to try and make a parallel, what if customers were a dict of orders, and when you made an order it was associated with only one customer, and you put the order in the customer. This may be bizarre--what's the key? can more than one customer be associated with an order? But that would mean that customer.values() would get all of the customer's orders, and order.parent would be the associated customer. This example subclasses zc.dict, a super simple package that only depends on ZODB, btw. It does not have full dict behavior, as Jim likes to point out, because items are stored by BTree sorting, not hashes, but it looks like a dict otherwise. Maybe it can help you out. If you want to try out this example, copy the code out and get the zc.dict egg and give it a try. Ok, I can see how this makes sense for the simple example I gave. You can get all the info you need for any contained items by starting from the customer. That part I was already seeing but hadnt fully put it together because I was distracted by the bigger complication part that keeps my brain churning. Let me give a more detailed example of where I hit a problem ( I'm still leaving some stuff out, just trying to get the general map of things here ): Customer name has many addresses has many orders has many paymethods Order name total salestax has many items belongs to customer LineItem quantity has a product belongs to order Product sku description In this example if we have 3 Products w/ skus of: Widget A Widget B Widget C And we want to change the the description of Widget B and have that change appear for everything, how do I do that? basically, I want all LineItems for Widget A to all refer to a single instance of Product. In a relational database, that is easy... there are 3 entries in the Products table and there is a foreign key in LineItems. Here, I have no idea how to handle. Which makes me think that I'm missing some fundamental point. I'm sure the answer to this will start a whole slew of new questions, so thanks in advance for the help with this and going forward. Its very frustrating to keep beating my head against these conceptual points that I'm just not getting. Usually, I have eureka moments after spending this much time on something, this time, its just not coming. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Feb 27, 2008, at 10:40 AM, Gary Poster wrote: Been over that. Still have serious questions. And yes, we have a large number of objects, something in the area of 16 million at present if you were to map each relational table row to an object. The biggest concern I have is how do to the layout/storage so that this slightly contrived example works: Product has a brand. There are many brands. How do I store so that I can find all products simply and all brands simply and also so that changes in a brand instance are reflected when the product instance is deserialized. By 'simply' I mean that it doesnt really work on our end to have to walk all Products looking for unique brands. Should just be able to go directly in and get said brands ( using keys() or similar call ). If I create 'brand' and 'product' as btrees, then if i do something like some_product.brand.name = 'something entirely different' and that brand already exists in 'brand', would it be updated? are references maintained in that fashion? do we have to handle manually on update and creation? Note that we would just be using ZODB not Zope in this scenario. Back references are not maintained automatically. I'd identify two classic solutions to this sort of thing. One is to make a custom mapping (using a BTree as the inner data structure) that maintains back-references when objects are placed in them or removed. zope.app(.container? .folder? I'd have to look) has code that does this, along with firing events. For simple stories like the one you describe here, that's what I'd probably recommend. It works to the strengths of the ZODB, which particularly shines in terms of readability when you just need to walk a tree of attributes to get what you want. The other is to keep an external index, a la zc.extrinsicreference or zc.relation. zc.extrinsicreference does not have too many dependencies beyond ZODB, and as long as zope.app.keyreference doesn't drag much along with it, might be usable as a library. That said, it's also very simple, and could be used as a model for you, even if you don't use it directly. It would also be a reasonable choice for a simple situation like the one you describe. It relies on events to update its data structures. zc.relation an almost-released-revision of zc.relationship that drastically reduces dependencies--actually, it has no additional dependencies to ZODB, as you can see at http://svn.zope.org/zc.relation/trunk/setup.py?view=markup . It's also a bit overwhelming and low-level: see the README:http://svn.zope.org/zc.relation/trunk/src/zc/relation/README.txt?view=auto . It doesn't hook anything up for you: you set the relationship catalog up and you arrange for it to be updated, via events or direct messages. That said, if you need its power, it is well- tested and would be a good choice for some jobs from at least some perspectives (caveat read-or: I'm the author). HTH I've been playing with this and I'm still having a disconnect 3 week later. 1. I love the zc.relation stuff it answers one area of stuff I hadn't even gotten to yet. 2. I'm just totally missing something If I have Customers and Orders and I want to be able update all of them independently of their relationships ( so that if an order is updated, when i get the customer at some later time, it has the updated order amongst that relation ) but I can't figure it out. I tried looking at the zope.app.folder stuff but I keep getting lost in the zope aspects of it and am having a hard time seeing the forest for the trees. Is there some more general ready on the patterns used that you know of? I think if I understood the idea more in an abstract sense, I could get a lot more out of the folder implementation. Thanks for your help, Sean ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Feb 27, 2008, at 5:33 AM, Laurence Rowe wrote: Sean Allen wrote: been looking for anything along those lines. in particular, strategies and gotchas for how to store objects. everything i've found is basically just a single type of object being stored. i'm really interested in tutorials and information on the best ways to setup large complicated hierarchies and all the various gotchas etc. anything like that exist? A good jumping off point is the wiki: http://wiki.zope.org/ZODB/Documentation The beauty of using the ZODB is that it really makes no difference whether you are storing homogeneous or heterogeneous hierarchies of objects. If you are likely to have a large number of objects in any one folder then make sure it is BTree based. This is a scalable data structure, so the whole list does not have to be loaded into memory to access a single child object. Been over that. Still have serious questions. And yes, we have a large number of objects, something in the area of 16 million at present if you were to map each relational table row to an object. The biggest concern I have is how do to the layout/storage so that this slightly contrived example works: Product has a brand. There are many brands. How do I store so that I can find all products simply and all brands simply and also so that changes in a brand instance are reflected when the product instance is deserialized. By 'simply' I mean that it doesnt really work on our end to have to walk all Products looking for unique brands. Should just be able to go directly in and get said brands ( using keys() or similar call ). If I create 'brand' and 'product' as btrees, then if i do something like some_product.brand.name = 'something entirely different' and that brand already exists in 'brand', would it be updated? are references maintained in that fashion? do we have to handle manually on update and creation? Note that we would just be using ZODB not Zope in this scenario. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Feb 27, 2008, at 10:03 AM, Sean Allen wrote: On Feb 27, 2008, at 5:33 AM, Laurence Rowe wrote: Sean Allen wrote: been looking for anything along those lines. in particular, strategies and gotchas for how to store objects. everything i've found is basically just a single type of object being stored. i'm really interested in tutorials and information on the best ways to setup large complicated hierarchies and all the various gotchas etc. anything like that exist? A good jumping off point is the wiki: http://wiki.zope.org/ZODB/Documentation The beauty of using the ZODB is that it really makes no difference whether you are storing homogeneous or heterogeneous hierarchies of objects. If you are likely to have a large number of objects in any one folder then make sure it is BTree based. This is a scalable data structure, so the whole list does not have to be loaded into memory to access a single child object. Been over that. Still have serious questions. And yes, we have a large number of objects, something in the area of 16 million at present if you were to map each relational table row to an object. The biggest concern I have is how do to the layout/storage so that this slightly contrived example works: Product has a brand. There are many brands. How do I store so that I can find all products simply and all brands simply and also so that changes in a brand instance are reflected when the product instance is deserialized. By 'simply' I mean that it doesnt really work on our end to have to walk all Products looking for unique brands. Should just be able to go directly in and get said brands ( using keys() or similar call ). If I create 'brand' and 'product' as btrees, then if i do something like some_product.brand.name = 'something entirely different' and that brand already exists in 'brand', would it be updated? are references maintained in that fashion? do we have to handle manually on update and creation? Note that we would just be using ZODB not Zope in this scenario. Back references are not maintained automatically. I'd identify two classic solutions to this sort of thing. One is to make a custom mapping (using a BTree as the inner data structure) that maintains back-references when objects are placed in them or removed. zope.app(.container? .folder? I'd have to look) has code that does this, along with firing events. For simple stories like the one you describe here, that's what I'd probably recommend. It works to the strengths of the ZODB, which particularly shines in terms of readability when you just need to walk a tree of attributes to get what you want. The other is to keep an external index, a la zc.extrinsicreference or zc.relation. zc.extrinsicreference does not have too many dependencies beyond ZODB, and as long as zope.app.keyreference doesn't drag much along with it, might be usable as a library. That said, it's also very simple, and could be used as a model for you, even if you don't use it directly. It would also be a reasonable choice for a simple situation like the one you describe. It relies on events to update its data structures. zc.relation an almost-released-revision of zc.relationship that drastically reduces dependencies--actually, it has no additional dependencies to ZODB, as you can see at http://svn.zope.org/zc.relation/trunk/setup.py?view=markup . It's also a bit overwhelming and low-level: see the README: http://svn.zope.org/zc.relation/trunk/src/zc/relation/README.txt?view=auto . It doesn't hook anything up for you: you set the relationship catalog up and you arrange for it to be updated, via events or direct messages. That said, if you need its power, it is well-tested and would be a good choice for some jobs from at least some perspectives (caveat read-or: I'm the author). HTH Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
