Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Sat, 2008-03-22 at 15:49 -0400, Sean Allen wrote: Ha! Ok. Questions then from there. Why not store each object type, customer, order etc in their own folders? You could, and I understand that you want to do so coming from a RDBMS world, but if you are working with objects then you have the luxury of organising content in a way that is closer to reality. I think it is rather convenient to store a customer's orders inside it. I can see the convenience. Ok, so here is a big question... Assuming you did store each object in its own folder and you had Customer has many orders At what point do you have to hook into zodb so that if you say did this: customer = Customer( ... ) customer.addOrder( Order( ... ) ) that the order in question when customer is saved, gets saved to its own folder and then customer gets a reference to that now saved persistent order before it itself is saved. You first create the order inside the orders folder, then you reference it from the customer: order = Order() orderfolder['order123'] = order # assuming you have a folder for orders in each customer, the following # will create a reference since the above code already persisted the # object. customer.orders['order123'] = order I don't see why you need the reference though. Why not just create the order directly in the order folder for the relevant customer: order = Order() customer.orders['order123'] = order There is another good reason - it makes a lot more sense to distribute object creation to different containers in the ZODB to avoid write conflict errors. I think I understand this but can you elaborate? By 'distribute object creation to different containers' do you mean store each object in own folders? No that would lead to an unnecessary proliferation of folders and it wouldn't actually remove conflict errors. Conflicts errors occur when two concurrent transactions modify the same attribute. When you are putting objects in a folder you are essentially modifying the same attribute (unless you are using a btree). A good application design strategy for the ZODB is to ensure that concurrent transactions are modifying attributes on different objects. If you are putting all your orders in one folder you are increasing the chances of conflict errors. Like I mentioned above you could use a btree folder since it has a fairly good conflict resolution strategy. But given a high enough insertion rate even a btree will not protect you against conflict errors. I would recommend you create orders inside a btree-based order folder inside each customer. This will significantly reduce the likeliness of conflict errors. is there a reason to have a folder that contains a more complete graph? what advantages would that have? speed of access? What do you mean with more complete graph? storing line items inside orders inside customers type scenario Access and insertion times for items in folders with fewer items are faster and like I mentioned above, distributing objects writes reduces conflict errors -- the database i am looking at moving over has 1.2 million entries in a transactions table 980,000 orders 775,000 customers 1.5 million order items I think these numbers are quite manageable. But think carefully what kind of queries you want to do on the data. You have a very rich query language in SQL that allows the construction of complex queries and it will come naturally to you. To do the same in the ZODB will take careful planning. Are there any mistakes that people usually make when doing this sort of mental context switch that you can make me aware of now? Any good reading on the subject? Either there isnt a ton of information on this on the net or my google skills are slipping. The most valuable resource is the mailing list. Unfortunately nobody has documented their experience elsewhere before. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Mar 23, 2008, at 3:59 AM, Roché Compaan wrote: On Sat, 2008-03-22 at 15:49 -0400, Sean Allen wrote: Ha! Ok. Questions then from there. Why not store each object type, customer, order etc in their own folders? You could, and I understand that you want to do so coming from a RDBMS world, but if you are working with objects then you have the luxury of organising content in a way that is closer to reality. I think it is rather convenient to store a customer's orders inside it. I can see the convenience. Ok, so here is a big question... Assuming you did store each object in its own folder and you had Customer has many orders At what point do you have to hook into zodb so that if you say did this: customer = Customer( ... ) customer.addOrder( Order( ... ) ) that the order in question when customer is saved, gets saved to its own folder and then customer gets a reference to that now saved persistent order before it itself is saved. You first create the order inside the orders folder, then you reference it from the customer: order = Order() orderfolder['order123'] = order # assuming you have a folder for orders in each customer, the following # will create a reference since the above code already persisted the # object. customer.orders['order123'] = order I don't see why you need the reference though. Why not just create the order directly in the order folder for the relevant customer: order = Order() customer.orders['order123'] = order Can that be wrapped in a transaction so that if something goes wrong with customer creation, the order gets rolled back as well? Or put higher level, do transactions only work within a single folder or across folders? There is another good reason - it makes a lot more sense to distribute object creation to different containers in the ZODB to avoid write conflict errors. I think I understand this but can you elaborate? By 'distribute object creation to different containers' do you mean store each object in own folders? No that would lead to an unnecessary proliferation of folders and it wouldn't actually remove conflict errors. Conflicts errors occur when two concurrent transactions modify the same attribute. When you are putting objects in a folder you are essentially modifying the same attribute (unless you are using a btree). A good application design strategy for the ZODB is to ensure that concurrent transactions are modifying attributes on different objects. If you are putting all your orders in one folder you are increasing the chances of conflict errors. If you dont put all orders in a single folder, when it comes time to search orders, how would you know where to find? Do you sequentially search the multiple folders for a given object? Like I mentioned above you could use a btree folder since it has a fairly good conflict resolution strategy. But given a high enough insertion rate even a btree will not protect you against conflict errors. I would recommend you create orders inside a btree-based order folder inside each customer. This will significantly reduce the likeliness of conflict errors. So if this occurs, you get an exception ( basically optimistic locking ) which you then deal with on a higher level in your application? -- the database i am looking at moving over has 1.2 million entries in a transactions table 980,000 orders 775,000 customers 1.5 million order items I think these numbers are quite manageable. But think carefully what kind of queries you want to do on the data. You have a very rich query language in SQL that allows the construction of complex queries and it will come naturally to you. To do the same in the ZODB will take careful planning. Are there any mistakes that people usually make when doing this sort of mental context switch that you can make me aware of now? Any good reading on the subject? Either there isnt a ton of information on this on the net or my google skills are slipping. The most valuable resource is the mailing list. Unfortunately nobody has documented their experience elsewhere before. Well, if I go with zeo/zodb and python, I'll be sure to document the entire thing. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Sun, 2008-03-23 at 11:38 -0400, Sean Allen wrote: Can that be wrapped in a transaction so that if something goes wrong with customer creation, the order gets rolled back as well? Or put higher level, do transactions only work within a single folder or across folders? You can modify as many objects as you like in a transaction. Have you read the ZODB/ZEO programming guide yet? http://www.zope.org/Wikis/ZODB/guide/zodb.html There is another good reason - it makes a lot more sense to distribute object creation to different containers in the ZODB to avoid write conflict errors. I think I understand this but can you elaborate? By 'distribute object creation to different containers' do you mean store each object in own folders? No that would lead to an unnecessary proliferation of folders and it wouldn't actually remove conflict errors. Conflicts errors occur when two concurrent transactions modify the same attribute. When you are putting objects in a folder you are essentially modifying the same attribute (unless you are using a btree). A good application design strategy for the ZODB is to ensure that concurrent transactions are modifying attributes on different objects. If you are putting all your orders in one folder you are increasing the chances of conflict errors. If you dont put all orders in a single folder, when it comes time to search orders, how would you know where to find? Do you sequentially search the multiple folders for a given object? No. You index the order attributes that you want to search on and perform a search on the indexes. Have a look at zc.index and zc.catalog http://svn.zope.org/zc.index/ http://svn.zope.org/zc.catalog/ I think you will also benefit a lot by looking at Zope as development platform. Have a look at the Zope3Book: http://wiki.zope.org/zope3/Zope3Book Like I mentioned above you could use a btree folder since it has a fairly good conflict resolution strategy. But given a high enough insertion rate even a btree will not protect you against conflict errors. I would recommend you create orders inside a btree-based order folder inside each customer. This will significantly reduce the likeliness of conflict errors. So if this occurs, you get an exception ( basically optimistic locking ) which you then deal with on a higher level in your application? Yes. The most valuable resource is the mailing list. Unfortunately nobody has documented their experience elsewhere before. Well, if I go with zeo/zodb and python, I'll be sure to document the entire thing. Cool! We really need that! -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: so your coming from the rdbms world tutorial?
On Mar 23, 2008, at 1:30 PM, Roché Compaan wrote: On Sun, 2008-03-23 at 11:38 -0400, Sean Allen wrote: Can that be wrapped in a transaction so that if something goes wrong with customer creation, the order gets rolled back as well? Or put higher level, do transactions only work within a single folder or across folders? You can modify as many objects as you like in a transaction. Have you read the ZODB/ZEO programming guide yet? http://www.zope.org/Wikis/ZODB/guide/zodb.html yes but the examples are always about objects in a single folder. wasnt sure if different folders would have any impact. i assumed not, but you know where assumptions get you. There is another good reason - it makes a lot more sense to distribute object creation to different containers in the ZODB to avoid write conflict errors. I think I understand this but can you elaborate? By 'distribute object creation to different containers' do you mean store each object in own folders? No that would lead to an unnecessary proliferation of folders and it wouldn't actually remove conflict errors. Conflicts errors occur when two concurrent transactions modify the same attribute. When you are putting objects in a folder you are essentially modifying the same attribute (unless you are using a btree). A good application design strategy for the ZODB is to ensure that concurrent transactions are modifying attributes on different objects. If you are putting all your orders in one folder you are increasing the chances of conflict errors. If you dont put all orders in a single folder, when it comes time to search orders, how would you know where to find? Do you sequentially search the multiple folders for a given object? No. You index the order attributes that you want to search on and perform a search on the indexes. Have a look at zc.index and zc.catalog http://svn.zope.org/zc.index/ http://svn.zope.org/zc.catalog/ I think you will also benefit a lot by looking at Zope as development platform. Have a look at the Zope3Book: http://wiki.zope.org/zope3/Zope3Book I have not as it seemed rather Zope specific. Are there particular parts I could zero in on as being particularly relevant? When I review the index, there isnt anything that jumps out and says, o read me. I have read the zodb/zeo guide as well as several other things my googling turned up plus some misc other documents from zope.org that seemed applicable. The big problem I have is that all the zodb stuff i have found is either very trivial or tied up zope. Zope by this point in time, is a highly advanced and complicated system from the outside. Having to slog through zope specific stuff trying to pick out zodb details is frustrating. If I go with zodb as a backend, I would intend to help others who come in the future by providing more zodb specific docs that arent at all tied to zope. ( I've actually come across many blog point that make that specific point ). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
