Re: [ZODB-Dev] Re: ZODB Benchmarks
On Thu, 2008-02-07 at 20:26 +0100, Dieter Maurer wrote: Roché Compaan wrote at 2008-2-7 21:21 +0200: ... So if I asked you to build a data structure for the ZODB that can do insertions at a rate comparable to Postgres on high volumes, do you think that it can be done? If you need a high write rate, the ZODB is probably not optimal. Ask yourself whether it is not better to put such high frequency write data directly into a relational database. Whenever you have large amounts of highly structured data, a relational database is necessary more efficient than the ZODB. I know it is not optimal for high write scenarios, but I'm asking if it is possible to build a data structure for the ZODB that can do insertions quickly. There are use cases where having a container in the ZODB that can handle large volumes and maintain a high insertion rate would be very convenient. An example of such a use case would be a site with millions of members where each member has their own folder containing different content types. The rate at which new members register is very high as well so the folder needs to handle insertions quickly. In this use case you are not dealing with structured data. If members in a site with such large volumes start to generate content, indexes in the ZODB become problematic too because of the slow rate of insertion. And it this point you start to stuff everything in relational database and the whole experience becomes painful ... -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Roché Compaan wrote at 2008-2-7 21:21 +0200: ... So if I asked you to build a data structure for the ZODB that can do insertions at a rate comparable to Postgres on high volumes, do you think that it can be done? If you need a high write rate, the ZODB is probably not optimal. Ask yourself whether it is not better to put such high frequency write data directly into a relational database. Whenever you have large amounts of highly structured data, a relational database is necessary more efficient than the ZODB. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
On Thu, 2008-02-07 at 00:39 +0100, Dieter Maurer wrote: If I understand correctly, for each insertion 3 calls are made to persistent_id? This is still very far from the 66 I mentioned above? You did not understand correctly. You insert an entry. The insertion modifies (at least) one OOBucket. The OOBucket needs to be written back. For each of its entries (one is your new one, but there may be up to 29 others) 3 persistent_id calls will happen. Thanks I understand now. So if I asked you to build a data structure for the ZODB that can do insertions at a rate comparable to Postgres on high volumes, do you think that it can be done? If so, would it not be worth investing time and money into this? If not, why not? -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
