On Sat, 2009-11-14 at 14:23 -0700, Shane Hathaway wrote: > Roché Compaan wrote: > > On Fri, 2009-11-13 at 13:33 -0700, Shane Hathaway wrote: > >> Stephan Richter wrote: > >>> http://svn.zope.org/z3c.sharding/trunk > > > > Great stuff! This approaches scaling a large data set at application > > level though. Don't you think a ZODB storage doing this for you would > > solve the problem more generally? > > I think proper construction of horizontally scalable databases must be > done partly at application level, since a lot of the issues to be solved > are specific to the application.
What are the issues you're talking about? > > I think that the master index needs to be partitioned as well. In > > benchmarks I performed early last year (http://bit.ly/pSVmd), a BTree > > could only handle about 250 inserts / second when it approached 10 > > million objects, so I'm guessing it will be almost unusable at a 100 > > million. > > Right. The top level probably ought to be a dynamic hash, not a BTree. > I intended z3c.sharding more as a proof of concept. Sure, just thought I'd mention it. Also keep in mind that the dynamic hash needs to handle the introduction of new partitions. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
