Stephan Richter wrote: > On Friday 13 November 2009, Roché Compaan wrote: >> We had such an opportunity about 2 years ago and although the client >> never reached (and probably will never) reach the membership they >> dreamed about, they did pay us to develop a storage for members that >> could scale to more than a 100 million members. We implemented a data >> partitioning strategy at application level. If I had another shot at it, >> I would try and develop a distributed ZODB storage, because it would be >> a lot simpler compared to what we had to do at application level. > > Note that Shane developed a sharding solution a year ago with me. It provides > container-level partitioning. > > http://svn.zope.org/z3c.sharding/trunk
Thanks for the reminder. :-) > This in combination with the encryption work that we did for the ZODB makes > the ZODB actually be a lot more advanced than some of the new comers. > > I am very intrigued now to setup an EC2 cluster and install a z3c.sharding > based solution demonstrating 100M users with some data. Mmmh... I've been studying how to build an enormous database based on what I know. There are an incredible number of distributed databases these days, but all of them concern me in one way or another. I'm wondering if ZODB might actually have a fighting chance in the distributed database realm. With z3c.sharding or something like it, I think I would set things up as follows: - In-memory ZODB caches would probably be pointlessly painful at that scale, so I would set the ZODB cache size for all partitions to 0. A cache size of 0 allows ZODB to cache for the duration of a request, but flushes all objects out of the cache at transaction boundaries. - With the cache size set to 0, we can disable cache invalidation, which will probably be a major win. - I would rely heavily on memcached to provide the pickles. I would try to use the cache checkpointing algorithm I recently added to RelStorage. - I would aim to read or write only a small number of objects per request from partitions. Shane _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
