On 10/3/07, David Barrett <[EMAIL PROTECTED]> wrote: > ... > However, I wonder if the real use for the DHT will be in vast server farms, > as suggested by Amazon's Dynamo project: > ...
I think David has hit on a much broader and more interesting topic here, that of the value of decentralized architectures even within traditionally centralized (at least politically centralized) services that live inside datacenter[s]. For any of us who have worked on datacenter-centric systems that need to be massively scalable, servicing either millions of users or petabytes of data (or both), one thing always becomes abundantly clear: centralized components in these systems suck. They suck for a number of reasons. They suck because: 1. they create performance bottlenecks. 2. they don't scale by simply adding more resources 3. they form single-points-of-failure. 4. modifying/updating these components in a live system is sooper scary and causes great ph34r (because sometimes even breathing on these components can trigger events that make us deal with numbers 1-3). The Amazon Dynamo paper drives all of these points home better than I can here, but it is worth noting that we've known this for a long time -- people first started extolling the virtues of "shared-nothing" architectures back in the mid-80s for heaven's sake. Even with all that, it still surprises me when smart people defend the practice of using traditional centralized techniques for systems which they expect to scale, when we know that getting that to work involves the ugly prospect of doing things like sharding dbs, segregating user pools, etc., when there are clearly better alternatives. Thanks to the likes of the Dynamo paper (and friends such as BigTable), we can even now say "when there are clearly better /proven/ alternatives." Maybe centralized techniques remain so attractive because they are so simple to implement initially, and because they work in the beginning. And maybe since most projects fail before getting to the point where they /really/ need to scale, people come away thinking that the centralized thing they did works just fine. Maybe it's just human nature. But when popular systems grow rapidly and hit that scalability wall where centralized components start to creak and pop, I think you'll find few engineers who will claim they were ever a good idea. Alen _______________________________________________ p2p-hackers mailing list [email protected] http://lists.zooko.com/mailman/listinfo/p2p-hackers
