Hi Avishay, On Sun, 21 Feb 2010, Gregory Farnum wrote: > It's been a while since we've expressly tested scalability in Ceph, > because doing so is pretty hard -- it takes a lot of nodes. The most > recent figures I'm aware of come from an early version of the code > (2007) and are described in Sage's thesis (available at > ceph.newdream.net), in which per-MDS performance drops by about 50% > going from a 1-node to a 128-node metadata cluster (out of 430 total > nodes). At this point it was servicing ~250k metadata ops/second, which > is enough to handle many thousands of OSDs (he says 25k in a file > create/write scenario with 10MB files).
The one thing I'll add here is that the MDS load balancing algorithm hasn't changed much since these tests a few years back. If you were to try to scale the system today, I suspect the first wall you'd run into would be related to the MDS cluster. The problem is trickier than it sounds. For example, an 'optimum' distribution of load across a full loaded/saturated MDS cluster is a local performance _minimum_: any shifting load between two nodes to cause any imbalance will mean better cache performance on one node and poorer performance on the other, but the faster node will service more requests than the slower node loses. Unfairness means better total system throughput. This, among other things, makes it a bit difficult to find the right performance metrics to 'balance' when shifting load around. In any case, as we start deploying larger clusters we'll be revisiting and retuning these algorithms with real workloads. sage ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Ceph-devel mailing list Ceph-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ceph-devel
