John-Paul, Do you mind posting your config files to go with each run? Some questions that come to mind:
1. Did you start fresh for each run? (i.e. no data) 2. After adding the 4th node did you add it's IP to the bench config? 3. Do you have any iostat numbers to go with these runs? (both for good and bad would be useful) 4. Do you see any differences in the Riak logs between the good/bad runs? E.g. maybe errors or merge activity? -Ryan On Thu, Feb 23, 2012 at 4:56 PM, John-Paul Bader <[email protected]>wrote: > Hey, > > > so I'm currently benchmarking riak on a bare metal cluster (2xquadcore > xeon 2.2GHz, 24GB RAM, 120GB SSD, gigabit ethernet between nodes) > > I wrote some custom benchmarks in basho bench which connected to all three > nodes and I filled up the cluster consisting of 3 nodes with about 68GB of > Data (50mio objects in one bucket and 35mio objects in another). Storage > backend is leveldb and caching is tuned according to the wiki guide. > > Now with those three nodes i could get up to a constant 30k ops/s - see > http://smyck.org/fast.png for example. > > I ran basho bench with a concurrency of 80 workers from another node next > to the three riak nodes. Empirical tests proved that my benchmarks peaked > at 80 workers in terms of ops/s. Above that it just wasn't getting faster > and latency increased. > > Then I wanted to add another node to see the cluster scale up in > throughput. What I saw with the same settings except for adding th 4th node > to the basho bench config was this: > > http://smyck.org/slow.png > > So while it kicked off a lot faster, the performance decreased quickly. I > even increased the leveldb cache further but the result was the same. Then > I reduced the concurrency to 70 and I got a constant 35k ops/s again. See > http://smyck.org/35concurrent.**png <http://smyck.org/35concurrent.png> > > Now I did kind of a binary search for the sweet spot and it turned out i > could set the concurrency setting up to 74 - everything above 74 showed the > behavior from http://smyck.org/slow.png > > Now - any ideas on that? Is running basho bench from one node the problem? > The machine running basho bench is at around 45% cpu load. Should it be > running like tsung - from multiple nodes? > > Any ideas? As I'm fairly new to riak it is also quite likely that I missed > something - so I'd be happy if you'd point me to common mistakes in > scenarios like these. > > > ~ John > > -- > John-Paul Bader | Software Development > > www.wooga.com > wooga GmbH | Saarbruecker Str. 38 | D-10405 Berlin > Sitz der Gesellschaft: Berlin; HRB 117846 B > Registergericht Berlin-Charlottenburg > Geschaeftsfuehrung: Jens Begemann, Philipp Moeser > > ______________________________**_________________ > riak-users mailing list > [email protected] > http://lists.basho.com/**mailman/listinfo/riak-users_**lists.basho.com<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com> >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
