Ken, Are your vms on different bare metal? Could they potentially be on the same bare metal?
Are you seeing any io contention? Sean Carey @densone On Wednesday, December 5, 2012 at 20:41, Ken Perkins wrote: > Yes, we're thrashing on all of the boxes, due to disk access when looking > through merge_index. It's not noisy neighbors, given how consistent the > thrashing is. We had a box with a corrupted index (we had to remove > merge_index and rebuild) and that machine instantly went to 0% thrashing. So > we have a pretty good indication of the source. > > The cost for 10 8GB VMs is roughly equivalent to 5 16GB ones. > > Thanks for your input Michael! > > Ken > > > On Wed, Dec 5, 2012 at 4:47 PM, Michael Johnson <[email protected] > (mailto:[email protected])> wrote: > > There are a lot of things that go into this, but I would tend to suggest in > > a hosted VM senario, upping the ram is likely the right solution. > > > > You mention thrashing, but what is that thrashing coming from? I assume > > all the boxes are thrashing and not just one or two of them? Is it due to > > swapping or is it just the raw disk access? Maybe you logging too > > aggressively? > > > > Perhaps your are suffering from a bad neighbor effect. If this is the > > case, increasing the amount of ram will likely put you on a physical host > > with few customers and thus you would be less likely to have a bad > > neighbor. > > > > Cost-wise in the VM world, you might be better off adding a few nodes > > rather than increasing the ram in your existing vm's. > > > > But then we are talking VMs and thus it should be fairly painless to > > experiment. I would try adding ram first and if that doesn't work, add a > > few nodes. Someone else my have a different opinion, but that is my two > > cents. > > > > > > On Wed, Dec 5, 2012 at 4:33 PM, Ken Perkins <[email protected] > > (mailto:[email protected])> wrote: > > > Hello all, > > > > > > We're seeing enough thrashing and low-memory on our production ring that > > > we've decided to upgrade our hardware. The real question is should we > > > scale up or out. > > > > > > Currently our ring is 512 partitions. We know that it's a sub-optimal > > > size but we can't easily solve that now. We're currently running a > > > search-heavy app on 5 8GB VMs. I'm debating between moving the VMs up to > > > 16GB, or adding a few more 8GB VMs. > > > > > > Some of the talk in #riak has pushed me towards adding more machines > > > (thus lowering the per node number of partitions) but I wanted to do a > > > quick sanity check here with folks that it's better than scaling up my > > > current machines. > > > > > > Thanks! > > > Ken Perkins > > > clipboard.com (http://clipboard.com) > > > > > > _______________________________________________ > > > riak-users mailing list > > > [email protected] (mailto:[email protected]) > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
