Re: High CPU on a single node in production

2016-01-07 Thread Josh Yudaken
Hi Fred, That makes a lot of sense. Yes the server was under high indexing load as the handoffs have all caused re-indexing. I'm going to drop transfer-limit to 1 [from 2] across the board which will hopefully reduce further issues. At the moment our cluster wants to perform a ton of handoffs, al

Re: High CPU on a single node in production

2016-01-06 Thread Luke Bakken
> We're planning on having a rather large cluster rather soon, which was > the reason for the large ring size. Your documentation indicates ring > resize is *not* possible with search 2.0 [1], although an issue I > found on github indicated it might be now? [2] Yeah, resize not applicable in your

Re: High CPU on a single node in production

2016-01-06 Thread Fred Dushin
Hi Josh, Sorry for not getting back sooner. I am not entirely sure what is going on with your handoffs. It could be that you have overloaded Solr with handoff activity, and that is causing vnodes to become unresponsive. We are actively working on a fix for this, which allows vnodes to conti

Re: High CPU on a single node in production

2016-01-06 Thread Josh Yudaken
Hi Luke, We're planning on having a rather large cluster rather soon, which was the reason for the large ring size. Your documentation indicates ring resize is *not* possible with search 2.0 [1], although an issue I found on github indicated it might be now? [2] If the situation is resolved we mi

Re: High CPU on a single node in production

2016-01-06 Thread Luke Bakken
Hi Josh, 1024 is too large of a ring size for 10 nodes. If it's possible to rebuild your cluster using a ring size of 128 or 256 that would be ideal (http://docs.basho.com/riak/latest/ops/building/planning/cluster/#Ring-Size-Number-of-Partitions). Ring resizing is possible as well (http://docs.ba

High CPU on a single node in production

2016-01-06 Thread Josh Yudaken
Hi, We're attempting to use Riak as our primary key-value and search database for an analytics-typed solution to blocking spam/fraud. As we expect to eventually be handling a huge amount of data, I started with a ring size of 1024. We currently have 10 nodes on Google Cloud n1-standard-16 instanc