Storage management during rapid growth

2013-10-31 Thread Dave Cowen
Hi, all - I'm currently managing a small Cassandra cluster, several nodes with local SSD storage. It's difficult for to forecast the growth of the Cassandra data over the next couple of years for various reasons, but it is virtually guaranteed to grow substantially. During this time, there may b

Re: nodetool repair fails after expansion

2013-10-04 Thread Dave Cowen
I should clarify that we are running Cassandra 1.1.12. Dave On Fri, Oct 4, 2013 at 2:08 PM, Dave Cowen wrote: > We're testing expanding a 4-node cluster into an 8-node cluster, and we > keep running into issues with the repair process near the end. > > We're bringing up

nodetool repair fails after expansion

2013-10-04 Thread Dave Cowen
We're testing expanding a 4-node cluster into an 8-node cluster, and we keep running into issues with the repair process near the end. We're bringing up nodes 1-by-1 into the cluster, retokening nodes for an 8-node configuration, running nodetool cleanup on the nodes after each retokening, and the

Nodes separating from the ring

2013-09-13 Thread Dave Cowen
Hi, all - We've been running Cassandra 1.1.12 in production since February, and have experienced a vexing problem with an arbitrary node "falling out of" or separating from the ring on occasion. When a node "falls out" of the ring, running nodetool ring on the misbehaving node shows that the misb