Thanks for your reply. Yes, it would be nice to know the root cause. Now running a full repair. Hopefully this will solve the problem
On Tue, Apr 11, 2017 at 9:43 AM, Roland Otta <roland.o...@willhaben.at> wrote: > well .. thats pretty much the same we saw in our environment (cassandra > 3.7). > > in our case a full repair fixed the issues. > but no doubt .. it would be more satisfying to know the root cause for > that issue > > br, > roland > > > On Mon, 2017-04-10 at 19:12 +0200, George Sigletos wrote: > > In 3 out of 5 nodes of our new DC the rebuild process finished > successfully. In the other two nodes not (the process was hanging doing > nothing) so we killed it, removed all data and started again. This time > finished successfully. > > Here is the netstats output of one of the new newly added nodes: > > Mode: NORMAL > Not sending any streams. > Read Repair Statistics: > Attempted: 269142 > Mismatch (Blocking): 169866 > Mismatch (Background): 4 > Pool Name Active Pending Completed Dropped > Commands n/a 2 10031126 1935 > Responses n/a 97 22565129 n/a > > > On Mon, Apr 10, 2017 at 5:28 PM, Roland Otta <roland.o...@willhaben.at> > wrote: > > Hi, > > we have seen similar issues here. > > have you verified that your rebuilds have been finished successfully? we > have seen rebuilds that stopped streaming and working but have not finished. > what does nodetool netstats output for your newly built up nodes? > > br, > roland > > > On Mon, 2017-04-10 at 17:15 +0200, George Sigletos wrote: > > Hello, > > We recently added a new datacenter to our cluster and run "nodetool > rebuild -- <DC_current>" in all 5 new nodes, one by one. > > After this process finished we noticed there is data missing from the new > datacenter, although it exists on the current one. > > How would that be possible? Should I maybe have run repair in all nodes of > the current DC before adding the new one? > > Running Cassandra 2.1.15 > > Kind regards, > George > > > > > >
