Hello Bram You're probably running into this : https://issues.apache.org/jira/browse/CASSANDRA-4417
It's marked as won't fix because it is related to the current counter design. Fortunately C* 2.1 will fix this. Worth reading : www.datastax.com/dev/blog/whats-new-in-cassandra-2-1-a-better-implementation-of-counters Le 2 sept. 2014 21:24, "Bram Avontuur" <b...@longtailvideo.com> a écrit : > Hi, > > Cassandra setup: > > * 2 nodes on EC2, m1.large > * Cassandra version 2.0.10 > > One node died over the weekend, and I couldn't revive it. I deleted it > with nodetool removenode, and added a new node with a copy of the > cassandra.yaml config with the ip addresses changed. > > Once reconfigured and started, nodetool status listed it as part of the > 2-node cluster. I then ran nodetool repair on the new node to get it to > take replication data from a keyspace with replication factor 2. The first > 600-ish MB (of 14GB) synced pretty fast, but then the system.log starts > spawning " invalid remote counter shard detected" nodes at a rapid rate > (too fast to follow with tail -f). Example log line: > > WARN [CompactionExecutor:6] 2014-09-02 19:18:49,109 CounterContext.java > (line 467) invalid remote counter shard detected; > (03afc080-2f01-11e4-948b-15a04b0b4bd9, 1, 158) and > (03afc080-2f01-11e4-948b-15a04b0b4bd9, 1, 79) differ only in count; will > pick highest to self-heal on compaction > > Transfer speed from that point on was quite slow, couple hundred MB's per > 10 minutes. > > After a while nodetool netstats stops listing transfers, and the warnings > also calm down. There's still a handful of them per minute, while the > cluster is not being used though. > > Any idea what could be going on here? > > Bram >
