Hello, This is a 2.8.8. cluster
That's an exotic version! I understand you use Apache Cassandra 2.2.8. :) This single node was a seed node and it was running a ‘repair -pr’ at the > time In Apache Cassandra 2.2.8, this triggers incremental repairs I believe, and they are relatively (some would say completely) broken. Let's say they caused a lot of troubles in many cases. If I am wrong and you are not running incremental repairs (default in your version off the top of my head) then you node might not have enough resource available to handle both the repair and the standard load. It might be something to check. Consequences of incremental repairs are: - Keeping SSTables split between repaired and not repaired table, increasing the number of SSTable - Anti-compaction (splits SSTables) is used to keep them grouped. This induces a lot of performances downsides such as (but not only): - inefficient tombstone eviction - More disk hit for the same queries - More compaction work Machine are then performing very poorly. My coworker Alexander wrote about this a few month ago, it might be of interest: http://thelastpickle.com/blog/2017/12/14/should-you-use-incremental-repair.html If repairs are a pain point, you might be interested in checking http://cassandra-reaper.io/, that aims at making this operation easier and more efficient. I would say the fact this node is a seed nodes did not impact here, it is a coincidence due to the fact you picked a seed for the repair. Seed nodes are mostly working as any other node, excepted during bootstrap. So we decided to bootstrap it. > I am not sure what happen when bootstrapping a seed node. I always removed it from the seed list first. Did you replace the node in-place? I guess if you had no warning and have no consistency issues, it's all good. All we were able to see is that the seed node in question was different in > that it had 5000 sstables while all others had around 2300. After > bootstrap, seed node sstables reduced to 2500. > I would say this is fairly common (even more when using vnodes) as streaming of the data from all the other nodes is fast and compaction might take a while to catch up. Why would starting C* on a single seed node affect the cluster this bad? That's a fair question. It depends on factors such as the client configuration, the replication factor, the consistency level used. If the node is involved in some reads, then the average latency will drop. You should be able to have nodes going down, or being fairly slow and use the right nodes if the client is recent enough and well configured. > Is it gossip? > It might be, there were issues, but I believe in previous versions and / or on bigger cluster. I would dig for a 'repair' issue first, it seems more probable to me. I hope this helped, C*heers, ----------------------- Alain Rodriguez - @arodream - al...@thelastpickle.com France / Spain The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2018-03-02 14:42 GMT+00:00 Fd Habash <fmhab...@gmail.com>: > This is a 2.8.8. cluster with three AWS AZs, each with 4 nodes. > > > > Few days ago we noticed a single node’s read latency reaching 1.5 secs > there was 8 others with read latencies going up near 900 ms. > > > > This single node was a seed node and it was running a ‘repair -pr’ at the > time. We intervened as follows … > > > > - Stopping compactions during repair did not improve latency. > - Killing repair brought down latency to 200 ms on the seed node and > the other 8. > - Restarting C* on the seed node increased latency again back to near > 1.5 secs on the seed and other 8. At this point, there was no repair > running and compactions were running. We left them alone. > > > > At this point, we saw that putting the seed node back in the cluster > consistently worsened latencies on seed and 8 nodes = 9 out of the 12 nodes > in the cluster. > > > > So we decided to bootstrap it. During the bootstrapping and afterwards, > latencies remained near 200 ms which is what we wanted for now. > > > > All we were able to see is that the seed node in question was different in > that it had 5000 sstables while all others had around 2300. After > bootstrap, seed node sstables reduced to 2500. > > > > Why would starting C* on a single seed node affect the cluster this bad? > Again, no repair just 4 compactions that run routinely on it as well all > others. Is it gossip? What other plausible explanations are there? > > > > ---------------- > Thank you > > >
