Re: Cassandra 3.11 is compacting forever

2017-09-12 Thread Ioannis Zafiropoulos
A maybe long shot, have you deleted the cassandra-topology.properties file since you are using GossipingPropertyFileSnitch? I am sure I have seen a ticket about problems caused in some cases if that file stays around. I removed it from all the nodes and non-stop compaction stopped (after a proper

Re: Cassandra 3.11 is compacting forever

2017-09-08 Thread Romain Hardouin
Hi, It might be useful to enable compaction logging with log_all subproperties. Best, Romain Le vendredi 8 septembre 2017 à 00:15:19 UTC+2, kurt greaves a écrit : Might be worth turning on debug logging for that node and when the compaction kicks off and CPU

Re: Cassandra 3.11 is compacting forever

2017-09-07 Thread kurt greaves
Might be worth turning on debug logging for that node and when the compaction kicks off and CPU skyrockets send through the logs.​

Re: Cassandra 3.11 is compacting forever

2017-09-06 Thread Igor Leão
Last week I moved all nodes back to cassandra 3.9. Everything worked fine since then. Yesterday I tried to upgrade again, running a rolling restart after the upgrade.Nodes were just fine. Today one node started consuming 94.6% of its CPU. Compacting is running all the time for this node. I'm

Re: Cassandra 3.11 is compacting forever

2017-09-03 Thread kurt greaves
Can't say that message explains why the compaction would be stuck. Generally not a good sign and you might need to investigate more but hopefully shouldn't be related. Has that stuck compaction moved since last week? On 1 September 2017 at 22:54, Fay Hou [Storage Service] ­ < fay...@coupang.com>

Re: Cassandra 3.11 is compacting forever

2017-09-01 Thread Fay Hou [Storage Service] ­
try to do a rolling restart for the cluster before doing a compation On Fri, Sep 1, 2017 at 3:09 PM, Igor Leão wrote: > Some generic errors: > > *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i error* > *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log

Re: Cassandra 3.11 is compacting forever

2017-09-01 Thread Igor Leão
Some generic errors: *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i error* *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i excep* *[aladdin@ip-172-16-1-10 cassandra]$ tail cassandra.log | grep -i fail* *[aladdin@ip-172-16-1-10 cassandra]$ tail debug.log | grep

Re: Cassandra 3.11 is compacting forever

2017-09-01 Thread kurt greaves
are you seeing any errors in the logs? Is that one compaction still getting stuck?

Re: Cassandra 3.11 is compacting forever

2017-08-31 Thread Igor Leão
Hey Kurt, Thanks for your reply. Soon as the whole cluster was upgraded (using existing nodes) it worked pretty well. After a while, the high cpu usage/ pending compactions was back affecting all cluster. It's still an open problem. 2017-08-21 20:24 GMT-03:00 kurt greaves :

Re: Cassandra 3.11 is compacting forever

2017-08-21 Thread kurt greaves
Why are you adding new nodes? If you're upgrading you should upgrade the existing nodes first and then add nodes. ​

Cassandra 3.11 is compacting forever

2017-08-21 Thread Igor Leão
Hi there, I've been trying to upgrade a Cassandra 3.9 cluster to Cassandra 3.11. Whenever I try to add a new Cassandra 3.11 node to the main datacenter, using `-Dcassandra.force_3_0_protocol_version=true` on the new node, this new node uses almost 100% of its CPU. Checking `nodetool