Re: Long running nodetool repair
It should not take that long. For my 200G node, it takes about an hour to calculate the Merkle tree and then data streaming. By the way, how do you know the repair is not done? If you run nodetool tpstats, it should give you the AntiEntropy session info, active/pending/completed etc. While calculating Merkle tree, you can see the progress from nodetool compactionstats. While streaming data, you can see the progress from nodetool netstats. Also you can grep the log by "Merkle" and "repair". - Original Message - From: "Haithem Jarraya" To: user@cassandra.apache.org Sent: Tuesday, February 19, 2013 1:29:19 AM Subject: Long running nodetool repair Hi, I am new to Cassandra and I am not sure if this is the normal behavior but nodetool repair runs for too long even for small dataset per node. As I am writing I started a nodetool repair last night at 18:41 and now it's 9:18 and it's still running, the size of my data is only ~500mb per node. We have 3 Node cluster in DC1 with RF 3 1 Node Cluster in DC2 with RF 1 1 Node cluster in DC3 with RF 1 and running Cassandra V1.2.1 with 256 vNodes. >From cassandra logs I do not see AntiEntropy logs anymore only compaction Task >and FlushWriter. Is this a normal behaviour of nodetool repair? Is the running time grow linearly with the size of the data? Any help or direction will be much appreciated. Thanks, H
Re: Long running nodetool repair
This is very normal (unfortunately). Are you doing a repair –pr or a straight up repair? Does nodetool netstats show anything? I frequently see repair hang in 1.2.1, and I haven't been able to figure out why yet though. Feel free to take a stack dump with jstack on the node doing the repair and see if there are any deadlocks potentially occurring after the merkel tree's are received. And to help more, do you have the last logs after AntiEntrophy? Any streaming sessions from other nodes? Bug is being tracked here: https://issues.apache.org/jira/browse/CASSANDRA-5146 Best, Michael From: Haithem Jarraya mailto:haithem.jarr...@struq.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Date: Tuesday, February 19, 2013 1:29 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Long running nodetool repair Hi, I am new to Cassandra and I am not sure if this is the normal behavior but nodetool repair runs for too long even for small dataset per node. As I am writing I started a nodetool repair last night at 18:41 and now it's 9:18 and it's still running, the size of my data is only ~500mb per node. We have 3 Node cluster in DC1 with RF 3 1 Node Cluster in DC2 with RF 1 1 Node cluster in DC3 with RF 1 and running Cassandra V1.2.1 with 256 vNodes. >From cassandra logs I do not see AntiEntropy logs anymore only compaction Task >and FlushWriter. Is this a normal behaviour of nodetool repair? Is the running time grow linearly with the size of the data? Any help or direction will be much appreciated. Thanks, H
Long running nodetool repair
Hi, I am new to Cassandra and I am not sure if this is the normal behavior but nodetool repair runs for too long even for small dataset per node. As I am writing I started a nodetool repair last night at 18:41 and now it's 9:18 and it's still running, the size of my data is only ~500mb per node. We have 3 Node cluster in DC1 with RF 3 1 Node Cluster in DC2 with RF 1 1 Node cluster in DC3 with RF 1 and running Cassandra V1.2.1 with 256 vNodes. >From cassandra logs I do not see AntiEntropy logs anymore only compaction Task and FlushWriter. Is this a normal behaviour of nodetool repair? Is the running time grow linearly with the size of the data? Any help or direction will be much appreciated. Thanks, H