Thank you for the update.
The repair fails with the Error 'Failed Creating merkle tree' but does not give
any additional details.
With -pr running on all DC nodes, we see a peer connection reset error, which
then results in hanged repair process even though the TCP connection settings
looks good on all nodes.
From: Anuj Wadehra <anujw_2...@yahoo.co.in>
Sent: Wednesday, October 12, 2016 2:41 PM
Subject: Re: Repair in Multi Datacenter - Should you use -dc Datacenter repair
or repair with -pr
First thing you should be concerned about is : Why the repair -pr operation
doesnt complete ?
Second comes the question : Which repair option is best?
One probable cause of stuck repairs is : if the firewall between DCs is closing
TCP connections and Cassandra is trying to use such connections, repairs will
hang. Please refer
. We faced that.
Also make sure you comply with basic bandwidth requirement between DCs.
Recommended is 1000 Mb/s (1 gigabit) or greater.
Answers for specific questions:
1.As per my understanding, all replicas will not participate in dc local
repairs and thus repair would be ineffective. You need to make sure that all
replicas of a data in all dcs are in sync.
2. Every DC is not a ring. All DCs together form a token ring. So, I think yes
you should run repair -pr on all nodes.
3. Yes. I dont have experience with incremental repairs. But you can run repair
-pr on all nodes of all DCs.
Regarding Best approach of repair, you should see some repair presentations of
Cassandra Summit 2016. All are online now.
I attended the summit and people using large clusters generally use sub range
repairs to repair their clusters. But such large deployments are on older
Cassandra versions and these deployments generally dont use vnodes. So people
know easily which nodes hold which token range.
From: Leena Ghatpande <lghatpa...@hotmail.com>;
To: firstname.lastname@example.org <email@example.com>;
Subject: Repair in Multi Datacenter - Should you use -dc Datacenter repair or
repair with -pr
Sent: Wed, Oct 12, 2016 2:15:51 PM
Please advice. Cannot find any clear documentation on what is the best strategy
for repairing nodes on a regular basis with multiple datacenters involved.
We are running cassandra 3.7 in multi datacenter with 4 nodes in each data
center. We are trying to run repairs every other night to keep the nodes in
good state.We currently run repair with -pr option , but the repair process
gets hung and does not complete gracefully. Dont see any errors in the logs
What is the best way to perform repairs on multiple data centers on large
1. Can we run Datacenter repair using -dc option for each data center? Do we
need to run repair on each node in that case or will it repair all nodes within
2. Is running repair with -pr across all nodes required , if we perform the
step 1 every night?
3. Is cross data center repair required and if so whats the best option?