Hi Kurt,

our provisioning layer allows extending a cluster only one-by-one, thus we 
didn’t add multiple nodes at the same time.

What we did have was some sort of overlapping between our daily repair cronjob 
and the newly added node still in progress joining. Don’t know if this sort of 
combination might causing troubles.

I did some further testing and run on the same node the following repair call.

nodetool repair -pr ks cf1 cf2

with waiting a few minutes after each finished execution and every time I see 
“… out of sync …” log messages in context of the repair, so it looks like, that 
each repair execution is detecting inconsistencies. Does this make sense?

As said: We are using vnodes (256), RF=3. Additionally, we are writing at CL 
ANY, reading at ONE and repair chance for the 2 CFs in question is default 0.1

Currently testing a few consecutives executions without -pr on the same node.


From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Montag, 05. März 2018 01:10
To: User <user@cassandra.apache.org>
Subject: Re: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K 
SSTables for a single small (GBytes) CF

Repairs with vnodes is likely to cause a lot of small SSTables if you have 
inconsistencies (at least 1 per vnode). Did you have any issues when adding 
nodes, or did you add multiple nodes at a time? Anything that could have lead 
to a bit of inconsistency could have been the cause.

I'd probably avoid running the repairs across all the nodes simultaneously and 
instead spread them out over a week. That likely made it worse. Also worth 
noting that in versions 3.0+ you won't be able to run nodetool repair in such a 
way because anti-compaction will be triggered which will fail if multiple 
anti-compactions are attempted simultaneously (if you run multiple repairs 

Have a look at orchestrating your repairs with TLP's fork of 
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Reply via email to