our provisioning layer allows extending a cluster only one-by-one, thus we
didn’t add multiple nodes at the same time.
What we did have was some sort of overlapping between our daily repair cronjob
and the newly added node still in progress joining. Don’t know if this sort of
combination might causing troubles.
I did some further testing and run on the same node the following repair call.
nodetool repair -pr ks cf1 cf2
with waiting a few minutes after each finished execution and every time I see
“… out of sync …” log messages in context of the repair, so it looks like, that
each repair execution is detecting inconsistencies. Does this make sense?
As said: We are using vnodes (256), RF=3. Additionally, we are writing at CL
ANY, reading at ONE and repair chance for the 2 CFs in question is default 0.1
Currently testing a few consecutives executions without -pr on the same node.
From: kurt greaves [mailto:k...@instaclustr.com]
Sent: Montag, 05. März 2018 01:10
To: User <firstname.lastname@example.org>
Subject: Re: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K
SSTables for a single small (GBytes) CF
Repairs with vnodes is likely to cause a lot of small SSTables if you have
inconsistencies (at least 1 per vnode). Did you have any issues when adding
nodes, or did you add multiple nodes at a time? Anything that could have lead
to a bit of inconsistency could have been the cause.
I'd probably avoid running the repairs across all the nodes simultaneously and
instead spread them out over a week. That likely made it worse. Also worth
noting that in versions 3.0+ you won't be able to run nodetool repair in such a
way because anti-compaction will be triggered which will fail if multiple
anti-compactions are attempted simultaneously (if you run multiple repairs
Have a look at orchestrating your repairs with TLP's fork of
The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose it
to anyone else. If you received it in error please notify us immediately and
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a
company registered in Linz whose registered office is at 4040 Linz, Austria,