Hi Kurt, our provisioning layer allows extending a cluster only one-by-one, thus we didn’t add multiple nodes at the same time.
What we did have was some sort of overlapping between our daily repair cronjob and the newly added node still in progress joining. Don’t know if this sort of combination might causing troubles. I did some further testing and run on the same node the following repair call. nodetool repair -pr ks cf1 cf2 with waiting a few minutes after each finished execution and every time I see “… out of sync …” log messages in context of the repair, so it looks like, that each repair execution is detecting inconsistencies. Does this make sense? As said: We are using vnodes (256), RF=3. Additionally, we are writing at CL ANY, reading at ONE and repair chance for the 2 CFs in question is default 0.1 Currently testing a few consecutives executions without -pr on the same node. Thanks, Thomas From: kurt greaves [mailto:k...@instaclustr.com] Sent: Montag, 05. März 2018 01:10 To: User <user@cassandra.apache.org> Subject: Re: Cassandra 2.1.18 - Concurrent nodetool repair resulting in > 30K SSTables for a single small (GBytes) CF Repairs with vnodes is likely to cause a lot of small SSTables if you have inconsistencies (at least 1 per vnode). Did you have any issues when adding nodes, or did you add multiple nodes at a time? Anything that could have lead to a bit of inconsistency could have been the cause. I'd probably avoid running the repairs across all the nodes simultaneously and instead spread them out over a week. That likely made it worse. Also worth noting that in versions 3.0+ you won't be able to run nodetool repair in such a way because anti-compaction will be triggered which will fail if multiple anti-compactions are attempted simultaneously (if you run multiple repairs simultaneously). Have a look at orchestrating your repairs with TLP's fork of cassandra-reaper<http://cassandra-reaper.io/>. The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
