Hi Jeff,
Does subrange repair mark the SSTable as repaired? From my memory, it
doesn't.
Regards,
Bowen
On 27/11/2023 16:47, Jeff Jirsa wrote:
I don’t work for datastax, thats not my blog, and I’m on a phone and
potentially missing nuance, but I’d never try to convert a cluster to
IR by disabling auto compaction . It sounds very much out of date or
its optimized for fixing one node in a cluster somehow. It didn’t make
sense in the 4.0 era.
Instead I’d leave compaction running and slowly run incremental repair
across parts of the token range, slowing down as pending compactions
increase
I’d choose token ranges such that you’d repair 5-10% of the data on
each node at a time
On Nov 23, 2023, at 11:31 PM, Sebastian Marsching
<sebast...@marsching.com> wrote:
Hi,
we are currently in the process of migrating from C* 3.11 to C* 4.1
and we want to start using incremental repairs after the upgrade has
been completed. It seems like all the really bad bugs that made using
incremental repairs dangerous in C* 3.x have been fixed in 4.x, and
for our specific workload, incremental repairs should offer a
significant performance improvement.
Therefore, I am currently devising a plan how we could migrate to
using incremental repairs. I am aware of the guide from DataStax
(https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/operations/opsRepairNodesMigration.html),
but this guide is quite old and was written with C* 3.0 in mind, so I
am not sure whether this still fully applies to C* 4.x.
In addition to that, I am not sure whether this approach fits our
workload. In particular, I am wary about disabling autocompaction for
an extended period of time (if you are interested in the reasons why,
they are at the end of this e-mail).
Therefore, I am wondering whether a slighly different process might
work better for us:
1. Run a full repair (we periodically run those anyway).
2. Mark all SSTables as repaired, even though they will include data
that has not been repaired yet because it was added while the repair
process was running.
3. Run another full repair.
4. Start using incremental repairs (and the occasional full repair in
order to handle bit rot etc.).
If I understood the interactions between full repairs and incremental
repairs correctly, step 3 should repair potential inconsistencies in
the SSTables that were marked as repaired in step 2 while avoiding
the problem of overstreaming that would happen when only marking
those SSTables as repaired that already existed before step 1.
Does anyone see a flaw in this concept or has experience with a
similar scenario (migrating to incremental repairs in an environment
with high-density nodes, where a single table contains most of the data)?
I am also interested in hearing about potential problems other C*
users experienced when migrating to incremental repairs, so that we
get a better idea what to expect.
Thanks,
Sebastian
Here is the explanation why I am being cautious:
More than 95 percent of our data is stored in a single table, and we
use high density nodes (storing about 3 TB of data per node). This
means that a full repair for the whole cluster takes about a week.
The reason for this layout is that most of our data is “cold”,
meaning that it is written once, never updated, and rarely deleted or
read. However, new data is added continuously, so disabling
autocompaction for the duration of a full repair would lead to a high
number of small SSTables accumulating over the course of the week,
and I am not sure how well the cluster would handle such a situation
(and the increased load when autocompaction is enabled again).
