Re: Switching to Incremental Repair

[Kristijonas Zalys]

Hi folks,

One last question regarding incremental repair.

What would be a safe approach to temporarily stop running incremental
repair on a cluster (e.g.: during a Cassandra major version upgrade)? My
understanding is that if we simply stop running incremental repair, the
cluster's nodes can, in the worst case, double in disk size as the repaired
dataset will not get compacted with the unrepaired dataset. Similar to
Sebastian, we have nodes where the disk usage is multiple TiBs so
significant growth can be quite dangerous in our case. Would the only safe
choice be to mark all SSTables as unrepaired before stopping regular
incremental repair?

Thanks,
Kristijonas


On Wed, Feb 7, 2024 at 4:33 PM Bowen Song via user <
user@cassandra.apache.org> wrote:

> The over-streaming is only problematic for the repaired SSTables, but it
> can be triggered by inconsistencies within the unrepaired SSTables
> during an incremental repair session. This is because although an
> incremental repair will only compare the unrepaired SSTables, but it
> will stream both the unrepaired and repaired SSTables for the
> inconsistent token ranges. Keep in mind that the source SSTables for
> streaming is selected based on the token ranges, not the
> repaired/unrepaired state.
>
> Base on the above, I'm unsure running an incremental repair before a
> full repair can fully avoid the over-streaming issue.
>
> On 07/02/2024 22:41, Sebastian Marsching wrote:
> > Thank you very much for your explanation.
> >
> > Streaming happens on the token range level, not the SSTable level,
> right? So, when running an incremental repair before the full repair, the
> problem that “some unrepaired SSTables are being marked as repaired on one
> node but not on another” should not exist any longer. Now this data should
> be marked as repaired on all nodes.
> >
> > Thus, when repairing the SSTables that are marked as repaired, this data
> should be included on all nodes when calculating the Merkle trees and no
> overstreaming should happen.
> >
> > Of course, this means that running an incremental repair *first* after
> marking SSTables as repaired and only running the full repair *after* that
> is critical. I have to admit that previously I wasn’t fully aware of how
> critical this step is.
> >
> >> Am 07.02.2024 um 20:22 schrieb Bowen Song via user <
> user@cassandra.apache.org>:
> >>
> >> Unfortunately repair doesn't compare each partition individually.
> Instead, it groups multiple partitions together and calculate a hash of
> them, stores the hash in a leaf of a merkle tree, and then compares the
> merkle trees between replicas during a repair session. If any one of the
> partitions covered by a leaf is inconsistent between replicas, the hash
> values in these leaves will be different, and all partitions covered by the
> same leaf will need to be streamed in full.
> >>
> >> Knowing that, and also know that your approach can create a lots of
> inconsistencies in the repaired SSTables because some unrepaired SSTables
> are being marked as repaired on one node but not on another, you would then
> understand why over-streaming can happen. The over-streaming is only
> problematic for the repaired SSTables, because they are much bigger than
> the unrepaired.
> >>
> >>
> >> On 07/02/2024 17:00, Sebastian Marsching wrote:
>  Caution, using the method you described, the amount of data streamed
> at the end with the full repair is not the amount of data written between
> stopping the first node and the last node, but depends on the table size,
> the number of partitions written, their distribution in the ring and the
> 'repair_session_space' value. If the table is large, the writes touch a
> large number of partitions scattered across the token ring, and the value
> of 'repair_session_space' is small, you may end up with a very expensive
> over-streaming.
> >>> Thanks for the warning. In our case it worked well (obviously we
> tested it on a test cluster before applying it on the production clusters),
> but it is good to know that this might not always be the case.
> >>>
> >>> Maybe I misunderstand how full and incremental repairs work in C* 4.x.
> I would appreciate if you could clarify this for me.
> >>>
> >>> So far, I assumed that a full repair on a cluster that is also using
> incremental repair pretty much works like on a cluster that is not using
> incremental repair at all, the only difference being that the set of
> repaired und unrepaired data is repaired separately, so the Merkle trees
> that are calculated for repaired and unrepaired data are completely
> separate.
> >>>
> >>> I also assumed that incremental repair only looks at unrepaired data,
> which is why it is so fast.
> >>>
> >>> Is either of these two assumptions wrong?
> >>>
> >>> If not, I do not quite understand how a lot of overstreaming might
> happen, as long as (I forgot to mention this step in my original e-mail) I
> run an incremental repair directly after restarting the nodes 

Re: Switching to Incremental Repair

[Kristijonas Zalys]

Hi folks,

Thank you all for your insight, this has been very helpful.

I was going through the migration process here

and I’m not entirely sure why disabling autocompaction on the node is
required? Could anyone clarify what would be the side effects of not
disabling autocompaction and starting with step 2 of the migration?

Thanks,

Kristijonas


On Sun, Feb 4, 2024 at 12:18 AM Alexander DEJANOVSKI 
wrote:

> Hi Sebastian,
>
> That's a feature we need to implement in Reaper. I think disallowing the
> start of the new incremental repair would be easier to manage than pausing
> the full repair that's already running. It's also what I think I'd expect
> as a user.
>
> I'll create an issue to track this.
>
> Le sam. 3 févr. 2024, 16:19, Sebastian Marsching 
> a écrit :
>
>> Hi,
>>
>> 2. use an orchestration tool, such as Cassandra Reaper, to take care of
>> that for you. You will still need monitor and alert to ensure the repairs
>> are run successfully, but fixing a stuck or failed repair is not very time
>> sensitive, you can usually leave it till Monday morning if it happens at
>> Friday night.
>>
>> Does anyone know how such a schedule can be created in Cassandra Reaper?
>>
>> I recently learned the hard way that running both a full and an
>> incremental repair for the same keyspace and table in parallel is not a
>> good idea (it caused a very unpleasant overload situation on one of our
>> clusters).
>>
>> At the moment, we have one schedule for the full repairs (every 90 days)
>> and another schedule for the incremental repairs (daily). But as full
>> repairs take much longer than a day (about a week, in our case), the two
>> schedules collide. So, Cassandra Reaper starts an incremental repair while
>> the full repair is still in process.
>>
>> Does anyone know how to avoid this? Optimally, the full repair would be
>> paused (no new segments started) for the duration of the incremental
>> repair. The second best option would be inhibiting the incremental repair
>> while a full repair is in progress.
>>
>> Best regards,
>> Sebastian
>>
>>

Switching to Incremental Repair

[Kristijonas Zalys]

Hi folks,

I am working on switching from full to incremental repair in Cassandra
v4.0.6 (soon to be v4.1.3) and I have a few questions.


   1.

   Is it necessary to run regular full repair on a cluster if I already run
   incremental repair? If yes, what frequency would you recommend for full
   repair?
   2.

   Has anyone experienced disk usage spikes while using incremental repair?
   I have noticed temporary disk footprint increases of up to 2x (from ~15 GiB
   to ~30 GiB) caused by anti-compaction while testing and am wondering how
   likely that is to happen in bigger real world use cases?


Thank you all in advance!

Kristijonas

Re: Switching to Incremental Repair

[Kristijonas Zalys]

Hi Bowen,

Thank you for your help!

So given that we would need to run both incremental and full repair for a
given cluster, is it safe to have both types of repair running for the same
token ranges at the same time? Would it not create a race condition?

Thanks,
Kristijonas

On Fri, Feb 2, 2024 at 3:36 PM Bowen Song via user <
user@cassandra.apache.org> wrote:

> Hi Kristijonas,
>
> To answer your questions:
>
> 1. It's still necessary to run full repair on a cluster on which
> incremental repair is run periodically. The frequency of full repair is
> more of an art than science. Generally speaking, the less reliable the
> storage media, the more frequently full repair should be run. The
> documentation on this topic is available here
> <https://cassandra.apache.org/doc/stable/cassandra/operating/repair.html#incremental-and-full-repairs>
>
> 2. Run incremental repair for the first time on an existing cluster does
> cause Cassandra to re-compact all SSTables, and can lead to disk usage
> spikes. This can be avoided by following the steps mentioned here
> <https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/operations/opsRepairNodesMigration.html>
>
> I hope that helps.
>
> Cheers,
> Bowen
> On 02/02/2024 20:57, Kristijonas Zalys wrote:
>
> Hi folks,
>
> I am working on switching from full to incremental repair in Cassandra
> v4.0.6 (soon to be v4.1.3) and I have a few questions.
>
>
>1.
>
>Is it necessary to run regular full repair on a cluster if I already
>run incremental repair? If yes, what frequency would you recommend for full
>repair?
>2.
>
>Has anyone experienced disk usage spikes while using incremental
>repair? I have noticed temporary disk footprint increases of up to 2x (from
>~15 GiB to ~30 GiB) caused by anti-compaction while testing and am
>wondering how likely that is to happen in bigger real world use cases?
>
>
> Thank you all in advance!
>
> Kristijonas
>
>