Full repair running for an entire week sounds excessively long. Even if you've got 1 TB of data per node, 1 week means the repair speed is less than 2 MB/s, that's very slow. Perhaps you should focus on finding the bottleneck of the full repair speed and work on that instead.

On 03/02/2024 16:18, Sebastian Marsching wrote:

2. use an orchestration tool, such as Cassandra Reaper, to take care of that for you. You will still need monitor and alert to ensure the repairs are run successfully, but fixing a stuck or failed repair is not very time sensitive, you can usually leave it till Monday morning if it happens at Friday night.

Does anyone know how such a schedule can be created in Cassandra Reaper?

I recently learned the hard way that running both a full and an incremental repair for the same keyspace and table in parallel is not a good idea (it caused a very unpleasant overload situation on one of our clusters).

At the moment, we have one schedule for the full repairs (every 90 days) and another schedule for the incremental repairs (daily). But as full repairs take much longer than a day (about a week, in our case), the two schedules collide. So, Cassandra Reaper starts an incremental repair while the full repair is still in process.

Does anyone know how to avoid this? Optimally, the full repair would be paused (no new segments started) for the duration of the incremental repair. The second best option would be inhibiting the incremental repair while a full repair is in progress.

Best regards,

Reply via email to