Re: Transitioning to incremental repair

[Marcus Eriksson]

Bryan, this should be improved with
https://issues.apache.org/jira/browse/CASSANDRA-10768 - could you try it
out?

On Tue, Dec 1, 2015 at 10:58 PM, Bryan Cheng  wrote:

> Sorry if I misunderstood, but are you asking about the LCS case?
>
> Based on our experience, I would absolutely recommend you continue with
> the migration procedure. Even if the compaction strategy is the same, the
> process of anticompaction is incredibly painful. We observed our test
> cluster running 2.1.11 experiencing a dramatic increase in latency and not
> responding to nodetool queries over JMX while anticompacting the largest
> SSTables. This procedure also took several times longer than a standard
> full repair.
>
> If you absolutely cannot perform the migration procedure, I believe 2.2.x
> contains the changes to automatically set the RepairedAt flags after a full
> repair, so you may be able to do a full repair on 2.2.x and then transition
> directly to incremental without migrating (can someone confirm?)
>

Re: Transitioning to incremental repair

[Bryan Cheng]

Ah Marcus, that looks very promising- unfortunately we have already
switched back to full repairs and our test cluster has been re-purposed for
other tasks atm. I will be sure to apply the patch/try a fixed version of
Cassandra if we attempt to migrate to incremental repair again.

Re: Transitioning to incremental repair

[Bryan Cheng]

Sorry if I misunderstood, but are you asking about the LCS case?

Based on our experience, I would absolutely recommend you continue with the
migration procedure. Even if the compaction strategy is the same, the
process of anticompaction is incredibly painful. We observed our test
cluster running 2.1.11 experiencing a dramatic increase in latency and not
responding to nodetool queries over JMX while anticompacting the largest
SSTables. This procedure also took several times longer than a standard
full repair.

If you absolutely cannot perform the migration procedure, I believe 2.2.x
contains the changes to automatically set the RepairedAt flags after a full
repair, so you may be able to do a full repair on 2.2.x and then transition
directly to incremental without migrating (can someone confirm?)

Re: Transitioning to incremental repair

[Marcus Eriksson]

Yes, it should now be safe to just run a repair with -inc -par to migrate
to incremental repairs

BUT, if you currently use for example repair service in OpsCenter or
Spotifys Cassandra reaper, you might still want to migrate the way it is
documented as you will have to run a full repair to migrate to incremental
repairs, not many sub range repairs and that might not be possible for some
users with a lot of data or with vnodes etc.

I would also wait until
https://issues.apache.org/jira/browse/CASSANDRA-10768 has been committed
and released as it will improve anticompaction performance

/Marcus

On Tue, Dec 1, 2015 at 3:24 PM, Sam Klock  wrote:

> Hi folks,
>
> A question like this was recently asked, but I don't think anyone ever
> supplied an unambiguous answer.  We have a set of clusters currently
> using sequential repair, and we'd like to transition them to
> incremental repair.  According to the documentation, this is a very
> manual (and likely time-consuming) process:
>
>
> http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesMigration.html
>
> Our understanding is that this process is necessary for tables that use
> LCS, as unrepaired tables are compacted using STCS and (without the
> process described in the doc) all tables start in the unrepaired
> state.  The pain of this migration strategy is supposed to be offset by
> the savings in undesired compaction activity.  The docs aren't
> especially clear, but it sounds like this strategy is not needed for
> tables that use STCS.
>
> However, CASSANDRA-8004 (resolved against 2.1.2) appears intended to
> have both the repaired and unrepaired sstable sets use the same
> compaction strategy.  It seems like that obviates the rationale for a
> migration procedure, which is supported by offhand comments on this
> list, e.g.:
>
> https://www.mail-archive.com/user%40cassandra.apache.org/msg40303.html
> https://www.mail-archive.com/user%40cassandra.apache.org/msg44896.html
>
> In other words, it *looks* like the docs are obsolete, and the
> migration process for existing clusters only consists of flipping the
> switch (i.e., adding "-inc" to invocations of "nodetool repair").
>
> Our questions:
>
> 1) Is our understanding of the status quo following 2.1.2 correct?
> Does migrating existing clusters to incremental repair only require
> adding the "-inc" argument, or is a process still required?
>
> 2) If a process is still required, have there been any changes since
> 2.1.2?  Are the docs up-to-date?
>
> 3) If there is no process or if the process has changed, are there
> plans on the DataStax side to update the documentation accordingly?
>
> Thanks,
> SK
>

Transitioning to incremental repair

[Sam Klock]

Hi folks,

A question like this was recently asked, but I don't think anyone ever 
supplied an unambiguous answer.  We have a set of clusters currently 
using sequential repair, and we'd like to transition them to 
incremental repair.  According to the documentation, this is a very 
manual (and likely time-consuming) process:

http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesMigration.html

Our understanding is that this process is necessary for tables that use 
LCS, as unrepaired tables are compacted using STCS and (without the 
process described in the doc) all tables start in the unrepaired 
state.  The pain of this migration strategy is supposed to be offset by 
the savings in undesired compaction activity.  The docs aren't 
especially clear, but it sounds like this strategy is not needed for 
tables that use STCS.

However, CASSANDRA-8004 (resolved against 2.1.2) appears intended to 
have both the repaired and unrepaired sstable sets use the same 
compaction strategy.  It seems like that obviates the rationale for a 
migration procedure, which is supported by offhand comments on this 
list, e.g.:

https://www.mail-archive.com/user%40cassandra.apache.org/msg40303.html
https://www.mail-archive.com/user%40cassandra.apache.org/msg44896.html

In other words, it *looks* like the docs are obsolete, and the 
migration process for existing clusters only consists of flipping the 
switch (i.e., adding "-inc" to invocations of "nodetool repair").

Our questions:

1) Is our understanding of the status quo following 2.1.2 correct?  
Does migrating existing clusters to incremental repair only require 
adding the "-inc" argument, or is a process still required?

2) If a process is still required, have there been any changes since 
2.1.2?  Are the docs up-to-date?

3) If there is no process or if the process has changed, are there 
plans on the DataStax side to update the documentation accordingly?

Thanks,
SK