Anyone have anymore thoughts on this at all? Struggling to understand it..
> On 9 Jun 2017, at 11:32, Chris Stokesmore <chris.elsm...@demandlogic.co>
> Hi Anuj,
> Thanks for the reply.
> 1). We are using Cassandra 2.2.8, and our repair commands we are comparing
> "nodetool repair --in-local-dc --partitioner-range” and
> "nodetool repair --in-local-dc”
> Since 2.2 I believe inc repairs are the default - that seems to be confirmed
> in the logs that list the repair details when a repair starts.
> 2) From looks at a few runsr, on average:
> with -pr repairs, each node is approx 6.5 - 8 hours, so a total over the 7
> nodes of 53 hours
> With just inc repairs, each node ~26 - 29 hours, so a total of 193
> 3) we currently have two DCs in total, the ‘production’ ring with 7 nodes and
> RF=3, and a testing ring with one single node and RF=1 for our single
> keyspace we currently use.
> 4) Yeah that number came from the Cassandra repair logs from an inc repair, I
> can share the number reports when using a pr repair later this evening when
> the currently running repair has completed.
> Many thanks for the reply again,
>> On 6 Jun 2017, at 17:50, Anuj Wadehra <anujw_2...@yahoo.co.in
>> <mailto:anujw_2...@yahoo.co.in>> wrote:
>> Hi Chris,
>> Can your share following info:
>> 1. Exact repair commands you use for inc repair and pr repair
>> 2. Repair time should be measured at cluster level for inc repair. So, whats
>> the total time it takes to run repair on all nodes for incremental vs pr
>> 3. You are repairing one dc DC3. How many DCs are there in total and whats
>> the RF for keyspaces? Running pr on a specific dc would not repair entire
>> 4. 885 ranges? From where did you get this number? Logs? Can you share the
>> number ranges printed in logs for both inc and pr case?
>> Sent from Yahoo Mail on Android
>> On Tue, Jun 6, 2017 at 9:33 PM, Chris Stokesmore
>> <chris.elsm...@demandlogic.co <mailto:chris.elsm...@demandlogic.co>> wrote:
>> Thank you for the excellent and clear description of the different versions
>> of repair Anuj, that has cleared up what I expect to be happening.
>> The problem now is in our cluster, we are running repairs with options
>> (parallelism: parallel, primary range: false, incremental: true, job
>> threads: 1, ColumnFamilies: , dataCenters: [DC3], hosts: , # of ranges:
>> 885) and when we do our repairs are taking over a day to complete when
>> previously when running with the partition range option they were taking
>> more like 8-9 hours.
>> As I understand it, using incremental should have sped this process up as
>> all three sets of data on each repair job should be marked as repaired
>> however this does not seem to be the case. Any ideas?
>>> On 6 Jun 2017, at 16:08, Anuj Wadehra <anujw_2...@yahoo.co.in.INVALID
>>> <mailto:anujw_2...@yahoo.co.in.INVALID>> wrote:
>>> Hi Chris,
>>> Using pr with incremental repairs does not make sense. Primary range repair
>>> is an optimization over full repair. If you run full repair on a n node
>>> cluster with RF=3, you would be repairing each data thrice.
>>> E.g. in a 5 node cluster with RF=3, a range may exist on node A,B and C .
>>> When full repair is run on node A, the entire data in that range gets
>>> synced with replicas on node B and C. Now, when you run full repair on
>>> nodes B and C, you are wasting resources on repairing data which is already
>>> Primary range repair ensures that when you run repair on a node, it ONLY
>>> repairs the data which is owned by the node. Thus, no node repairs data
>>> which is not owned by it and must be repaired by other node. Redundant work
>>> is eliminated.
>>> Even in pr, each time you run pr on all nodes, you repair 100% of data. Why
>>> to repair complete data in each cycle?? ..even data which has not even
>>> changed since the last repair cycle?
>>> This is where Incremental repair comes as an improvement. Once repaired, a
>>> data would be marked repaired so that the next repair cycle could just
>>> focus on repairing the delta. Now, lets go back to the example of 5 node
>>> cluster with RF =3.This time we run incremental repair on all nodes. When
>>> you repair entire data on node A, all 3 replicas are marked as repaired.
>>> Even if you run inc repair on all ranges on the second node, you would not
>>> re-repair the already repaired data. Thus, there is no advantage of
>>> repairing only the data owned by the node (primary range of the node). You
>>> can run inc repair on all the data present on a node and Cassandra would
>>> make sure that when you repair data on other nodes, you only repair
>>> unrepaired data.
>>> Sent from Yahoo Mail on Android
>>> On Tue, Jun 6, 2017 at 4:27 PM, Chris Stokesmore
>>> <chris.elsm...@demandlogic.co <mailto:chris.elsm...@demandlogic.co>> wrote:
>>> Hi all,
>>> Wondering if anyone had any thoughts on this? At the moment the long
>>> running repairs cause us to be running them on two nodes at once for a bit
>>> of time, which obivould increases the cluster load.
>>> On 2017-05-25 16:18 (+0100), Chris Stokesmore <c...@demandlogic.co
>>> <mailto:c...@demandlogic.co>> wrote:
>>> > Hi,>
>>> > We are running a 7 node Cassandra 2.2.8 cluster, RF=3, and had been
>>> > running repairs with the -pr option, via a cron job that runs on each
>>> > node once per week.>
>>> > We changed that as some advice on the Cassandra IRC channel said it would
>>> > cause more anticompaction and
>>> > http://docs.datastax.com/en/archived/cassandra/2.2/cassandra/tools/toolsRepair.html
>>> > <http://docs.datastax.com/en/archived/cassandra/2.2/cassandra/tools/toolsRepair.html>says
>>> > 'Performing partitioner range repairs by using the -pr option is
>>> > generally considered a good choice for doing manual repairs. However,
>>> > this option cannot be used with incremental repairs (default for
>>> > Cassandra 2.2 and later)'
>>> > Only problem is our -pr repairs were taking about 8 hours, and now the
>>> > non-pr repair are taking 24+ - I guess this makes sense, repairing 1/7 of
>>> > data increased to 3/7, except I was hoping to see a speed up after the
>>> > first loop through the cluster as each repair will be marking much more
>>> > data as repaired, right?>
>>> > Is running -pr with incremental repairs really that bad? >
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org