Re: nodetool repair uses option '-local' and '-pr' togather

[Paulo Motta]

You're right Christopher, I missed the fact that with RF=3 NTS will always
place a replica on us-east-1d, so in this case repair on this node would be
sufficient. Thanks for clarifying!

2016-09-05 11:28 GMT-03:00 Christopher Bradford :

> If each AZ has a different rack identifier and the keyspace uses
> NetworkTopologyStrategy with a replication factor of 3 then the single host
> in us-east-1d *will receive 100% of the data*. This is due
> to NetworkTopologyStrategy's preference for placing replicas across
> different racks before placing a second replica in a rack where data
> already resides. Check it out with CCM:
>
> > ccm node1 status
>
> Datacenter: us-east-1
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID
> Rack
> UN  127.0.0.1  98.31 KiB  140.0%
> a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
> UN  127.0.0.2  98.31 KiB  140.0%
> 30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
> UN  127.0.0.3  98.3 KiB   140.0%
> 8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
> UN  127.0.0.4  98.31 KiB  140.0%
> 1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
> UN  127.0.0.5  98.31 KiB  140.0%
> 2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d
>
> cqlsh> CREATE KEYSPACE replication_test WITH replication = {'class':
> 'NetworkTopologyStrategy', 'us-east-1': 3};
>
> > ccm node1 status replication_test
> Datacenter: us-east-1
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID
> Rack
> UN  127.0.0.1  88.38 KiB  180.0%
> a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
> UN  127.0.0.2  98.31 KiB  120.0%
> 30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
> UN  127.0.0.3  98.3 KiB   180.0%
> 8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
> UN  127.0.0.4  98.31 KiB  120.0%
> 1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
> UN  127.0.0.5  98.31 KiB  1100.0%
>  2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d
>
> This can be tested further with a simple table and nodetool getendpoints.
>
> > ccm node1 nodetool getendpoints replication_test sample bar
>
> 127.0.0.2
> 127.0.0.3
> 127.0.0.5
>
> > ccm node1 nodetool getendpoints replication_test sample baz
>
> 127.0.0.1
> 127.0.0.3
> 127.0.0.5
>
> > ccm node1 nodetool getendpoints replication_test sample bif
>
> 127.0.0.3
> 127.0.0.5
> 127.0.0.1
>
> > ccm node1 nodetool getendpoints replication_test sample biz
>
> 127.0.0.2
> 127.0.0.3
> 127.0.0.5
>
> On Fri, Sep 2, 2016 at 9:41 AM Paulo Motta 
> wrote:
>
>> If I understand the way replication is done, the node in us-east-1d has
>> all the (data) replicas, right?
>>
>> No, for this to be correct, you'd need to have one DC per AZ, which is
>> not this case since you have a single DC encompassing multiple AZs. Right
>> now, replicas will be spread in 3 distinct AZs, which are represented as
>> racks in the single NTS DC if you are using EC2*Snitch. So your best bet is
>> probably to run repair -pr in all nodes.
>>
>>
>> 2016-09-01 14:28 GMT-03:00 Li, Guangxing :
>>
>>> Thanks for the info, Paulo.
>>>
>>> My cluster is in AWS, the keyspace has replication factor 3 with
>>> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
>>> us-east-1c and 1 in us-east-1d. If I understand the way replication is
>>> done, the node in us-east-1d has all the (data) replicas, right? If so, if
>>> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
>>> on the node in us-east-1d? In other words, does 'nodetool repair' started
>>> on node in us-east-1d also cause repairs on replicas on other nodes? I am
>>> seeing different answers in discussion like this
>>> http://dba.stackexchange.com/questions/82414/do-you-hav
>>> e-to-run-nodetool-repair-on-every-node.
>>>
>>> Thanks again.
>>>
>>> George
>>>
>>> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta 
>>> wrote:
>>>
 https://issues.apache.org/jira/browse/CASSANDRA-7450

 2016-09-01 13:11 GMT-03:00 Li, Guangxing :

> Hi,
>
> I have a cluster running 2.0.9 with 2 data centers. I noticed that
> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
> node's data size is 39 GB and the largest SSTable size is like 7 GB so the
> column family is not huge, SizeTieredCompactionStrategy is used). 
> Repairing
> a column family on a single node takes over 5 hours. So I am wondering if 
> I
> can use option '-local' and '-pr' together, hoping to get some speed up.
> But according to documentation at https://docs.datastax.com/e
> n/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr

Re: nodetool repair uses option '-local' and '-pr' togather

[Christopher Bradford]

If each AZ has a different rack identifier and the keyspace uses
NetworkTopologyStrategy with a replication factor of 3 then the single host
in us-east-1d *will receive 100% of the data*. This is due
to NetworkTopologyStrategy's preference for placing replicas across
different racks before placing a second replica in a rack where data
already resides. Check it out with CCM:

> ccm node1 status

Datacenter: us-east-1
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens   Owns (effective)  Host ID
Rack
UN  127.0.0.1  98.31 KiB  140.0%
a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
UN  127.0.0.2  98.31 KiB  140.0%
30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
UN  127.0.0.3  98.3 KiB   140.0%
8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
UN  127.0.0.4  98.31 KiB  140.0%
1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
UN  127.0.0.5  98.31 KiB  140.0%
2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d

cqlsh> CREATE KEYSPACE replication_test WITH replication = {'class':
'NetworkTopologyStrategy', 'us-east-1': 3};

> ccm node1 status replication_test
Datacenter: us-east-1
=
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens   Owns (effective)  Host ID
Rack
UN  127.0.0.1  88.38 KiB  180.0%
a887ef23-c7ea-4f7a-94a4-1ed12b1caa38  us-east-1b
UN  127.0.0.2  98.31 KiB  120.0%
30152aaa-cc5e-485d-9b98-1c51f1141155  us-east-1b
UN  127.0.0.3  98.3 KiB   180.0%
8e1f68f7-571e-4479-bb1f-1ed526fefa9e  us-east-1c
UN  127.0.0.4  98.31 KiB  120.0%
1c9b45ed-02ca-48b5-b619-a87107ff8eba  us-east-1c
UN  127.0.0.5  98.31 KiB  1100.0%
 2a33751a-c718-44fc-8442-cce9996ebc0c  us-east-1d

This can be tested further with a simple table and nodetool getendpoints.

> ccm node1 nodetool getendpoints replication_test sample bar

127.0.0.2
127.0.0.3
127.0.0.5

> ccm node1 nodetool getendpoints replication_test sample baz

127.0.0.1
127.0.0.3
127.0.0.5

> ccm node1 nodetool getendpoints replication_test sample bif

127.0.0.3
127.0.0.5
127.0.0.1

> ccm node1 nodetool getendpoints replication_test sample biz

127.0.0.2
127.0.0.3
127.0.0.5

On Fri, Sep 2, 2016 at 9:41 AM Paulo Motta  wrote:

> If I understand the way replication is done, the node in us-east-1d has
> all the (data) replicas, right?
>
> No, for this to be correct, you'd need to have one DC per AZ, which is not
> this case since you have a single DC encompassing multiple AZs. Right now,
> replicas will be spread in 3 distinct AZs, which are represented as racks
> in the single NTS DC if you are using EC2*Snitch. So your best bet is
> probably to run repair -pr in all nodes.
>
>
> 2016-09-01 14:28 GMT-03:00 Li, Guangxing :
>
>> Thanks for the info, Paulo.
>>
>> My cluster is in AWS, the keyspace has replication factor 3 with
>> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
>> us-east-1c and 1 in us-east-1d. If I understand the way replication is
>> done, the node in us-east-1d has all the (data) replicas, right? If so, if
>> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
>> on the node in us-east-1d? In other words, does 'nodetool repair' started
>> on node in us-east-1d also cause repairs on replicas on other nodes? I am
>> seeing different answers in discussion like this
>> http://dba.stackexchange.com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node
>> .
>>
>> Thanks again.
>>
>> George
>>
>> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta 
>> wrote:
>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-7450
>>>
>>> 2016-09-01 13:11 GMT-03:00 Li, Guangxing :
>>>
 Hi,

 I have a cluster running 2.0.9 with 2 data centers. I noticed that
 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
 node's data size is 39 GB and the largest SSTable size is like 7 GB so the
 column family is not huge, SizeTieredCompactionStrategy is used). Repairing
 a column family on a single node takes over 5 hours. So I am wondering if I
 can use option '-local' and '-pr' together, hoping to get some speed up.
 But according to documentation at
 https://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsRepair.html
 '...Do not use -pr with this option to repair only a local data
 center...'. Can someone tell me the reason why we should not use options
 '-local' and '-pr' together?

 Thanks.

 George

>>>
>>>
>>
>

Re: nodetool repair uses option '-local' and '-pr' togather

[Paulo Motta]

 If I understand the way replication is done, the node in us-east-1d has
all the (data) replicas, right?

No, for this to be correct, you'd need to have one DC per AZ, which is not
this case since you have a single DC encompassing multiple AZs. Right now,
replicas will be spread in 3 distinct AZs, which are represented as racks
in the single NTS DC if you are using EC2*Snitch. So your best bet is
probably to run repair -pr in all nodes.


2016-09-01 14:28 GMT-03:00 Li, Guangxing :

> Thanks for the info, Paulo.
>
> My cluster is in AWS, the keyspace has replication factor 3 with
> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
> us-east-1c and 1 in us-east-1d. If I understand the way replication is
> done, the node in us-east-1d has all the (data) replicas, right? If so, if
> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
> on the node in us-east-1d? In other words, does 'nodetool repair' started
> on node in us-east-1d also cause repairs on replicas on other nodes? I am
> seeing different answers in discussion like this http://dba.stackexchange.
> com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node.
>
> Thanks again.
>
> George
>
> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta 
> wrote:
>
>> https://issues.apache.org/jira/browse/CASSANDRA-7450
>>
>> 2016-09-01 13:11 GMT-03:00 Li, Guangxing :
>>
>>> Hi,
>>>
>>> I have a cluster running 2.0.9 with 2 data centers. I noticed that
>>> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
>>> node's data size is 39 GB and the largest SSTable size is like 7 GB so the
>>> column family is not huge, SizeTieredCompactionStrategy is used). Repairing
>>> a column family on a single node takes over 5 hours. So I am wondering if I
>>> can use option '-local' and '-pr' together, hoping to get some speed up.
>>> But according to documentation at https://docs.datastax.com/e
>>> n/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr
>>> with this option to repair only a local data center...'. Can someone tell
>>> me the reason why we should not use options '-local' and '-pr' together?
>>>
>>> Thanks.
>>>
>>> George
>>>
>>
>>
>

Re: nodetool repair uses option '-local' and '-pr' togather

[Li, Guangxing]

Thanks for the info, Paulo.

My cluster is in AWS, the keyspace has replication factor 3 with
NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
us-east-1c and 1 in us-east-1d. If I understand the way replication is
done, the node in us-east-1d has all the (data) replicas, right? If so, if
I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
on the node in us-east-1d? In other words, does 'nodetool repair' started
on node in us-east-1d also cause repairs on replicas on other nodes? I am
seeing different answers in discussion like this
http://dba.stackexchange.com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node
.

Thanks again.

George

On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta 
wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-7450
>
> 2016-09-01 13:11 GMT-03:00 Li, Guangxing :
>
>> Hi,
>>
>> I have a cluster running 2.0.9 with 2 data centers. I noticed that
>> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
>> node's data size is 39 GB and the largest SSTable size is like 7 GB so the
>> column family is not huge, SizeTieredCompactionStrategy is used). Repairing
>> a column family on a single node takes over 5 hours. So I am wondering if I
>> can use option '-local' and '-pr' together, hoping to get some speed up.
>> But according to documentation at https://docs.datastax.com/e
>> n/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr with
>> this option to repair only a local data center...'. Can someone tell me the
>> reason why we should not use options '-local' and '-pr' together?
>>
>> Thanks.
>>
>> George
>>
>
>

Re: nodetool repair uses option '-local' and '-pr' togather

[Paulo Motta]

https://issues.apache.org/jira/browse/CASSANDRA-7450

2016-09-01 13:11 GMT-03:00 Li, Guangxing :

> Hi,
>
> I have a cluster running 2.0.9 with 2 data centers. I noticed that
> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
> node's data size is 39 GB and the largest SSTable size is like 7 GB so the
> column family is not huge, SizeTieredCompactionStrategy is used). Repairing
> a column family on a single node takes over 5 hours. So I am wondering if I
> can use option '-local' and '-pr' together, hoping to get some speed up.
> But according to documentation at https://docs.datastax.com/
> en/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr with
> this option to repair only a local data center...'. Can someone tell me the
> reason why we should not use options '-local' and '-pr' together?
>
> Thanks.
>
> George
>

nodetool repair uses option '-local' and '-pr' togather

[Li, Guangxing]

Hi,

I have a cluster running 2.0.9 with 2 data centers. I noticed that
'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
node's data size is 39 GB and the largest SSTable size is like 7 GB so the
column family is not huge, SizeTieredCompactionStrategy is used). Repairing
a column family on a single node takes over 5 hours. So I am wondering if I
can use option '-local' and '-pr' together, hoping to get some speed up.
But according to documentation at
https://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsRepair.html
'...Do not use -pr with this option to repair only a local data center...'.
Can someone tell me the reason why we should not use options '-local' and
'-pr' together?

Thanks.

George