If each AZ has a different rack identifier and the keyspace uses
NetworkTopologyStrategy with a replication factor of 3 then the single host
in us-east-1d *will receive 100% of the data*. This is due
to NetworkTopologyStrategy's preference for placing replicas across
different racks before placing a second replica in a rack where data
already resides. Check it out with CCM:
> ccm node1 status
Datacenter: us-east-1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 127.0.0.1 98.31 KiB 1 40.0%
a887ef23-c7ea-4f7a-94a4-1ed12b1caa38 us-east-1b
UN 127.0.0.2 98.31 KiB 1 40.0%
30152aaa-cc5e-485d-9b98-1c51f1141155 us-east-1b
UN 127.0.0.3 98.3 KiB 1 40.0%
8e1f68f7-571e-4479-bb1f-1ed526fefa9e us-east-1c
UN 127.0.0.4 98.31 KiB 1 40.0%
1c9b45ed-02ca-48b5-b619-a87107ff8eba us-east-1c
UN 127.0.0.5 98.31 KiB 1 40.0%
2a33751a-c718-44fc-8442-cce9996ebc0c us-east-1d
cqlsh> CREATE KEYSPACE replication_test WITH replication = {'class':
'NetworkTopologyStrategy', 'us-east-1': 3};
> ccm node1 status replication_test
Datacenter: us-east-1
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 127.0.0.1 88.38 KiB 1 80.0%
a887ef23-c7ea-4f7a-94a4-1ed12b1caa38 us-east-1b
UN 127.0.0.2 98.31 KiB 1 20.0%
30152aaa-cc5e-485d-9b98-1c51f1141155 us-east-1b
UN 127.0.0.3 98.3 KiB 1 80.0%
8e1f68f7-571e-4479-bb1f-1ed526fefa9e us-east-1c
UN 127.0.0.4 98.31 KiB 1 20.0%
1c9b45ed-02ca-48b5-b619-a87107ff8eba us-east-1c
UN 127.0.0.5 98.31 KiB 1 100.0%
2a33751a-c718-44fc-8442-cce9996ebc0c us-east-1d
This can be tested further with a simple table and nodetool getendpoints.
> ccm node1 nodetool getendpoints replication_test sample bar
127.0.0.2
127.0.0.3
127.0.0.5
> ccm node1 nodetool getendpoints replication_test sample baz
127.0.0.1
127.0.0.3
127.0.0.5
> ccm node1 nodetool getendpoints replication_test sample bif
127.0.0.3
127.0.0.5
127.0.0.1
> ccm node1 nodetool getendpoints replication_test sample biz
127.0.0.2
127.0.0.3
127.0.0.5
On Fri, Sep 2, 2016 at 9:41 AM Paulo Motta <[email protected]> wrote:
> If I understand the way replication is done, the node in us-east-1d has
> all the (data) replicas, right?
>
> No, for this to be correct, you'd need to have one DC per AZ, which is not
> this case since you have a single DC encompassing multiple AZs. Right now,
> replicas will be spread in 3 distinct AZs, which are represented as racks
> in the single NTS DC if you are using EC2*Snitch. So your best bet is
> probably to run repair -pr in all nodes.
>
>
> 2016-09-01 14:28 GMT-03:00 Li, Guangxing <[email protected]>:
>
>> Thanks for the info, Paulo.
>>
>> My cluster is in AWS, the keyspace has replication factor 3 with
>> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in
>> us-east-1c and 1 in us-east-1d. If I understand the way replication is
>> done, the node in us-east-1d has all the (data) replicas, right? If so, if
>> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY
>> on the node in us-east-1d? In other words, does 'nodetool repair' started
>> on node in us-east-1d also cause repairs on replicas on other nodes? I am
>> seeing different answers in discussion like this
>> http://dba.stackexchange.com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node
>> .
>>
>> Thanks again.
>>
>> George
>>
>> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta <[email protected]>
>> wrote:
>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-7450
>>>
>>> 2016-09-01 13:11 GMT-03:00 Li, Guangxing <[email protected]>:
>>>
>>>> Hi,
>>>>
>>>> I have a cluster running 2.0.9 with 2 data centers. I noticed that
>>>> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the
>>>> node's data size is 39 GB and the largest SSTable size is like 7 GB so the
>>>> column family is not huge, SizeTieredCompactionStrategy is used). Repairing
>>>> a column family on a single node takes over 5 hours. So I am wondering if I
>>>> can use option '-local' and '-pr' together, hoping to get some speed up.
>>>> But according to documentation at
>>>> https://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsRepair.html
>>>> '...Do not use -pr with this option to repair only a local data
>>>> center...'. Can someone tell me the reason why we should not use options
>>>> '-local' and '-pr' together?
>>>>
>>>> Thanks.
>>>>
>>>> George
>>>>
>>>
>>>
>>
>
