Re: nodetool repair uses option '-local' and '-pr' togather
You're right Christopher, I missed the fact that with RF=3 NTS will always place a replica on us-east-1d, so in this case repair on this node would be sufficient. Thanks for clarifying! 2016-09-05 11:28 GMT-03:00 Christopher Bradford: > If each AZ has a different rack identifier and the keyspace uses > NetworkTopologyStrategy with a replication factor of 3 then the single host > in us-east-1d *will receive 100% of the data*. This is due > to NetworkTopologyStrategy's preference for placing replicas across > different racks before placing a second replica in a rack where data > already resides. Check it out with CCM: > > > ccm node1 status > > Datacenter: us-east-1 > = > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- AddressLoad Tokens Owns (effective) Host ID > Rack > UN 127.0.0.1 98.31 KiB 140.0% > a887ef23-c7ea-4f7a-94a4-1ed12b1caa38 us-east-1b > UN 127.0.0.2 98.31 KiB 140.0% > 30152aaa-cc5e-485d-9b98-1c51f1141155 us-east-1b > UN 127.0.0.3 98.3 KiB 140.0% > 8e1f68f7-571e-4479-bb1f-1ed526fefa9e us-east-1c > UN 127.0.0.4 98.31 KiB 140.0% > 1c9b45ed-02ca-48b5-b619-a87107ff8eba us-east-1c > UN 127.0.0.5 98.31 KiB 140.0% > 2a33751a-c718-44fc-8442-cce9996ebc0c us-east-1d > > cqlsh> CREATE KEYSPACE replication_test WITH replication = {'class': > 'NetworkTopologyStrategy', 'us-east-1': 3}; > > > ccm node1 status replication_test > Datacenter: us-east-1 > = > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- AddressLoad Tokens Owns (effective) Host ID > Rack > UN 127.0.0.1 88.38 KiB 180.0% > a887ef23-c7ea-4f7a-94a4-1ed12b1caa38 us-east-1b > UN 127.0.0.2 98.31 KiB 120.0% > 30152aaa-cc5e-485d-9b98-1c51f1141155 us-east-1b > UN 127.0.0.3 98.3 KiB 180.0% > 8e1f68f7-571e-4479-bb1f-1ed526fefa9e us-east-1c > UN 127.0.0.4 98.31 KiB 120.0% > 1c9b45ed-02ca-48b5-b619-a87107ff8eba us-east-1c > UN 127.0.0.5 98.31 KiB 1100.0% > 2a33751a-c718-44fc-8442-cce9996ebc0c us-east-1d > > This can be tested further with a simple table and nodetool getendpoints. > > > ccm node1 nodetool getendpoints replication_test sample bar > > 127.0.0.2 > 127.0.0.3 > 127.0.0.5 > > > ccm node1 nodetool getendpoints replication_test sample baz > > 127.0.0.1 > 127.0.0.3 > 127.0.0.5 > > > ccm node1 nodetool getendpoints replication_test sample bif > > 127.0.0.3 > 127.0.0.5 > 127.0.0.1 > > > ccm node1 nodetool getendpoints replication_test sample biz > > 127.0.0.2 > 127.0.0.3 > 127.0.0.5 > > On Fri, Sep 2, 2016 at 9:41 AM Paulo Motta > wrote: > >> If I understand the way replication is done, the node in us-east-1d has >> all the (data) replicas, right? >> >> No, for this to be correct, you'd need to have one DC per AZ, which is >> not this case since you have a single DC encompassing multiple AZs. Right >> now, replicas will be spread in 3 distinct AZs, which are represented as >> racks in the single NTS DC if you are using EC2*Snitch. So your best bet is >> probably to run repair -pr in all nodes. >> >> >> 2016-09-01 14:28 GMT-03:00 Li, Guangxing : >> >>> Thanks for the info, Paulo. >>> >>> My cluster is in AWS, the keyspace has replication factor 3 with >>> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in >>> us-east-1c and 1 in us-east-1d. If I understand the way replication is >>> done, the node in us-east-1d has all the (data) replicas, right? If so, if >>> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY >>> on the node in us-east-1d? In other words, does 'nodetool repair' started >>> on node in us-east-1d also cause repairs on replicas on other nodes? I am >>> seeing different answers in discussion like this >>> http://dba.stackexchange.com/questions/82414/do-you-hav >>> e-to-run-nodetool-repair-on-every-node. >>> >>> Thanks again. >>> >>> George >>> >>> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta >>> wrote: >>> https://issues.apache.org/jira/browse/CASSANDRA-7450 2016-09-01 13:11 GMT-03:00 Li, Guangxing : > Hi, > > I have a cluster running 2.0.9 with 2 data centers. I noticed that > 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the > node's data size is 39 GB and the largest SSTable size is like 7 GB so the > column family is not huge, SizeTieredCompactionStrategy is used). > Repairing > a column family on a single node takes over 5 hours. So I am wondering if > I > can use option '-local' and '-pr' together, hoping to get some speed up. > But according to documentation at https://docs.datastax.com/e > n/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr
Re: nodetool repair uses option '-local' and '-pr' togather
If each AZ has a different rack identifier and the keyspace uses NetworkTopologyStrategy with a replication factor of 3 then the single host in us-east-1d *will receive 100% of the data*. This is due to NetworkTopologyStrategy's preference for placing replicas across different racks before placing a second replica in a rack where data already resides. Check it out with CCM: > ccm node1 status Datacenter: us-east-1 = Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- AddressLoad Tokens Owns (effective) Host ID Rack UN 127.0.0.1 98.31 KiB 140.0% a887ef23-c7ea-4f7a-94a4-1ed12b1caa38 us-east-1b UN 127.0.0.2 98.31 KiB 140.0% 30152aaa-cc5e-485d-9b98-1c51f1141155 us-east-1b UN 127.0.0.3 98.3 KiB 140.0% 8e1f68f7-571e-4479-bb1f-1ed526fefa9e us-east-1c UN 127.0.0.4 98.31 KiB 140.0% 1c9b45ed-02ca-48b5-b619-a87107ff8eba us-east-1c UN 127.0.0.5 98.31 KiB 140.0% 2a33751a-c718-44fc-8442-cce9996ebc0c us-east-1d cqlsh> CREATE KEYSPACE replication_test WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east-1': 3}; > ccm node1 status replication_test Datacenter: us-east-1 = Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- AddressLoad Tokens Owns (effective) Host ID Rack UN 127.0.0.1 88.38 KiB 180.0% a887ef23-c7ea-4f7a-94a4-1ed12b1caa38 us-east-1b UN 127.0.0.2 98.31 KiB 120.0% 30152aaa-cc5e-485d-9b98-1c51f1141155 us-east-1b UN 127.0.0.3 98.3 KiB 180.0% 8e1f68f7-571e-4479-bb1f-1ed526fefa9e us-east-1c UN 127.0.0.4 98.31 KiB 120.0% 1c9b45ed-02ca-48b5-b619-a87107ff8eba us-east-1c UN 127.0.0.5 98.31 KiB 1100.0% 2a33751a-c718-44fc-8442-cce9996ebc0c us-east-1d This can be tested further with a simple table and nodetool getendpoints. > ccm node1 nodetool getendpoints replication_test sample bar 127.0.0.2 127.0.0.3 127.0.0.5 > ccm node1 nodetool getendpoints replication_test sample baz 127.0.0.1 127.0.0.3 127.0.0.5 > ccm node1 nodetool getendpoints replication_test sample bif 127.0.0.3 127.0.0.5 127.0.0.1 > ccm node1 nodetool getendpoints replication_test sample biz 127.0.0.2 127.0.0.3 127.0.0.5 On Fri, Sep 2, 2016 at 9:41 AM Paulo Mottawrote: > If I understand the way replication is done, the node in us-east-1d has > all the (data) replicas, right? > > No, for this to be correct, you'd need to have one DC per AZ, which is not > this case since you have a single DC encompassing multiple AZs. Right now, > replicas will be spread in 3 distinct AZs, which are represented as racks > in the single NTS DC if you are using EC2*Snitch. So your best bet is > probably to run repair -pr in all nodes. > > > 2016-09-01 14:28 GMT-03:00 Li, Guangxing : > >> Thanks for the info, Paulo. >> >> My cluster is in AWS, the keyspace has replication factor 3 with >> NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in >> us-east-1c and 1 in us-east-1d. If I understand the way replication is >> done, the node in us-east-1d has all the (data) replicas, right? If so, if >> I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY >> on the node in us-east-1d? In other words, does 'nodetool repair' started >> on node in us-east-1d also cause repairs on replicas on other nodes? I am >> seeing different answers in discussion like this >> http://dba.stackexchange.com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node >> . >> >> Thanks again. >> >> George >> >> On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta >> wrote: >> >>> https://issues.apache.org/jira/browse/CASSANDRA-7450 >>> >>> 2016-09-01 13:11 GMT-03:00 Li, Guangxing : >>> Hi, I have a cluster running 2.0.9 with 2 data centers. I noticed that 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the node's data size is 39 GB and the largest SSTable size is like 7 GB so the column family is not huge, SizeTieredCompactionStrategy is used). Repairing a column family on a single node takes over 5 hours. So I am wondering if I can use option '-local' and '-pr' together, hoping to get some speed up. But according to documentation at https://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr with this option to repair only a local data center...'. Can someone tell me the reason why we should not use options '-local' and '-pr' together? Thanks. George >>> >>> >> >
Re: nodetool repair uses option '-local' and '-pr' togather
If I understand the way replication is done, the node in us-east-1d has all the (data) replicas, right? No, for this to be correct, you'd need to have one DC per AZ, which is not this case since you have a single DC encompassing multiple AZs. Right now, replicas will be spread in 3 distinct AZs, which are represented as racks in the single NTS DC if you are using EC2*Snitch. So your best bet is probably to run repair -pr in all nodes. 2016-09-01 14:28 GMT-03:00 Li, Guangxing: > Thanks for the info, Paulo. > > My cluster is in AWS, the keyspace has replication factor 3 with > NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in > us-east-1c and 1 in us-east-1d. If I understand the way replication is > done, the node in us-east-1d has all the (data) replicas, right? If so, if > I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY > on the node in us-east-1d? In other words, does 'nodetool repair' started > on node in us-east-1d also cause repairs on replicas on other nodes? I am > seeing different answers in discussion like this http://dba.stackexchange. > com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node. > > Thanks again. > > George > > On Thu, Sep 1, 2016 at 10:22 AM, Paulo Motta > wrote: > >> https://issues.apache.org/jira/browse/CASSANDRA-7450 >> >> 2016-09-01 13:11 GMT-03:00 Li, Guangxing : >> >>> Hi, >>> >>> I have a cluster running 2.0.9 with 2 data centers. I noticed that >>> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the >>> node's data size is 39 GB and the largest SSTable size is like 7 GB so the >>> column family is not huge, SizeTieredCompactionStrategy is used). Repairing >>> a column family on a single node takes over 5 hours. So I am wondering if I >>> can use option '-local' and '-pr' together, hoping to get some speed up. >>> But according to documentation at https://docs.datastax.com/e >>> n/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr >>> with this option to repair only a local data center...'. Can someone tell >>> me the reason why we should not use options '-local' and '-pr' together? >>> >>> Thanks. >>> >>> George >>> >> >> >
Re: nodetool repair uses option '-local' and '-pr' togather
Thanks for the info, Paulo. My cluster is in AWS, the keyspace has replication factor 3 with NetworkTopologyStrategy in one DC which have 5 nodes: 2 in us-east-1b, 2 in us-east-1c and 1 in us-east-1d. If I understand the way replication is done, the node in us-east-1d has all the (data) replicas, right? If so, if I do not use '-pr' option, would it be enough to run 'nodetool repair' ONLY on the node in us-east-1d? In other words, does 'nodetool repair' started on node in us-east-1d also cause repairs on replicas on other nodes? I am seeing different answers in discussion like this http://dba.stackexchange.com/questions/82414/do-you-have-to-run-nodetool-repair-on-every-node . Thanks again. George On Thu, Sep 1, 2016 at 10:22 AM, Paulo Mottawrote: > https://issues.apache.org/jira/browse/CASSANDRA-7450 > > 2016-09-01 13:11 GMT-03:00 Li, Guangxing : > >> Hi, >> >> I have a cluster running 2.0.9 with 2 data centers. I noticed that >> 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the >> node's data size is 39 GB and the largest SSTable size is like 7 GB so the >> column family is not huge, SizeTieredCompactionStrategy is used). Repairing >> a column family on a single node takes over 5 hours. So I am wondering if I >> can use option '-local' and '-pr' together, hoping to get some speed up. >> But according to documentation at https://docs.datastax.com/e >> n/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr with >> this option to repair only a local data center...'. Can someone tell me the >> reason why we should not use options '-local' and '-pr' together? >> >> Thanks. >> >> George >> > >
Re: nodetool repair uses option '-local' and '-pr' togather
https://issues.apache.org/jira/browse/CASSANDRA-7450 2016-09-01 13:11 GMT-03:00 Li, Guangxing: > Hi, > > I have a cluster running 2.0.9 with 2 data centers. I noticed that > 'nodetool repair -pr keyspace cf' runs very slow (OpsCenter shows that the > node's data size is 39 GB and the largest SSTable size is like 7 GB so the > column family is not huge, SizeTieredCompactionStrategy is used). Repairing > a column family on a single node takes over 5 hours. So I am wondering if I > can use option '-local' and '-pr' together, hoping to get some speed up. > But according to documentation at https://docs.datastax.com/ > en/cassandra/2.0/cassandra/tools/toolsRepair.html '...Do not use -pr with > this option to repair only a local data center...'. Can someone tell me the > reason why we should not use options '-local' and '-pr' together? > > Thanks. > > George >