[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162539#comment-14162539
]
Robert Coli commented on CASSANDRA-7317:
{quote}
If -pr is meant to be used that way, I don't think it is communicated very
well.
{quote}
As a meta-aside, the way that I explain to new users what -pr is for is by
saying :
If you are repairing ALL nodes in your cluster, use -pr. If you are not
repairing ALL nodes, do not use -pr.
I agree that the docs could make this idea clearer, of course with the new
-local wrinkle reflected.
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Assignee: Yuki Morishita
Fix For: 2.0.9
Attachments: 7317-2.0.txt, Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14035647#comment-14035647
]
Marcus Eriksson commented on CASSANDRA-7317:
Comment:
* looks like you missed passing repairingRange to forceKeyspaceRepairRange(..)
in one place (row 2610)
Nit:
* could we break out the range splitting in a separate method, we are doing
exactly the same thing in two places?
other than that, +1
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Assignee: Yuki Morishita
Fix For: 2.0.9
Attachments: 7317-2.0.txt, Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017659#comment-14017659
]
Nick Bailey commented on CASSANDRA-7317:
Well we at least have some sort of special logic for the case where a keyspace
doesn't exist in both datacenters, since -pr in that case repaired 2 ranges. I
do think the right solution is to make TokenMetadata KS/DC aware at least for
these cases.
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Assignee: Yuki Morishita
Fix For: 2.0.9
Attachments: Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017839#comment-14017839
]
Jonathan Ellis commented on CASSANDRA-7317:
---
bq. I'm not sure if that's expected or not
Sure sounds like a bug to me.
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Assignee: Yuki Morishita
Fix For: 2.0.9
Attachments: Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018073#comment-14018073
]
Yuki Morishita commented on CASSANDRA-7317:
---
Isn't -pr supposed to use when repairing *all* nodes in the cluster? I think
current behavior works for this purpose.
TokenMetadata has been representing a ring and we only have one per cluster.
Multi-DC support is built on top of it.
So currently, from coding perspective, multi-DC set up looks like the image
that Mikhail put previously (in spite of mental model some people have).
It needs some effort and consideration to make TokenMetadata to be KS/DC aware.
One thing that I noticed during some experiment is that repair should accept
-st and -et considering multi-DC replication. (accept -st -10 -et
-9223372036854775808 when primary ranges are (-10,0],(0,-9223372036854775808]).
Right now it just check each ranges against given -st/-et range so this ends up
IAE.
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Assignee: Yuki Morishita
Fix For: 2.0.9
Attachments: Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018294#comment-14018294
]
Nick Bailey commented on CASSANDRA-7317:
If -pr is meant to be used that way, I don't think it is communicated very
well. If that is the case we should be clearer in the documentation and I think
we would still need to fix combining -pr and -local. Repairing just the local
dc and using the -pr flag should be a supported use case. Personally it seems
to me like the best way to fix that is to make the definition of -pr, dc aware,
but perhaps not.
My bigger issue is the validation of the range when specifying a range, so I
agree that should be fixed.
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Assignee: Yuki Morishita
Fix For: 2.0.9
Attachments: Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016258#comment-14016258
]
Nick Bailey commented on CASSANDRA-7317:
Another thing that was brought up on the mailing list (at least incidentally),
is that the current implementation basically makes the '-local' flag and the
'-pr' flag incompatible.
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Fix For: 2.0.9
Attachments: Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017327#comment-14017327
]
Mikhail Stepura commented on CASSANDRA-7317:
Current behavior, when {{-pr}} is specified, is to treat a multi-DC setup as a
single ring. Because {{TokenMetadata.getPredecessor(Token)}} doesn't take into
account a DC for a token, and just search for a predecessor across all tokens
from all DCs.
I'm not sure if that's expected or not
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Assignee: Yuki Morishita
Fix For: 2.0.9
Attachments: Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012930#comment-14012930
]
Mikhail Stepura commented on CASSANDRA-7317:
bq. The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
The catch for IllegalArgumentException doesn't extract an error message from
the exception, and just reuses the previous value of the message, that's why
you see two identical lines in the output
https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/service/StorageService.java#L2621
{code}
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
{code}
That can be easily fixed adding something like {{message =
String.format(Ivalid repair command #%d: %s , cmd, e.getMessage());}}
I can't comment on your concern about a validity of a command, though.
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Fix For: 2.0.9
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012954#comment-14012954
]
Mikhail Stepura commented on CASSANDRA-7317:
BTW, it looks like you misplaced start and end tokens, {{-st -10 -et
-9223372036854775808}}. Shouldn't a start token be less that the end one?
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Fix For: 2.0.9
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012961#comment-14012961
]
Nick Bailey commented on CASSANDRA-7317:
Well there are only two nodes, so this is the wraparound case. If we were
repairing node 2 it would be (-9223372036854775808, -10].
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Fix For: 2.0.9
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013260#comment-14013260
]
Nick Bailey commented on CASSANDRA-7317:
Perhaps it is a difference in mental models, but let me try to explain my
reasoning. In general, I think datacenter's should be considered as separate
rings. Replication is configured separately for each datacenter, tokens are
balanced separately in each datacenter (for non vnodes), and consistency levels
can be specified with specific datacenter requirements.
To somewhat further illustrate my point, Cassandra agrees with me when you
modify the schema of the system_traces keyspace to only exist in dc 1:
{noformat}
[Nicks-MacBook-Pro:22:26:41 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-29 22:26:46,958] Starting repair command #3, repairing 2 ranges for
keyspace system_traces
[2014-05-29 22:26:47,148] Repair session 3ae1cce0-e7aa-11e3-aaee-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-29 22:26:47,149] Repair session 3afc80d0-e7aa-11e3-aaee-5f8011daec21
for range (-10,0] finished
[2014-05-29 22:26:47,149] Repair command #3 finished
[Nicks-MacBook-Pro:22:26:47 cassandra-2.0] cassandra$ bin/nodetool -p 7300
repair -pr system_traces
[2014-05-29 22:26:54,907] Nothing to repair for keyspace 'system_traces'
[Nicks-MacBook-Pro:22:34:55 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -9223372036854775808 -et 10 system_traces
[2014-05-29 22:35:02,604] Starting repair command #6, repairing 1 ranges for
keyspace system_traces
[2014-05-29 22:35:02,604] Starting repair command #6, repairing 1 ranges for
keyspace system_traces
[2014-05-29 22:35:02,604] Repair command #6 finished
{noformat}
Repairing the 'primary range' of node1 actually repairs two ranges (although
those two ranges are really just one). Repairing the primary range of node3
does nothing. And asking C* to repair the entire range that it just repaired as
two separate ranges still fails.
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Fix For: 2.0.9
Attachments: Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges
[jira] [Commented] (CASSANDRA-7317) Repair range validation and calculation is off
[
https://issues.apache.org/jira/browse/CASSANDRA-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013262#comment-14013262
]
Nick Bailey commented on CASSANDRA-7317:
Even if we were to decide that this is the correct behavior, it gets hard to
reason about as you increase tokens, or datacenters. If we expand the above
example to three datacenters and the keyspace exists in dc1 and dc2 but not
dc3, which ranges is dc1 the primary for and which ranges is dc2 the primary
for. Somebody will have to be responsible for the ranges dc3 'owns'.
Repair range validation and calculation is off
--
Key: CASSANDRA-7317
URL: https://issues.apache.org/jira/browse/CASSANDRA-7317
Project: Cassandra
Issue Type: Bug
Reporter: Nick Bailey
Fix For: 2.0.9
Attachments: Untitled Diagram(1).png
From what I can tell the calculation (using the -pr option) and validation of
tokens for repairing ranges is broken. Or at least should be improved. Using
an example with ccm:
Nodetool ring:
{noformat}
Datacenter: dc1
==
AddressRackStatus State LoadOwns
Token
-10
127.0.0.1 r1 Up Normal 188.96 KB 50.00%
-9223372036854775808
127.0.0.2 r1 Up Normal 194.77 KB 50.00% -10
Datacenter: dc2
==
AddressRackStatus State LoadOwns
Token
0
127.0.0.4 r1 Up Normal 160.58 KB 0.00%
-9223372036854775798
127.0.0.3 r1 Up Normal 139.46 KB 0.00% 0
{noformat}
Schema:
{noformat}
CREATE KEYSPACE system_traces WITH replication = {
'class': 'NetworkTopologyStrategy',
'dc2': '2',
'dc1': '2'
};
{noformat}
Repair -pr:
{noformat}
[Nicks-MacBook-Pro:21:35:58 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -pr system_traces
[2014-05-28 21:36:01,977] Starting repair command #12, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:02,207] Repair session f984d290-e6d9-11e3-9edc-5f8011daec21
for range (0,-9223372036854775808] finished
[2014-05-28 21:36:02,207] Repair command #12 finished
[Nicks-MacBook-Pro:21:36:02 cassandra-2.0] cassandra$ bin/nodetool -p 7200
repair -pr system_traces
[2014-05-28 21:36:14,086] Starting repair command #1, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:36:14,406] Repair session 00bd45b0-e6da-11e3-98fc-5f8011daec21
for range (-9223372036854775798,-10] finished
[2014-05-28 21:36:14,406] Repair command #1 finished
{noformat}
Note that repairing both nodes in dc1, leaves very small ranges unrepaired.
For example (-10,0]. Repairing the 'primary range' in dc2 will repair those
small ranges. Maybe that is the behavior we want but it seems
counterintuitive.
The behavior when manually trying to repair the full range of 127.0.0.01
definitely needs improvement though.
Repair command:
{noformat}
[Nicks-MacBook-Pro:21:50:44 cassandra-2.0] cassandra$ bin/nodetool -p 7100
repair -st -10 -et -9223372036854775808 system_traces
[2014-05-28 21:50:55,803] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Starting repair command #17, repairing 1 ranges for
keyspace system_traces
[2014-05-28 21:50:55,804] Repair command #17 finished
[Nicks-MacBook-Pro:21:50:56 cassandra-2.0] cassandra$ echo $?
1
{noformat}
system.log:
{noformat}
ERROR [Thread-96] 2014-05-28 21:40:05,921 StorageService.java (line 2621)
Repair session failed:
java.lang.IllegalArgumentException: Requested range intersects a local range
but is not fully contained in one; this would lead to imprecise repair
{noformat}
* The actual output of the repair command doesn't really indicate that there
was an issue. Although the command does return with a non zero exit status.
* The error here is invisible if you are using the synchronous jmx repair
api. It will appear as though the repair completed successfully.
* Personally, I believe that should be a valid repair command. For the
system_traces keyspace, 127.0.0.1 is responsible for this range (and I would
argue the 'primary range' of the node).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
