[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[
https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-14740:
Fix Version/s: (was: 4.0-beta)
4.0-alpha
Since Version: 4.0-alpha
Source Control Link:
https://github.com/apache/cassandra/commit/0f22dab1a015cb84d9857f940de5a256bfbee083
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Committed to trunk in {{0f22dab1a015cb84d9857f940de5a256bfbee083}}. Final CI
run after rebasing with only previously known failures:
[jdk8|https://circleci.com/workflow-run/3cb3c20e-6dea-4fd6-8aff-193e1e7d298b] ;
[jdk11|https://circleci.com/workflow-run/024ba3cd-c112-4f73-a562-b364ea420e3c]
> BlockingReadRepair does not maintain monotonicity during range movements
>
>
> Key: CASSANDRA-14740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14740
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Urgent
> Labels: correctness
> Fix For: 4.0, 4.0-alpha
>
>
> The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each
> of the queried nodes are written to, but pending nodes are not considered.
> If there is a pending range movement, one of these writes may be ‘lost’ when
> the range movement completes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-14740: Status: Ready to Commit (was: Changes Suggested) > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Urgent > Labels: correctness > Fix For: 4.0, 4.0-beta > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[
https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-14740:
Status: Changes Suggested (was: Review In Progress)
This is nice. As well as addressing the bug, using a {{ReplicaPlan}} for the
writes really clarifies {{BlockingPartitionRepair}} & {{BlockingReadRepair}}.
There's a copy/paste bug in {{RowIteratorMergeListener::applyToPartition}} in
the {{buildFullDiff}} branch, where it should be setting
{{repairs[repairs.length - 1]}}. This is causing the new
{{movingTokenReadRepairTest}} failure, but locally at least, it doesn't affect
the other tests in that fixture so I'm not sure what's going on in CircleCI.
The patch also needs a rebase, but it doesn't look too onerous.
Nits:
* {{BlockingReadRepair}} #54: extraneous comment
* {{ReadRepairTest}} #199: formatting
* {{RowIteratorMergeListener}}: unused import
* I find the style of having the conditions and statements for an {{if/else}}
on the same line ({{RowIteratorMergeListener}} #373) makes them harder to
parse. This is clearly rather subjective though and maybe just because it's not
followed more universally in the project - feel free to ignore.
> BlockingReadRepair does not maintain monotonicity during range movements
>
>
> Key: CASSANDRA-14740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14740
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Urgent
> Labels: correctness
> Fix For: 4.0, 4.0-beta
>
>
> The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each
> of the queried nodes are written to, but pending nodes are not considered.
> If there is a pending range movement, one of these writes may be ‘lost’ when
> the range movement completes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-14740: Reviewers: Sam Tunnicliffe, Sam Tunnicliffe (was: Sam Tunnicliffe) Sam Tunnicliffe, Sam Tunnicliffe (was: Sam Tunnicliffe) Status: Review In Progress (was: Patch Available) > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Urgent > Labels: correctness > Fix For: 4.0, 4.0-beta > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-14740: Reviewers: Sam Tunnicliffe > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Urgent > Labels: correctness > Fix For: 4.0, 4.0-beta > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-14740: --- Fix Version/s: 4.0 > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Urgent > Labels: correctness > Fix For: 4.0, 4.0-beta > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-14740: --- Fix Version/s: (was: 4.0) 4.0-beta > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Urgent > Labels: correctness > Fix For: 4.0-beta > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-14740: --- Status: Patch Available (was: Open) > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Urgent > Labels: correctness > Fix For: 4.0 > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14740: - Status: Open (was: Patch Available) > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict >Assignee: Benedict >Priority: Urgent > Labels: correctness > Fix For: 4.0 > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[
https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict updated CASSANDRA-14740:
-
Status: Patch Available (was: Open)
The basic approach is quite simple: when we repair, we build a {{WritePlan}},
but we only select those nodes we need to meet the consistency level of the
operation we are performing, and we only consider live nodes. We prefer those
nodes we have read from. If they are all present, and they are sufficient to
meet consistency, we behave as before. In any other scenario, we build a
partition representing all differences we have seen, and propagate this to any
node that wasn't one of the original targets.
There are some minor inefficiencies, such as not handling the case where the
ownership of only one node has changed, and there is no node pending, in which
case we _might_ be able to only propagate the difference found on reconciling
the presumably replaced node (though if unsafe bootstrap occurred even this
would not be acceptable, but this might be an acceptable consistency failure
given the semantic guarantees of this).
We also don't bother to avoid merging the complete diff row with any other
pending repairs if we have to perform an additional write. It's assume that
these scenario are rare, and not worth the significant extra complexity.
Unfortunately fixing unit tests was painful and not super beautiful. This is
because read-repair now consults the ring to decide who a repair should be
routed to, instead of assuming it is sufficient to write to those we read from.
The tests as written assume the ring can be empty, and also that replication
factor isn't relevant, so to avoid completely rewriting the tests, I have done
some ugly things.
> BlockingReadRepair does not maintain monotonicity during range movements
>
>
> Key: CASSANDRA-14740
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14740
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Coordination
>Reporter: Benedict
>Assignee: Benedict
>Priority: Urgent
> Labels: correctness
> Fix For: 4.0
>
>
> The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each
> of the queried nodes are written to, but pending nodes are not considered.
> If there is a pending range movement, one of these writes may be ‘lost’ when
> the range movement completes.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14740: - Status: Open (was: Patch Available) > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict >Assignee: Benedict >Priority: Urgent > Labels: correctness > Fix For: 4.0 > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14740: - Test and Documentation Plan: dtest included Status: Patch Available (was: In Progress) Patch available [here|https://github.com/belliottsmith/cassandra/tree/14740], [tests|https://circleci.com/workflow-run/9b4bc0a4-ba24-4a29-a6fd-4bf78f125432] > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict >Assignee: Benedict >Priority: Urgent > Labels: correctness > Fix For: 4.0 > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14740: - Complexity: Normal Discovered By: Code Inspection Bug Category: Parent values: Correctness(12982)Level 1 values: Response Corruption / Loss(12987) > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Coordination >Reporter: Benedict >Priority: Urgent > Labels: correctness > Fix For: 4.0 > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14740: - Component/s: (was: Core) Coordination > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Coordination >Reporter: Benedict >Priority: Critical > Labels: correctness > Fix For: 4.0 > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14740) BlockingReadRepair does not maintain monotonicity during range movements
[ https://issues.apache.org/jira/browse/CASSANDRA-14740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-14740: - Fix Version/s: (was: 4.0.x) 4.0 > BlockingReadRepair does not maintain monotonicity during range movements > > > Key: CASSANDRA-14740 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14740 > Project: Cassandra > Issue Type: Bug > Components: Coordination >Reporter: Benedict >Priority: Critical > Labels: correctness > Fix For: 4.0 > > > The BlockingReadRepair code introduced by CASSANDRA-10726 requires that each > of the queried nodes are written to, but pending nodes are not considered. > If there is a pending range movement, one of these writes may be ‘lost’ when > the range movement completes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
