[
https://issues.apache.org/jira/browse/CASSANDRA-15566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17053776#comment-17053776
]
David Capwell commented on CASSANDRA-15566:
-------------------------------------------
Thanks!
I have and have not started! There are several things going on so this task
really should be split into smaller tasks (kinda like this is an epic!).
I am currently trying to finish CASSANDRA-15564 which fixes some of the issues
but doesn't fix anything that fails because of networking; that JIRA also adds
a @Ignored test which fails 100% of the time (which is what triggered this
JIRA).
I have a POC patch for CASSANDRA-15399 which exposes the state across the
coordinator and participants
I am working on longevity testing (without explicitly causing failures) with a
focus on repair; this was to help boost my confidence on any structural changes
made.
Thats the current state of this all!
There are a few things already found problematic, so the next steps are mostly
1) testing
2) come up with a proposal for how to fix
I have not fleshed out #2 so if you want to take a stab at fleshing out how to
solve that would be great! Without that we can't really split this JIRA.
> Repair coordinator can hang under some cases
> --------------------------------------------
>
> Key: CASSANDRA-15566
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15566
> Project: Cassandra
> Issue Type: Improvement
> Components: Consistency/Repair
> Reporter: David Capwell
> Assignee: David Capwell
> Priority: Normal
> Fix For: 4.0-beta
>
>
> Repair coordination makes a few assumptions about message delivery which
> cause it to hang forever when those assumptions don’t hold true: fire and
> forget will not get rejected (participate has an issue and rejects the
> message), and a very delayed message will one day be seen (messaging can be
> dropped under load or when failure detector thinks a node is bad but is just
> GCing).
> Given this and the desire to have better observability with repair (see
> CASSANDRA-15399), coordination should be changed into a request/response
> pattern (with retries) and polling (validation status and MerkleTree
> sending). This would allow the coordinator to detect changes in state (it
> was known participate was working on validation, but it no longer knows about
> the validation task), and to be able to recover from ephemeral issues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
