[
https://issues.apache.org/jira/browse/CASSANDRA-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922536#comment-17922536
]
Dmitry Konstantinov commented on CASSANDRA-20194:
-------------------------------------------------
So, about the root cause.. The analysis below is for
TestReadRepair.test_failed_read_repair specifically, I have not checked yet
other failing methods, but with a high probability they suffer from the same
issue.
1) Initially we are doing usual quorum insert which is replicated to all 3 nodes
!step1.png|width=400!
2) Then we disable usual write and read repair write handling by 2nd and 3rd
replicas.
!step2.png|width=400!
3) Then we do another quorum write, which is saved on 1st replica but not added
to 2nd and 3rd replica due to the previous action, so it fails but we still
have the new data on 1st replica.
!step3.png|width=400!
4) A step which we analyse a bit later:
node2.byteman_submit([mk_bman_path('read_repair/sorted_live_endpoints.btm')])
5) We do a quorum partition read from 2nd node.
!step5.png|width=400!
To handle it the node as a coordinator needs to select 2 replicas to read data
from and we have 3 combinations:
- (1, 2) => data are mismatched, so a read repair is triggered but it cannot
succeed due to the disabling which we did on step 2
- (1, 3) => same as the previous case
- (2, 3) - data on the replicas are matched, so we return the result
successfully without read repair triggering
!variants.png|width=400!
We try to test read repair, so we want to avoid (2, 3) replicas selection for
the last read. And this is the role of step 4): we inject specific order into
SimpleSnitch to force coordinator to use specific replicas for the read, for
the same purpose we disable dynamic_snitch option in the test.
Our read is not failing with timeout as we expect in the test because we are
facing (2, 3) case and it means our injection on step 4) does not work anymore.
Why: because of the changes introduced in 5.1 for TCM: CASSANDRA-19488
([https://github.com/apache/cassandra/commit/48dcf5e0921b8a5b7a73fa15ebfb2341d48c857b]).
Here we moved away from the old snitch logic to a new concept/interface
NodeProximity which is not covered by the injection on step 4).
In the same commit there were changes in cassandra.yaml and
cassandra_latest.yaml. In case of cassandra.yaml we still use the injected
SimpleSnitch by decorating it with SnitchAdapter to adopt it to the new
interfaces. In case of cassandra_latest.yaml we configure node_proximity
explicitly.
So, this is why we have the test passing for cassandra.yaml and failing for
cassandra_latest.yaml.
A possible way to fix it is to add a 5.1 specific rule which will apply the
same kind of injection to NetworkTopologyProximity.sortedByProximity method as
we had for SimpleSnitch, like:
{code:java}
RULE sorted live endpoints
CLASS org.apache.cassandra.locator.BaseProximity
METHOD sortedByProximity
AT ENTRY
IF true
DO
return $unsortedAddress.sorted(java.util.Comparator.naturalOrder());
ENDRULE
{code}
> Test failure: read_repair_test.py::TestSpeculativeReadRepair
> ------------------------------------------------------------
>
> Key: CASSANDRA-20194
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20194
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Test/dtest/python
> Reporter: Brandon Williams
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Fix For: 5.x
>
> Attachments: TestSpeculativeReadRepair_pytest.out,
> TestSpeculativeReadRepair_pytest_latest.out, step1.png, step2.png, step3.png,
> step5.png, variants.png
>
>
> Seen at
> https://app.circleci.com/pipelines/github/driftx/cassandra/1795/workflows/3f53c905-7e98-47bb-a73d-dc3d27d910c5/jobs/108602/tests
> {noformat}
> test_failed_read_repair failed and was not selected for rerun.
> <class 'Failed'>
> DID NOT RAISE <class 'cassandra.ReadTimeout'>
> [<TracebackEntry
> /home/cassandra/cassandra-dtest/read_repair_test.py:555>]
> {noformat}
> {noformat}
> test_normal_read_repair failed and was not selected for rerun.
> <class 'AssertionError'>
> assert [[1, 0, 1]] == [[1, 0, 1], [1, 1, 2]]
> Right contains one more item: [1, 1, 2]
> Full diff:
> - [[1, 0, 1], [1, 1, 2]]
> + [[1, 0, 1]]
> [<TracebackEntry
> /home/cassandra/cassandra-dtest/read_repair_test.py:592>]
> {noformat}
> {noformat}
> test_quorum_requirement failed and was not selected for rerun.
> <class 'AssertionError'>
> assert 0 == 1
> +0
> -1
> [<TracebackEntry
> /home/cassandra/cassandra-dtest/read_repair_test.py:724>]
> {noformat}
> {noformat}
> test_speculative_data_request failed and was not selected for rerun.
> <class 'AssertionError'>
> assert 0 == 1
> +0
> -1
> [<TracebackEntry
> /home/cassandra/cassandra-dtest/read_repair_test.py:639>]
> {noformat}
> {noformat}
> test_speculative_write failed and was not selected for rerun.
> <class 'AssertionError'>
> assert 0 == 1
> +0
> -1
> [<TracebackEntry
> /home/cassandra/cassandra-dtest/read_repair_test.py:680>]
> {noformat}
> This was against the 'latest' yaml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
