[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224341#comment-15224341 ] Jim Witschey commented on CASSANDRA-11225: -- Merged, thanks. Filed CASSANDRA-11488 as followup. > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15223598#comment-15223598 ] Stefania commented on CASSANDRA-11225: -- There were no failures in the latest run, so I've created a [pull request|https://github.com/riptano/cassandra-dtest/pull/910]. I wonder if it should be considered a bug that the coordinator may not read locally with simple snitch, even if it is a replica. > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221817#comment-15221817 ] Russ Hatch commented on CASSANDRA-11225: That's really interesting, thanks [~Stefania]. I've kicked off a new run with your latest dtest code here: https://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/59/ > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221059#comment-15221059
]
Stefania commented on CASSANDRA-11225:
--
It seems your observation is correct. StorageProxy has a method,
{{getLiveSortedEndpoints()}}, which is called by the read executor to determine
the target replicas to contact. This method, in turn relies on
{{IEndpointSnitch.sortByProximity(localhost, liveendpoints)}} and
{{SimpleSnitch}} does not implement this method. So we are not guaranteed the
coordinator is the first chosen replica in this case. {{SimpleSnitch}} is the
default snitch in cassandra.yaml and ccm only changes the snitch to
{{PropertyFileSnitch}} for multi data center tests.
Another interesting thing to point out, when writing counter mutations, is that
in storage proxy {{mutateCounter()}}, the leader is not necessarily the local
host because {{findSuitableEndpoint()}} picks a random local endpoint. So we
are not guaranteed the coordinator is the mutation leader even it is a replica.
This shouldn't matter though, as long as we write to the requested number of
replicas, which is what we are trying to test really.
As for reading though, if we cannot read from the replica we have contacted,
then this test needs to be changed to only validate the strong consistency
cases, that is R + W > RF. I've amended the patch so that we use
{{PropertyFileSnitch}} for the single data center tests as well. If my
reasoning is correct, the coordinator should always choose itself as the
preferred replica, and the test should pass all the times. If this is not the
case however, then we can change the test to only validate strong consistency,
since we are not guaranteed to be able to read from a specific replica then I
am not sure how to test that we have written to a specific number of replicas.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-11225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11225
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Russ Hatch
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_novnode_dtest #209
> error: "AssertionError: Failed to read value from sufficient number of nodes,
> required 2 but got 1 - [574, 2]"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219981#comment-15219981 ] Russ Hatch commented on CASSANDRA-11225: [~Stefania] Sounds like maybe those two failures could both be eventual consistency test problems? I learned recently while investigating another test, that even with an exclusive connection, with the coordinator in the replica set, it still may go out to another node for data (though rare). Maybe we're seeing a manifestation of that corner case? > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219311#comment-15219311 ] Stefania commented on CASSANDRA-11225: -- Two more tests failed, both where when reading at CL=ONE. In fact, I checked all failures so far, they all happen when reading at CL ONE. The two tests that failed: * The counter was incremented from 0 to 1 by contacting the first replica with CL=TWO and the following values were returned when reading at CL=1 from all replicas: \[0,0,1\], the test fails because it was expecting at least two replicas to have value 1. * The counter was incremented from 1 to 2 by contacting the second replica with CL=TWO and the following values were returned when reading at CL=1 from all replicas: \[1,1,2\], the test fails because it was expecting at least two replicas to have value 2. I don't think the counter update failed because we should have had an exception in that case. I checked the test code and I cannot find anything obviously wrong, we are definitely contacting replicas individually via exclusive connections and we have a 3 node cluster with RF=3 so all nodes should be replicas. I cannot rule out an issue with reading local counter values at this stage. We could try changing parameters such the counter cache or removing the parallelism on partition keys to try and gather more clues. If we could reproduce this problem with a new test that increments counters and reads them back from the same replica multiple times and in parallel on different partition keys this might help in determining if the bug is really in C* or in the test. > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218413#comment-15218413 ] Russ Hatch commented on CASSANDRA-11225: no problem [~stefania_alborghetti] -- kicked off a new run here with your latest dtest code: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/52/ > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217213#comment-15217213
]
Stefania commented on CASSANDRA-11225:
--
There was a problem with the test patch of yesterday, {{query_value()}} now
returns a value, not a boolean. I'm really sorry but we need to repeat the
tests with the fixed patch.
If we still have failures, either our test is too demanding for counters, in
which case we should relax it, perhaps by pausing a few milliseconds, or we may
have a bug hiding somewhere.
Here's what the test does: it reads at CL.ALL to find the current counter
value, it increments the counter at various consistency levels, it checks that
we can read back the incremented counter value from at least the number of
replicas we wrote to, and then it starts all over again but it contacts another
host.
I would expect that a read at CL.ALL would trigger a digest mismatch and a
subsequent repair, so that all nodes start from the same counter value when we
apply the next increment. [~iamaleksey] is this statement correct?
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-11225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11225
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Russ Hatch
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_novnode_dtest #209
> error: "AssertionError: Failed to read value from sufficient number of nodes,
> required 2 but got 1 - [574, 2]"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216667#comment-15216667 ] Russ Hatch commented on CASSANDRA-11225: looks like there was still 1/300 failures, but maybe that's an acceptable noise level. guess we just want to be sure there's not a legitimate bug hiding here. http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/50/testReport/node_0.consistency_test/TestAccuracy/test_simple_strategy_counters/ > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216391#comment-15216391 ] Russ Hatch commented on CASSANDRA-11225: doing another bulk run here: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/50/ > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215231#comment-15215231 ] Stefania commented on CASSANDRA-11225: -- The only thing I can think of is that the nodes are probably starting from different counter values and this causes problems. I've added a read at consistency level ALL before the next iteration, can you see if it helps by running it again 300 times? The patch is here: https://github.com/stef1927/cassandra-dtest/commits/11225 I've also improved the output messages a bit. > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15212032#comment-15212032 ] Russ Hatch commented on CASSANDRA-11225: [~stefania_alborghetti] Any idea what could be going on with this test? > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Russ Hatch > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[ https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208875#comment-15208875 ] Philip Thompson commented on CASSANDRA-11225: - I ran this 300 times: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/36/ > dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters > > > Key: CASSANDRA-11225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11225 > Project: Cassandra > Issue Type: Test >Reporter: Russ Hatch >Assignee: Philip Thompson > Labels: dtest > > example failure: > http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters > Failed on CassCI build cassandra-2.1_novnode_dtest #209 > error: "AssertionError: Failed to read value from sufficient number of nodes, > required 2 but got 1 - [574, 2]" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-11225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200294#comment-15200294
]
Jim Witschey commented on CASSANDRA-11225:
--
Seeing the same failure (while testing at {{LOCAL_QUORUM}}) on the vanilla 3.0
job:
http://cassci.datastax.com/job/cassandra-3.0_dtest/614/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters/
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-11225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11225
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_novnode_dtest #209
> error: "AssertionError: Failed to read value from sufficient number of nodes,
> required 2 but got 1 - [574, 2]"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
