[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452349#comment-15452349
]
Jim Witschey commented on CASSANDRA-12092:
--
Merged the PR, thanks [~Stefania] -- are we good to close this ticket?
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: jenkins-stef1927-dtest-multiplex-23_logs.000.tar.gz,
> node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432191#comment-15432191
]
Stefania commented on CASSANDRA-12092:
--
I traced the remaining rare failures down to a speculative retry. It was not
the async repair digest mismatch that caused problems, rather in some rare
cases a speculative retry (which is sent after a sleep period) arrives before
the local read. I've disabled speculative retry and I got two 5x200 builds
without any failures, running a modified test that repeated QUORUM/ONE 10
times, see
[here|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/26/]
and
[here|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/28/].
[Pull request|https://github.com/riptano/cassandra-dtest/pull/1269].
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: jenkins-stef1927-dtest-multiplex-23_logs.000.tar.gz,
> node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427952#comment-15427952
]
Stefania commented on CASSANDRA-12092:
--
I was able to reproduce one
[failure|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/23/]
with debug logs finally, see
[^jenkins-stef1927-dtest-multiplex-23_logs.000.tar.gz] attached.
It seems that when the test fails, a digest arrives before the local read
completes at CL ONE, and this triggers async repair due to the digest mismatch
if the other node has a different counter value, which is expected. However, I
haven't understood why none of the two nodes returns the up-to-date counter
from the local data response, since at least one of them should have applied
the mutation that it received from the leader - given that we write at
CL.QUORUM. So I'm still investigating.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: jenkins-stef1927-dtest-multiplex-23_logs.000.tar.gz,
> node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427538#comment-15427538
]
Stefania commented on CASSANDRA-12092:
--
I've reproduced a
[failure|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/20/testReport/junit/node_0_iter_169.consistency_test/TestAccuracy/test_simple_strategy_counters/]
with the extra log message, confirming that the test is definitely reading
from the host it contacts, so there could be a race in Cassandra.
I'm attempting another run, with more logs
[here|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/21/].
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425952#comment-15425952
]
Stefania commented on CASSANDRA-12092:
--
Unfortunately there was still one
[failure|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/17/testReport/junit/node_3_iter_169.consistency_test/TestAccuracy/test_simple_strategy_counters/]
out of 1000 attempts.
I've reduced the number of debug log messages to just one, which is essential
to confirm that we are reading from the host we contacted. I will perform one
more multiplexed run hoping that with one single debug log message we can
reproduce the failure. This would tell us if the problem is with the test or in
Cassandra. If we still cannot reproduce it, I propose to relax the test for
2.1. I've already added one unit test to ensure that we can correctly read
counter values that are updated by a different thread.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424416#comment-15424416
]
Stefania commented on CASSANDRA-12092:
--
Fixed one more problem with the test, relaunched a multiplexed job
[here|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/17/].
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420760#comment-15420760
]
Stefania commented on CASSANDRA-12092:
--
Multiplexing the test with additional log messages ensures that it never fails
([build
#9|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/9/]
and [build
#12|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/12/]).
However without log messages it fails at about a rate of 2 / 1000 ([build
#13|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/13/]),
not the easiest thing to debug.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418473#comment-15418473
]
Stefania commented on CASSANDRA-12092:
--
There was 1
[failure|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/7/testReport/junit/node_2_iter_091.consistency_test/TestAccuracy/test_simple_strategy_counters/]
(out of 600 repetitions) for the run above. I really have no idea why, I've
relaunched with additional log messages:
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/9/.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418339#comment-15418339
]
Stefania commented on CASSANDRA-12092:
--
Multiplexed run above completed without errors, I've launched a larger one:
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/7/.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416938#comment-15416938
]
Stefania commented on CASSANDRA-12092:
--
It seems dynamic snitch is still active, this would cause us to potentially
read from a host that is different from the coordinator. I've launched another
multiplexed run with dynamic_snitch set to false and read repair disabled:
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-dtest-multiplex/3/
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Stefania
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413053#comment-15413053
]
Stefania commented on CASSANDRA-12092:
--
Thank you for running the tests. There were no failures on 2.2, and 2 unrelated
failures on trunk (it seems we still have schema agreement problems).
For 2.1 there were 9 failures, all on key 200 except for 1, which was on key
598. Key 200 is the first key tested for the QUORUM/ONE/- range (200 to 249)
whilst key 598 refers to the TWO/ONE/- range (550 to 599).
So it seems this problem affects 2.1 only, I'm going to take a look.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Sean McCarthy
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412212#comment-15412212
]
Sean McCarthy commented on CASSANDRA-12092:
---
Running a multiplex (1000 iterations) to test if
{quote}
it is really always the first key of QUORUM/ONE/-
{quote}
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/222/
Also running a 2.2 and trunk multiplex tests.
Trunk:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/223/
2.2
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/224/
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Sean McCarthy
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401430#comment-15401430
]
Stefania commented on CASSANDRA-12092:
--
Nothing stands out in the test; I would suggest multiplexing a couple more
times to see if:
* it is only a 2.1 problem
* it is really always the first key of QUORUM/ONE/-
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400145#comment-15400145
]
Russ Hatch commented on CASSANDRA-12092:
[~Stefania] any idea what could be causing this test to fail (intermittently)
but on the same key when it does?
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1547#comment-1547
]
Russ Hatch commented on CASSANDRA-12092:
Interestingly both failures shown on this ticket happened at 'key 200', which
looks to be writing at quorum, reading back at one, with serial unset. For a
random-looking failure, the same key of 200 is a suspicious value.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1533#comment-1533
]
Russ Hatch commented on CASSANDRA-12092:
failure from recent multiplex:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/195/testReport/node_1_iter_059.consistency_test/TestAccuracy/test_simple_strategy_counters/
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1532#comment-1532
]
Russ Hatch commented on CASSANDRA-12092:
1 failure in 200 iterations. Either the test is bad or there's a bug here.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399825#comment-15399825
]
Russ Hatch commented on CASSANDRA-12092:
Since this is one isolated flap in recent history, testing with multiplex (200
iterations) here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/195/
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380068#comment-15380068
]
Jim Witschey commented on CASSANDRA-12092:
--
It's important to note that this failed with the {{WRITE/READ/SERIAL}}
consistency set to {{THREE/ONE/-}}
{code}
Error Message
Failed to read value from sufficient number of nodes, required 2 nodes to have
a counter value of 1 at key 200, instead got these values: [0, 0, 1]
>> begin captured logging <<
dtest: DEBUG: cluster ccm directory: /tmp/dtest-shYOzK
dtest: DEBUG: Custom init_config not found. Setting defaults.
dtest: DEBUG: Done setting configuration options:
{ 'initial_token': None,
'num_tokens': '32',
'phi_convict_threshold': 5,
'range_request_timeout_in_ms': 1,
'read_request_timeout_in_ms': 1,
'request_timeout_in_ms': 1,
'truncate_request_timeout_in_ms': 1,
'write_request_timeout_in_ms': 1}
dtest: DEBUG: Testing single dc, counters
dtest: DEBUG: Changing snitch for single dc case
dtest: DEBUG: Testing accuracy with WRITE/READ/SERIAL consistency set to
ALL/ALL/- (keys : 0 to 49)
dtest: DEBUG: Testing accuracy with WRITE/READ/SERIAL consistency set to
QUORUM/QUORUM/- (keys : 50 to 99)
dtest: DEBUG: Testing accuracy with WRITE/READ/SERIAL consistency set to
ALL/ONE/- (keys : 100 to 149)
dtest: DEBUG: Testing accuracy with WRITE/READ/SERIAL consistency set to
ONE/ALL/- (keys : 150 to 199)
dtest: DEBUG: Testing accuracy with WRITE/READ/SERIAL consistency set to
QUORUM/ONE/- (keys : 200 to 249)
dtest: DEBUG: Testing accuracy with WRITE/READ/SERIAL consistency set to
ONE/QUORUM/- (keys : 250 to 299)
dtest: DEBUG: Waiting for workers to complete
dtest: DEBUG: Testing accuracy with WRITE/READ/SERIAL consistency set to
TWO/TWO/- (keys : 300 to 349)
dtest: DEBUG: Testing accuracy with WRITE/READ/SERIAL consistency set to
ONE/THREE/- (keys : 350 to 399)
dtest: DEBUG: Testing accuracy with WRITE/READ/SERIAL consistency set to
THREE/ONE/- (keys : 400 to 449)
dtest: DEBUG: Failed with exception : Failed
to read value from sufficient number of nodes, required 2 nodes to have a
counter value of 1 at key 200, instead got these values: [0, 0, 1]
- >> end captured logging << -
{code}
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
