[jira] [Comment Edited] (CASSANDRA-11414) dtest failure in bootstrap_test.TestBootstrap.resumable_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364839#comment-15364839
]
Yuki Morishita edited comment on CASSANDRA-11414 at 7/9/16 12:25 AM:
-
Committed as {{00e7ecf1394f8704e2f13369f7950e129459ce2c}}.
was (Author: yukim):
Committed as {00e7ecf1394f8704e2f13369f7950e129459ce2c}.
> dtest failure in bootstrap_test.TestBootstrap.resumable_bootstrap_test
> --
>
> Key: CASSANDRA-11414
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11414
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Philip Thompson
>Assignee: Paulo Motta
> Labels: dtest
> Fix For: 2.2.8, 3.0.9, 3.9
>
>
> Stress is failing to read back all data. We can see this output from the
> stress read
> {code}
> java.io.IOException: Operation x0 on key(s) [314c384f304f4c325030]: Data
> returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:138)
> at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> java.io.IOException: Operation x0 on key(s) [33383438363931353131]: Data
> returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:138)
> at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> FAILURE
> {code}
> Started happening with build 1075. Does not appear flaky on CI.
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1076/testReport/bootstrap_test/TestBootstrap/resumable_bootstrap_test
> Failed on CassCI build trunk_dtest #1076
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11414) dtest failure in bootstrap_test.TestBootstrap.resumable_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15248094#comment-15248094
]
Philip Thompson edited comment on CASSANDRA-11414 at 4/19/16 4:25 PM:
--
So it appears there can sometimes be issues on the receiving node when the
stream dies, causing that node to fail to start listening for clients.
A successful run:
https://gist.github.com/ptnapoleon/b999e42e3e0a39de233d875ac00ee79b
A failed run:
https://gist.github.com/ptnapoleon/00e1e5739ad867b9dfb10fbf16da9022
Here is a collection of several hundred runs of this test, to give an idea of
how often it flakes, and how.
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/68/testReport/
was (Author: philipthompson):
So it appears there can sometimes be issues on the receiving node when the
stream dies, causing that node to fail to start listening for clients.
A successful run:
https://gist.github.com/ptnapoleon/b999e42e3e0a39de233d875ac00ee79b
A failed run:
https://gist.github.com/ptnapoleon/00e1e5739ad867b9dfb10fbf16da9022
> dtest failure in bootstrap_test.TestBootstrap.resumable_bootstrap_test
> --
>
> Key: CASSANDRA-11414
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11414
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Philip Thompson
>Assignee: Paulo Motta
> Labels: dtest
> Fix For: 3.x
>
>
> Stress is failing to read back all data. We can see this output from the
> stress read
> {code}
> java.io.IOException: Operation x0 on key(s) [314c384f304f4c325030]: Data
> returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:138)
> at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> java.io.IOException: Operation x0 on key(s) [33383438363931353131]: Data
> returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:138)
> at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> FAILURE
> {code}
> Started happening with build 1075. Does not appear flaky on CI.
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1076/testReport/bootstrap_test/TestBootstrap/resumable_bootstrap_test
> Failed on CassCI build trunk_dtest #1076
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11414) dtest failure in bootstrap_test.TestBootstrap.resumable_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218127#comment-15218127
]
Jim Witschey edited comment on CASSANDRA-11414 at 3/30/16 3:21 PM:
---
I haven't seen the stress failure since that PR was merged, so that's good.
However, I'm still seeing some failures where a node times out waiting for CQL:
http://cassci.datastax.com/job/trunk_dtest/1081/testReport/junit/bootstrap_test/TestBootstrap/resumable_bootstrap_test/
was (Author: mambocab):
I haven't seen the stress failure since that PR was merged, so that's good.
However, I'm still seeing some failures where a node fails to start.
> dtest failure in bootstrap_test.TestBootstrap.resumable_bootstrap_test
> --
>
> Key: CASSANDRA-11414
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11414
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Philip Thompson
>Assignee: DS Test Eng
> Labels: dtest
>
> Stress is failing to read back all data. We can see this output from the
> stress read
> {code}
> java.io.IOException: Operation x0 on key(s) [314c384f304f4c325030]: Data
> returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:138)
> at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> java.io.IOException: Operation x0 on key(s) [33383438363931353131]: Data
> returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:138)
> at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> FAILURE
> {code}
> Started happening with build 1075. Does not appear flaky on CI.
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1076/testReport/bootstrap_test/TestBootstrap/resumable_bootstrap_test
> Failed on CassCI build trunk_dtest #1076
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11414) dtest failure in bootstrap_test.TestBootstrap.resumable_bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208826#comment-15208826
]
Paulo Motta edited comment on CASSANDRA-11414 at 3/23/16 5:35 PM:
--
I'd say changing the stress write CL to TWO would fix this (we must keep the
read CL at ONE though)
was (Author: pauloricardomg):
I'd say changing the stress read and/or write CL to QUORUM would fix this.
> dtest failure in bootstrap_test.TestBootstrap.resumable_bootstrap_test
> --
>
> Key: CASSANDRA-11414
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11414
> Project: Cassandra
> Issue Type: Test
> Components: Testing
>Reporter: Philip Thompson
>Assignee: DS Test Eng
> Labels: dtest
>
> Stress is failing to read back all data. We can see this output from the
> stress read
> {code}
> java.io.IOException: Operation x0 on key(s) [314c384f304f4c325030]: Data
> returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:138)
> at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> java.io.IOException: Operation x0 on key(s) [33383438363931353131]: Data
> returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:138)
> at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:327)
> FAILURE
> {code}
> Started happening with build 1075. Does not appear flaky on CI.
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1076/testReport/bootstrap_test/TestBootstrap/resumable_bootstrap_test
> Failed on CassCI build trunk_dtest #1076
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
