[jira] [Updated] (CASSANDRA-12050) per-patch smoke suites as an early/fast testing tier
[
https://issues.apache.org/jira/browse/CASSANDRA-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12050:
---
Priority: Minor (was: Major)
> per-patch smoke suites as an early/fast testing tier
>
>
> Key: CASSANDRA-12050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12050
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Russ Hatch
>Priority: Minor
>
> Coverage data offers a unique opportunity to build metadata about tests and
> related Cassandra code.
> Using the jacoco coverage api we should be able to build a simple index of
> tests (dtest, unit) to the Cassandra code they touch (and the reverse).
> When a new patch is introduced, we do a lookup in that index, based on java
> source files touched (and possibly lines within those files), and use that to
> infer most relevant dtests or unit tests. Patch authors can then run that
> small test subset as a first testing pass.
> In this way we can build small, focused, test suites that are custom to each
> patch. Once this small custom smoke test appears successful things would of
> course need to be vetted across a more complete test run on CI.
> I think the best interface would simply be ant targets. One target would be
> used to build/refresh the test:source code index (run occasionally and saved
> somewhere; index building would be time consuming since it will require full
> job runs). A second target looks at files touched and does the index lookups,
> then outputs a list of tests to run.
> The dev user experience might looking something like this:
> {noformat}
> ant get-custom-smoke -Dcoverage_index=./trunk_coverage_index_SHA_foo.idx
> Generating test lists
> Please run the following two scripts to vet your changes:
> ./custom_smoke_junit.sh
> ./custom_smoke_dtest.sh
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12050) per-patch smoke suites as an early/fast testing tier
[
https://issues.apache.org/jira/browse/CASSANDRA-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12050:
---
Assignee: (was: Russ Hatch)
> per-patch smoke suites as an early/fast testing tier
>
>
> Key: CASSANDRA-12050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12050
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Russ Hatch
>
> Coverage data offers a unique opportunity to build metadata about tests and
> related Cassandra code.
> Using the jacoco coverage api we should be able to build a simple index of
> tests (dtest, unit) to the Cassandra code they touch (and the reverse).
> When a new patch is introduced, we do a lookup in that index, based on java
> source files touched (and possibly lines within those files), and use that to
> infer most relevant dtests or unit tests. Patch authors can then run that
> small test subset as a first testing pass.
> In this way we can build small, focused, test suites that are custom to each
> patch. Once this small custom smoke test appears successful things would of
> course need to be vetted across a more complete test run on CI.
> I think the best interface would simply be ant targets. One target would be
> used to build/refresh the test:source code index (run occasionally and saved
> somewhere; index building would be time consuming since it will require full
> job runs). A second target looks at files touched and does the index lookups,
> then outputs a list of tests to run.
> The dev user experience might looking something like this:
> {noformat}
> ant get-custom-smoke -Dcoverage_index=./trunk_coverage_index_SHA_foo.idx
> Generating test lists
> Please run the following two scripts to vet your changes:
> ./custom_smoke_junit.sh
> ./custom_smoke_dtest.sh
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12050) per-patch smoke suites as an early/fast testing tier
[
https://issues.apache.org/jira/browse/CASSANDRA-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12050:
---
Issue Type: Test (was: Improvement)
> per-patch smoke suites as an early/fast testing tier
>
>
> Key: CASSANDRA-12050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12050
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Priority: Minor
>
> Coverage data offers a unique opportunity to build metadata about tests and
> related Cassandra code.
> Using the jacoco coverage api we should be able to build a simple index of
> tests (dtest, unit) to the Cassandra code they touch (and the reverse).
> When a new patch is introduced, we do a lookup in that index, based on java
> source files touched (and possibly lines within those files), and use that to
> infer most relevant dtests or unit tests. Patch authors can then run that
> small test subset as a first testing pass.
> In this way we can build small, focused, test suites that are custom to each
> patch. Once this small custom smoke test appears successful things would of
> course need to be vetted across a more complete test run on CI.
> I think the best interface would simply be ant targets. One target would be
> used to build/refresh the test:source code index (run occasionally and saved
> somewhere; index building would be time consuming since it will require full
> job runs). A second target looks at files touched and does the index lookups,
> then outputs a list of tests to run.
> The dev user experience might looking something like this:
> {noformat}
> ant get-custom-smoke -Dcoverage_index=./trunk_coverage_index_SHA_foo.idx
> Generating test lists
> Please run the following two scripts to vet your changes:
> ./custom_smoke_junit.sh
> ./custom_smoke_dtest.sh
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12364) dtest failure in upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
[
https://issues.apache.org/jira/browse/CASSANDRA-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488071#comment-15488071
]
Russ Hatch commented on CASSANDRA-12364:
dtest pr merged. I'll create a separate issue somewhere to track "Make sure we
never throw RejectedExecutionException"
> dtest failure in
> upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
>
>
> Key: CASSANDRA-12364
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12364
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_cell_TTL_expiry_during_paging
> {code}
> Error Message
> Unexpected error in log, see stdout
> {code}
> {code}
> Standard Output
> http://git-wip-us.apache.org/repos/asf/cassandra.git
> git:5051c0f6eb3f984600600c9577d6b5ece9038c74
> Unexpected error in node1 log, error:
> ERROR [InternalResponseStage:4] 2016-07-28 03:23:02,097
> CassandraDaemon.java:217 - Exception in thread
> Thread[InternalResponseStage:4,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> ~[na:1.8.0_51]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.waitForFlushes(ColumnFamilyStore.java:930)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:892)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/2113926365.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> {code}
> Related failures:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/11/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_ttl_deletions/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/static_columns_with_distinct_test/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/refuse_in_with_indexes_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12364) dtest failure in upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
[
https://issues.apache.org/jira/browse/CASSANDRA-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12364:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> dtest failure in
> upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
>
>
> Key: CASSANDRA-12364
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12364
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_cell_TTL_expiry_during_paging
> {code}
> Error Message
> Unexpected error in log, see stdout
> {code}
> {code}
> Standard Output
> http://git-wip-us.apache.org/repos/asf/cassandra.git
> git:5051c0f6eb3f984600600c9577d6b5ece9038c74
> Unexpected error in node1 log, error:
> ERROR [InternalResponseStage:4] 2016-07-28 03:23:02,097
> CassandraDaemon.java:217 - Exception in thread
> Thread[InternalResponseStage:4,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> ~[na:1.8.0_51]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.waitForFlushes(ColumnFamilyStore.java:930)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:892)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/2113926365.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> {code}
> Related failures:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/11/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_ttl_deletions/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/static_columns_with_distinct_test/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/refuse_in_with_indexes_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9318) Bound the number of in-flight requests at the coordinator
[ https://issues.apache.org/jira/browse/CASSANDRA-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487712#comment-15487712 ] Russ Hatch commented on CASSANDRA-9318: --- Nothing extra to add here from me. > Bound the number of in-flight requests at the coordinator > - > > Key: CASSANDRA-9318 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9318 > Project: Cassandra > Issue Type: Improvement > Components: Local Write-Read Paths, Streaming and Messaging >Reporter: Ariel Weisberg >Assignee: Sergio Bossa > Attachments: 9318-3.0-nits-trailing-spaces.patch, backpressure.png, > limit.btm, no_backpressure.png > > > It's possible to somewhat bound the amount of load accepted into the cluster > by bounding the number of in-flight requests and request bytes. > An implementation might do something like track the number of outstanding > bytes and requests and if it reaches a high watermark disable read on client > connections until it goes back below some low watermark. > Need to make sure that disabling read on the client connection won't > introduce other issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11195) paging may returns incomplete results on small page size
[ https://issues.apache.org/jira/browse/CASSANDRA-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458674#comment-15458674 ] Russ Hatch commented on CASSANDRA-11195: Full upgrade suite runs look good. There was a singular failure on each due to an unrelated dtest import problem (I've opened a dtest PR for). > paging may returns incomplete results on small page size > > > Key: CASSANDRA-11195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11195 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey >Assignee: Benjamin Lerer > Labels: dtest > Attachments: allfiles.tar.gz, node1.log, node1_debug.log, node2.log, > node2_debug.log > > > This was found through a flapping test, and running that test is still the > easiest way to repro the issue. On CI we're seeing a 40-50% failure rate, but > locally this test fails much less frequently. > If I attach a python debugger and re-query the "bad" query, it continues to > return incomplete data indefinitely. If I go directly to cqlsh I can see all > rows just fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11195) paging may returns incomplete results on small page size
[ https://issues.apache.org/jira/browse/CASSANDRA-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457013#comment-15457013 ] Russ Hatch commented on CASSANDRA-11195: Multiplex jobs passed, so I started full upgrade suite runs: 3.0: http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-upgrade_blerer_11195_30-upgrade/ 3.9: http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-upgrade_blerer_11195_39-upgrade/ > paging may returns incomplete results on small page size > > > Key: CASSANDRA-11195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11195 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey >Assignee: Benjamin Lerer > Labels: dtest > Attachments: allfiles.tar.gz, node1.log, node1_debug.log, node2.log, > node2_debug.log > > > This was found through a flapping test, and running that test is still the > easiest way to repro the issue. On CI we're seeing a 40-50% failure rate, but > locally this test fails much less frequently. > If I attach a python debugger and re-query the "bad" query, it continues to > return incomplete data indefinitely. If I go directly to cqlsh I can see all > rows just fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11195) paging may returns incomplete results on small page size
[ https://issues.apache.org/jira/browse/CASSANDRA-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456632#comment-15456632 ] Russ Hatch edited comment on CASSANDRA-11195 at 9/1/16 11:13 PM: - OK, I've got some multiplex jobs running some test repetitions against your code: 3.0: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/306/ 3.9: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/307/ If I got everything correct those should have results within an hour or two, then I'll kick off a full upgrade suite run for both of those branches as well. was (Author: rhatch): OK, I've got some multiplex jobs running some test repetitions against your code: 3.0: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/303/ 3.9: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/304/ If I got everything correct those should have results within an hour or two, then I'll kick off a full upgrade suite run for both of those branches as well. > paging may returns incomplete results on small page size > > > Key: CASSANDRA-11195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11195 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey >Assignee: Benjamin Lerer > Labels: dtest > Attachments: allfiles.tar.gz, node1.log, node1_debug.log, node2.log, > node2_debug.log > > > This was found through a flapping test, and running that test is still the > easiest way to repro the issue. On CI we're seeing a 40-50% failure rate, but > locally this test fails much less frequently. > If I attach a python debugger and re-query the "bad" query, it continues to > return incomplete data indefinitely. If I go directly to cqlsh I can see all > rows just fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11195) paging may returns incomplete results on small page size
[ https://issues.apache.org/jira/browse/CASSANDRA-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456632#comment-15456632 ] Russ Hatch commented on CASSANDRA-11195: OK, I've got some multiplex jobs running some test repetitions against your code: 3.0: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/303/ 3.9: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/304/ If I got everything correct those should have results within an hour or two, then I'll kick off a full upgrade suite run for both of those branches as well. > paging may returns incomplete results on small page size > > > Key: CASSANDRA-11195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11195 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey >Assignee: Benjamin Lerer > Labels: dtest > Attachments: allfiles.tar.gz, node1.log, node1_debug.log, node2.log, > node2_debug.log > > > This was found through a flapping test, and running that test is still the > easiest way to repro the issue. On CI we're seeing a 40-50% failure rate, but > locally this test fails much less frequently. > If I attach a python debugger and re-query the "bad" query, it continues to > return incomplete data indefinitely. If I go directly to cqlsh I can see all > rows just fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11195) paging may returns incomplete results on small page size
[ https://issues.apache.org/jira/browse/CASSANDRA-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452803#comment-15452803 ] Russ Hatch commented on CASSANDRA-11195: Ah, interesting -- let me know if I can help out! > paging may returns incomplete results on small page size > > > Key: CASSANDRA-11195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11195 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey >Assignee: Benjamin Lerer > Labels: dtest > Attachments: allfiles.tar.gz, node1.log, node1_debug.log, node2.log, > node2_debug.log > > > This was found through a flapping test, and running that test is still the > easiest way to repro the issue. On CI we're seeing a 40-50% failure rate, but > locally this test fails much less frequently. > If I attach a python debugger and re-query the "bad" query, it continues to > return incomplete data indefinitely. If I go directly to cqlsh I can see all > rows just fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12171) counter mismatch during rolling upgrade from 2.2 to 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-12171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433120#comment-15433120
]
Russ Hatch edited comment on CASSANDRA-12171 at 8/23/16 4:16 PM:
-
Though it's a different test, I wonder if the dtest code fix used on
CASSANDRA-12092 might help out here (same symptom of counters not being at
expected value).
was (Author: rhatch):
Though it's a different test, I wonder if the test code fix used on
CASSANDRA-12092 might help out here (same symptom of counters not being at
expected value).
> counter mismatch during rolling upgrade from 2.2 to 3.0
> ---
>
> Key: CASSANDRA-12171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12171
> Project: Cassandra
> Issue Type: Bug
>Reporter: Russ Hatch
>Assignee: Aleksey Yeschenko
>
> This may occur on other versions, but 3.0 is where I observed it recently.
> N=RF=3, counter writes at quorum, reads at quorum.
> This is being seen on some upgrade tests I'm currently repairing here:
> https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix (this
> branch is to resolve an issue where counters were not being properly tested
> during rolling upgrade tests).
> The test runs a continuous counter incrementing process, as well as a
> continuous counter checking process. Once a counter value has been verified,
> the test code makes it eligible to be incremented again.
> The test is encountering the problem when trying to check an expected counter
> value and not matching expectations, for example:
> {noformat}
> Traceback (most recent call last):
> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> File
> "/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 210, in counter_checker
> tester.assertEqual(expected_count, actual_count)
> File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
> assertion_func(first, second, msg=msg)
> File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
> raise self.failureException(msg)
> AssertionError: 1 != 2
> ERROR
> {noformat}
> To check if something else could be going on, I did an experiment where I
> changed the test to not upgrade nodes (just drain, stop, start) and the
> mismatch didn't occur in several attempts. So it appears something about
> upgrading is possibly the culprit.
> To run the test and repro locally:
> {noformat}
> grab my dtest branch at
> https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix
> export UPGRADE_TEST_RUN=true
> nosetests -v
> upgrade_tests/upgrade_through_versions_test.py:TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_test
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12171) counter mismatch during rolling upgrade from 2.2 to 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-12171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433120#comment-15433120
]
Russ Hatch commented on CASSANDRA-12171:
Though it's a different test, I wonder if the test code fix used on
CASSANDRA-12092 might help out here (same symptom of counters not being at
expected value).
> counter mismatch during rolling upgrade from 2.2 to 3.0
> ---
>
> Key: CASSANDRA-12171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12171
> Project: Cassandra
> Issue Type: Bug
>Reporter: Russ Hatch
>Assignee: Aleksey Yeschenko
>
> This may occur on other versions, but 3.0 is where I observed it recently.
> N=RF=3, counter writes at quorum, reads at quorum.
> This is being seen on some upgrade tests I'm currently repairing here:
> https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix (this
> branch is to resolve an issue where counters were not being properly tested
> during rolling upgrade tests).
> The test runs a continuous counter incrementing process, as well as a
> continuous counter checking process. Once a counter value has been verified,
> the test code makes it eligible to be incremented again.
> The test is encountering the problem when trying to check an expected counter
> value and not matching expectations, for example:
> {noformat}
> Traceback (most recent call last):
> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> File
> "/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 210, in counter_checker
> tester.assertEqual(expected_count, actual_count)
> File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
> assertion_func(first, second, msg=msg)
> File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
> raise self.failureException(msg)
> AssertionError: 1 != 2
> ERROR
> {noformat}
> To check if something else could be going on, I did an experiment where I
> changed the test to not upgrade nodes (just drain, stop, start) and the
> mismatch didn't occur in several attempts. So it appears something about
> upgrading is possibly the culprit.
> To run the test and repro locally:
> {noformat}
> grab my dtest branch at
> https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix
> export UPGRADE_TEST_RUN=true
> nosetests -v
> upgrade_tests/upgrade_through_versions_test.py:TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_test
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12492) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.cql3_non_compound_range_tombstones_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428530#comment-15428530
]
Russ Hatch commented on CASSANDRA-12492:
100 runs without failure.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.cql3_non_compound_range_tombstones_test
>
>
> Key: CASSANDRA-12492
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12492
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_dtest_upgrade/25/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/cql3_non_compound_range_tombstones_test
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 1562, in cql3_non_compound_range_tombstones_test
> ThriftConsistencyLevel.ALL)
> File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py",
> line 1175, in batch_mutate
> self.recv_batch_mutate()
> File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py",
> line 1201, in recv_batch_mutate
> raise result.te
> "TimedOutException(acknowledged_by=1, paxos_in_progress=None,
> acknowledged_by_batchlog=None)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-12492) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.cql3_non_compound_range_tombstones_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch resolved CASSANDRA-12492.
Resolution: Cannot Reproduce
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.cql3_non_compound_range_tombstones_test
>
>
> Key: CASSANDRA-12492
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12492
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_dtest_upgrade/25/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/cql3_non_compound_range_tombstones_test
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 1562, in cql3_non_compound_range_tombstones_test
> ThriftConsistencyLevel.ALL)
> File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py",
> line 1175, in batch_mutate
> self.recv_batch_mutate()
> File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py",
> line 1201, in recv_batch_mutate
> raise result.te
> "TimedOutException(acknowledged_by=1, paxos_in_progress=None,
> acknowledged_by_batchlog=None)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-10848) Upgrade paging dtests involving deletion flap on CassCI
[
https://issues.apache.org/jira/browse/CASSANDRA-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch resolved CASSANDRA-10848.
Resolution: Fixed
> Upgrade paging dtests involving deletion flap on CassCI
> ---
>
> Key: CASSANDRA-10848
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10848
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jim Witschey
>Assignee: Russ Hatch
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> A number of dtests in the {{upgrade_tests.paging_tests}} that involve
> deletion flap with the following error:
> {code}
> Requested pages were not delivered before timeout.
> {code}
> This may just be an effect of CASSANDRA-10730, but it's worth having a look
> at separately. Here are some examples of tests flapping in this way:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10848) Upgrade paging dtests involving deletion flap on CassCI
[
https://issues.apache.org/jira/browse/CASSANDRA-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428379#comment-15428379
]
Russ Hatch commented on CASSANDRA-10848:
Given the current (happy) state of upgrade tests, I think this issue is safe to
resolve. If there turns out to be a recurrence I think it would be better
handled by a fresh issue for clarity.
> Upgrade paging dtests involving deletion flap on CassCI
> ---
>
> Key: CASSANDRA-10848
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10848
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jim Witschey
>Assignee: Russ Hatch
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> A number of dtests in the {{upgrade_tests.paging_tests}} that involve
> deletion flap with the following error:
> {code}
> Requested pages were not delivered before timeout.
> {code}
> This may just be an effect of CASSANDRA-10730, but it's worth having a look
> at separately. Here are some examples of tests flapping in this way:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12492) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.cql3_non_compound_range_tombstones_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428344#comment-15428344
]
Russ Hatch commented on CASSANDRA-12492:
multiplexing here:
https://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/268/
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.cql3_non_compound_range_tombstones_test
>
>
> Key: CASSANDRA-12492
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12492
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_dtest_upgrade/25/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/cql3_non_compound_range_tombstones_test
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 1562, in cql3_non_compound_range_tombstones_test
> ThriftConsistencyLevel.ALL)
> File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py",
> line 1175, in batch_mutate
> self.recv_batch_mutate()
> File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py",
> line 1201, in recv_batch_mutate
> raise result.te
> "TimedOutException(acknowledged_by=1, paxos_in_progress=None,
> acknowledged_by_batchlog=None)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12492) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.cql3_non_compound_range_tombstones_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427006#comment-15427006
]
Russ Hatch commented on CASSANDRA-12492:
Local attempt upgrading to same sha doesn't fail (c72a7bb), so it doesn't
appear to be a problem with that particular commit.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.cql3_non_compound_range_tombstones_test
>
>
> Key: CASSANDRA-12492
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12492
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_dtest_upgrade/25/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/cql3_non_compound_range_tombstones_test
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 1562, in cql3_non_compound_range_tombstones_test
> ThriftConsistencyLevel.ALL)
> File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py",
> line 1175, in batch_mutate
> self.recv_batch_mutate()
> File "/home/automaton/cassandra-dtest/thrift_bindings/v22/Cassandra.py",
> line 1201, in recv_batch_mutate
> raise result.te
> "TimedOutException(acknowledged_by=1, paxos_in_progress=None,
> acknowledged_by_batchlog=None)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-12441) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch resolved CASSANDRA-12441.
Resolution: Not A Problem
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
> -
>
> Key: CASSANDRA-12441
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12441
> Project: Cassandra
> Issue Type: Test
>Reporter: Craig Kodman
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.0_large_dtest/20/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_2_x_To_indev_3_0_x/rolling_upgrade_with_internode_ssl_test
> {code}
> Error Message
> Ran out of time waiting for queue size (1) to be 'le' to 0. Aborting.
> >> begin captured logging <<
> dtest: DEBUG: Upgrade test beginning, setting CASSANDRA_VERSION to 2.2.7, and
> jdk to 8. (Prior values will be restored after test).
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-U_C4o2
> ccm: DEBUG: Log-watching thread starting.
> dtest: DEBUG: Done setting configuration options:
> { 'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: Setting extra configuration options:
> { 'partitioner': 'org.apache.cassandra.dht.Murmur3Partitioner'}
> dtest: DEBUG: Versions to test ( 'upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x'>):
> ['2.2.7', 'git:465def8e295b453dc22430dc4bb7039f6921151e']
> dtest: DEBUG: ***using internode ssl***
> dtest: DEBUG: generating keystore.jks in [/tmp/dtest-U_C4o2]
> dtest: DEBUG: exporting cert from keystore.jks in [/tmp/dtest-U_C4o2]
> dtest: DEBUG: importing cert into truststore.jks in [/tmp/dtest-U_C4o2]
> dtest: DEBUG: Creating cluster (2.2.7)
> dtest: DEBUG: Current upgrade path: ['***2.2.7***',
> 'git:465def8e295b453dc22430dc4bb7039f6921151e']
> dtest: DEBUG: rows written (but not verified) queue size is at 417, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 442, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 474, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 503, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 536, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 565, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 597, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 634, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 663, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 690, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size (5044) is 'ge' to
> 5000. Continuing.
> dtest: DEBUG: counters incremented (but not verified) queue size (5042) is
> 'ge' to 5000. Continuing.
> dtest: DEBUG: Upgrading ['node1'] to
> git:465def8e295b453dc22430dc4bb7039f6921151e
> dtest: DEBUG: JAVA_HOME: /usr/lib/jvm/jdk1.8.0_51
> dtest: DEBUG: Shutting down node: node1
> ccm: INFO: Fetching Cassandra updates...
> dtest: DEBUG: Set new cassandra dir for node1:
> /home/automaton/.ccm/repository/gitCOLON465def8e295b453dc22430dc4bb7039f6921151e
> dtest: DEBUG: Starting node1 on new version
> (git:465def8e295b453dc22430dc4bb7039f6921151e)
> dtest: DEBUG: Successfully upgraded 1 of 3 nodes to
> git:465def8e295b453dc22430dc4bb7039f6921151e
> dtest: DEBUG: Upgrading ['node2'] to
> git:465def8e295b453dc22430dc4bb7039f6921151e
> dtest: DEBUG: JAVA_HOME: /usr/lib/jvm/jdk1.8.0_51
> dtest: DEBUG: Shutting down node: node2
> ccm: INFO: Fetching Cassandra updates...
> dtest: DEBUG: Set new cassandra dir for node2:
> /home/automaton/.ccm/repository/gitCOLON465def8e295b453dc22430dc4bb7039f6921151e
> dtest: DEBUG: Starting node2 on new version
> (git:465def8e295b453dc22430dc4bb7039f6921151e)
> dtest: DEBUG: Successfully upgraded 2 of 3 nodes to
> git:465def8e295b453dc22430dc4bb7039f6921151e
> dtest: DEBUG:
[jira] [Commented] (CASSANDRA-12441) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421888#comment-15421888
]
Russ Hatch commented on CASSANDRA-12441:
Sorry for the delay. So, the bad news is this error is probably useless. We've
had a long-standing programming error in the concurrency code where the wrong
functions are run in the counter related subprocesses (increment/verify).
I've got dtest code in progress to fix this issue, but I delayed while blocked
on CASSANDRA-12171.
In the meantime, I've got a small pr which effectively disables the "counter"
actions since they are not testing the right thing anyway. This will be a
stopgap until 12171 is fixed and counters can then be tested over upgrades.
https://github.com/riptano/cassandra-dtest/pull/1240
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
> -
>
> Key: CASSANDRA-12441
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12441
> Project: Cassandra
> Issue Type: Test
>Reporter: Craig Kodman
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.0_large_dtest/20/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_2_x_To_indev_3_0_x/rolling_upgrade_with_internode_ssl_test
> {code}
> Error Message
> Ran out of time waiting for queue size (1) to be 'le' to 0. Aborting.
> >> begin captured logging <<
> dtest: DEBUG: Upgrade test beginning, setting CASSANDRA_VERSION to 2.2.7, and
> jdk to 8. (Prior values will be restored after test).
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-U_C4o2
> ccm: DEBUG: Log-watching thread starting.
> dtest: DEBUG: Done setting configuration options:
> { 'initial_token': None,
> 'num_tokens': '32',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> dtest: DEBUG: Setting extra configuration options:
> { 'partitioner': 'org.apache.cassandra.dht.Murmur3Partitioner'}
> dtest: DEBUG: Versions to test ( 'upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x'>):
> ['2.2.7', 'git:465def8e295b453dc22430dc4bb7039f6921151e']
> dtest: DEBUG: ***using internode ssl***
> dtest: DEBUG: generating keystore.jks in [/tmp/dtest-U_C4o2]
> dtest: DEBUG: exporting cert from keystore.jks in [/tmp/dtest-U_C4o2]
> dtest: DEBUG: importing cert into truststore.jks in [/tmp/dtest-U_C4o2]
> dtest: DEBUG: Creating cluster (2.2.7)
> dtest: DEBUG: Current upgrade path: ['***2.2.7***',
> 'git:465def8e295b453dc22430dc4bb7039f6921151e']
> dtest: DEBUG: rows written (but not verified) queue size is at 417, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 442, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 474, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 503, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 536, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 565, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 597, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 634, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 663, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size is at 690, target is
> to reach 'ge' 5000
> dtest: DEBUG: rows written (but not verified) queue size (5044) is 'ge' to
> 5000. Continuing.
> dtest: DEBUG: counters incremented (but not verified) queue size (5042) is
> 'ge' to 5000. Continuing.
> dtest: DEBUG: Upgrading ['node1'] to
> git:465def8e295b453dc22430dc4bb7039f6921151e
> dtest: DEBUG: JAVA_HOME: /usr/lib/jvm/jdk1.8.0_51
> dtest: DEBUG: Shutting down node: node1
> ccm: INFO: Fetching Cassandra updates...
> dtest: DEBUG: Set new cassandra dir for node1:
> /home/automaton/.ccm/repository/gitCOLON465def8e295b453dc22430dc4bb7039f6921151e
> dtest: DEBUG: Starting node1 on new version
> (git:465def8e295b453dc22430dc4bb7039f6921151e)
> dtest: DEBUG: Successfully upgraded 1 of 3
[jira] [Commented] (CASSANDRA-12249) dtest failure in upgrade_tests.paging_test.TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.basic_paging_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414505#comment-15414505
]
Russ Hatch commented on CASSANDRA-12249:
There was a dtest hiccup (should now be patched up), so I kicked off another
build:
http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-upgrade_thobbs_12249-3.8-upgrade/2/
> dtest failure in
> upgrade_tests.paging_test.TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.basic_paging_test
> ---
>
> Key: CASSANDRA-12249
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12249
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Tyler Hobbs
> Labels: dtest
> Fix For: 3.8
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.paging_test/TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/basic_paging_test
> Failed on CassCI build cassandra-3.8_dtest_upgrade #1
> This is on a mixed version cluster, one node is 3.0.8 and the other is
> 3.8-tentative.
> Stack trace looks like:
> {code}
> ERROR [MessagingService-Incoming-/127.0.0.1] 2016-07-20 04:51:02,836
> CassandraDaemon.java:201 - Exception in thread
> Thread[MessagingService-Incoming-/127.0.0.1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$LegacyPagedRangeCommandSerializer.deserialize(ReadCommand.java:1042)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.ReadCommand$LegacyPagedRangeCommandSerializer.deserialize(ReadCommand.java:964)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> {code}
> This trace is from the 3.0.8 node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12249) dtest failure in upgrade_tests.paging_test.TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.basic_paging_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414390#comment-15414390
]
Russ Hatch commented on CASSANDRA-12249:
No problem -- job is here:
http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-upgrade_thobbs_12249-3.8-upgrade/
The job should queue up soon as build #1. Should take maybe 4-6 hours once
started (will run on ec2 w/10 nodes to match the last run of the 3.8 upgrade
job).
If you're using the autojobs setup, feel free to push branches with 'upgrade'
somewhere in the name, and the autojobs system will build an upgrade job for
you which you can run on-demand. If you do that, I'd suggest ec2 for now since
openstack has been a bit more noisy thus far wrt upgrades.
> dtest failure in
> upgrade_tests.paging_test.TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.basic_paging_test
> ---
>
> Key: CASSANDRA-12249
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12249
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Tyler Hobbs
> Labels: dtest
> Fix For: 3.8
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.paging_test/TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/basic_paging_test
> Failed on CassCI build cassandra-3.8_dtest_upgrade #1
> This is on a mixed version cluster, one node is 3.0.8 and the other is
> 3.8-tentative.
> Stack trace looks like:
> {code}
> ERROR [MessagingService-Incoming-/127.0.0.1] 2016-07-20 04:51:02,836
> CassandraDaemon.java:201 - Exception in thread
> Thread[MessagingService-Incoming-/127.0.0.1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$LegacyPagedRangeCommandSerializer.deserialize(ReadCommand.java:1042)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.ReadCommand$LegacyPagedRangeCommandSerializer.deserialize(ReadCommand.java:964)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> {code}
> This trace is from the 3.0.8 node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10848) Upgrade paging dtests involving deletion flap on CassCI
[
https://issues.apache.org/jira/browse/CASSANDRA-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412448#comment-15412448
]
Russ Hatch commented on CASSANDRA-10848:
I think of the remaining failures, a large number (maybe all) can be attributed
to CASSANDRA-12249. I'd like to keep this open until 12249 is resolved, but
that combined with the proto v4 test fixes might have us in pretty good shape
with these tests.
> Upgrade paging dtests involving deletion flap on CassCI
> ---
>
> Key: CASSANDRA-10848
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10848
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jim Witschey
>Assignee: Russ Hatch
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> A number of dtests in the {{upgrade_tests.paging_tests}} that involve
> deletion flap with the following error:
> {code}
> Requested pages were not delivered before timeout.
> {code}
> This may just be an effect of CASSANDRA-10730, but it's worth having a look
> at separately. Here are some examples of tests flapping in this way:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12249) dtest failure in upgrade_tests.paging_test.TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.basic_paging_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412433#comment-15412433
]
Russ Hatch commented on CASSANDRA-12249:
This issue appears to be the root cause of a number of CI test failures on the
upgrade suites.
> dtest failure in
> upgrade_tests.paging_test.TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.basic_paging_test
> ---
>
> Key: CASSANDRA-12249
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12249
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Benjamin Lerer
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.paging_test/TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/basic_paging_test
> Failed on CassCI build cassandra-3.8_dtest_upgrade #1
> This is on a mixed version cluster, one node is 3.0.8 and the other is
> 3.8-tentative.
> Stack trace looks like:
> {code}
> ERROR [MessagingService-Incoming-/127.0.0.1] 2016-07-20 04:51:02,836
> CassandraDaemon.java:201 - Exception in thread
> Thread[MessagingService-Incoming-/127.0.0.1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$LegacyPagedRangeCommandSerializer.deserialize(ReadCommand.java:1042)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.ReadCommand$LegacyPagedRangeCommandSerializer.deserialize(ReadCommand.java:964)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> {code}
> This trace is from the 3.0.8 node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12249) dtest failure in upgrade_tests.paging_test.TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.basic_paging_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412381#comment-15412381
]
Russ Hatch commented on CASSANDRA-12249:
I can get this consistently with:
{noformat}nosetests -v
upgrade_tests/paging_test.py:TestPagingDataNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_paging_using_secondary_indexes{noformat}
For that particular test, I can see that the first page of results appears to
be returned, but the remaining pages aren't.
> dtest failure in
> upgrade_tests.paging_test.TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.basic_paging_test
> ---
>
> Key: CASSANDRA-12249
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12249
> Project: Cassandra
> Issue Type: Bug
>Reporter: Philip Thompson
>Assignee: Benjamin Lerer
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.paging_test/TestPagingDataNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/basic_paging_test
> Failed on CassCI build cassandra-3.8_dtest_upgrade #1
> This is on a mixed version cluster, one node is 3.0.8 and the other is
> 3.8-tentative.
> Stack trace looks like:
> {code}
> ERROR [MessagingService-Incoming-/127.0.0.1] 2016-07-20 04:51:02,836
> CassandraDaemon.java:201 - Exception in thread
> Thread[MessagingService-Incoming-/127.0.0.1,5,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$LegacyPagedRangeCommandSerializer.deserialize(ReadCommand.java:1042)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.db.ReadCommand$LegacyPagedRangeCommandSerializer.deserialize(ReadCommand.java:964)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> {code}
> This trace is from the 3.0.8 node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10848) Upgrade paging dtests involving deletion flap on CassCI
[
https://issues.apache.org/jira/browse/CASSANDRA-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412149#comment-15412149
]
Russ Hatch commented on CASSANDRA-10848:
Ah ok, I misunderstood. Got a PR open for that change too, here:
https://github.com/riptano/cassandra-dtest/pull/1190.
> Upgrade paging dtests involving deletion flap on CassCI
> ---
>
> Key: CASSANDRA-10848
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10848
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jim Witschey
>Assignee: Russ Hatch
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> A number of dtests in the {{upgrade_tests.paging_tests}} that involve
> deletion flap with the following error:
> {code}
> Requested pages were not delivered before timeout.
> {code}
> This may just be an effect of CASSANDRA-10730, but it's worth having a look
> at separately. Here are some examples of tests flapping in this way:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12356) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_2_1_x_To_indev_2_2_x.bug_10652_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410146#comment-15410146 ] Russ Hatch commented on CASSANDRA-12356: yup > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_2_1_x_To_indev_2_2_x.bug_10652_test > - > > Key: CASSANDRA-12356 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12356 > Project: Cassandra > Issue Type: Test >Reporter: Craig Kodman >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node3.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.2_dtest_upgrade/7/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_1_x_To_indev_2_2_x/bug_10652_test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10848) Upgrade paging dtests involving deletion flap on CassCI
[
https://issues.apache.org/jira/browse/CASSANDRA-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408551#comment-15408551
]
Russ Hatch commented on CASSANDRA-10848:
I created a dtest fix for the 2.2/3.0 problem (CASSANDRA-10880), so hopefully
that can help at least a few tests. Dtest pr here:
https://github.com/riptano/cassandra-dtest/pull/1176
Digging into the results, looks like we're still seeing timeouts on other
version combinations, so I don't think the 2.2/3.0 v4 problem could account for
everything. Still not quite sure what else could be going on, perhaps a subtle
problem with the test code itself when the timeouts are encountered.
> Upgrade paging dtests involving deletion flap on CassCI
> ---
>
> Key: CASSANDRA-10848
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10848
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jim Witschey
>Assignee: DS Test Eng
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> A number of dtests in the {{upgrade_tests.paging_tests}} that involve
> deletion flap with the following error:
> {code}
> Requested pages were not delivered before timeout.
> {code}
> This may just be an effect of CASSANDRA-10730, but it's worth having a look
> at separately. Here are some examples of tests flapping in this way:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10848) Upgrade paging dtests involving deletion flap on CassCI
[
https://issues.apache.org/jira/browse/CASSANDRA-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408261#comment-15408261
]
Russ Hatch commented on CASSANDRA-10848:
I'll look into the protocol version issue you mentioned, that could very well
be impacting the tests.
> Upgrade paging dtests involving deletion flap on CassCI
> ---
>
> Key: CASSANDRA-10848
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10848
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jim Witschey
>Assignee: DS Test Eng
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> A number of dtests in the {{upgrade_tests.paging_tests}} that involve
> deletion flap with the following error:
> {code}
> Requested pages were not delivered before timeout.
> {code}
> This may just be an effect of CASSANDRA-10730, but it's worth having a look
> at separately. Here are some examples of tests flapping in this way:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12370) dtest failure in batch_test.TestBatch.acknowledged_by_batchlog_not_set_when_batchlog_write_fails_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406212#comment-15406212 ] Russ Hatch commented on CASSANDRA-12370: I think this might just be a symptom of test env problems with the latest python driver. > dtest failure in > batch_test.TestBatch.acknowledged_by_batchlog_not_set_when_batchlog_write_fails_test > - > > Key: CASSANDRA-12370 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12370 > Project: Cassandra > Issue Type: Test >Reporter: Craig Kodman >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-2.2_dtest/664/testReport/batch_test/TestBatch/acknowledged_by_batchlog_not_set_when_batchlog_write_fails_test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12092:
---
Assignee: DS Test Eng (was: Russ Hatch)
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-12363) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.list_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch resolved CASSANDRA-12363.
Resolution: Cannot Reproduce
100 iterations with no repro. Resolving.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.list_test
> --
>
> Key: CASSANDRA-12363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12363
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/5/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/list_test
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 1375, in list_test
> assert_one(cursor, "SELECT tags FROM user", [['foo', 'bar', 'foo',
> 'foobar']])
> File "/home/automaton/cassandra-dtest/assertions.py", line 124, in
> assert_one
> assert list_res == [expected], "Expected {} from {}, but got
> {}".format([expected], query, list_res)
> "Expected [[['foo', 'bar', 'foo', 'foobar']]] from SELECT tags FROM user, but
> got [[[u'foo', u'foo', u'bar', u'foo', u'foobar']]]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12363) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.list_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404945#comment-15404945
]
Russ Hatch commented on CASSANDRA-12363:
Trying to repro w/multiplex x100 over here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/210/
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.list_test
> --
>
> Key: CASSANDRA-12363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12363
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/5/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/list_test
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 1375, in list_test
> assert_one(cursor, "SELECT tags FROM user", [['foo', 'bar', 'foo',
> 'foobar']])
> File "/home/automaton/cassandra-dtest/assertions.py", line 124, in
> assert_one
> assert list_res == [expected], "Expected {} from {}, but got
> {}".format([expected], query, list_res)
> "Expected [[['foo', 'bar', 'foo', 'foobar']]] from SELECT tags FROM user, but
> got [[[u'foo', u'foo', u'bar', u'foo', u'foobar']]]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12363) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.list_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404881#comment-15404881
]
Russ Hatch commented on CASSANDRA-12363:
doesn't repro locally, even on exact same sha
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x.list_test
> --
>
> Key: CASSANDRA-12363
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12363
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/5/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_x/list_test
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 1375, in list_test
> assert_one(cursor, "SELECT tags FROM user", [['foo', 'bar', 'foo',
> 'foobar']])
> File "/home/automaton/cassandra-dtest/assertions.py", line 124, in
> assert_one
> assert list_res == [expected], "Expected {} from {}, but got
> {}".format([expected], query, list_res)
> "Expected [[['foo', 'bar', 'foo', 'foobar']]] from SELECT tags FROM user, but
> got [[[u'foo', u'foo', u'bar', u'foo', u'foobar']]]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-12364) dtest failure in upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
[
https://issues.apache.org/jira/browse/CASSANDRA-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch reassigned CASSANDRA-12364:
--
Assignee: Russ Hatch (was: DS Test Eng)
> dtest failure in
> upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
>
>
> Key: CASSANDRA-12364
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12364
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_cell_TTL_expiry_during_paging
> {code}
> Error Message
> Unexpected error in log, see stdout
> {code}
> {code}
> Standard Output
> http://git-wip-us.apache.org/repos/asf/cassandra.git
> git:5051c0f6eb3f984600600c9577d6b5ece9038c74
> Unexpected error in node1 log, error:
> ERROR [InternalResponseStage:4] 2016-07-28 03:23:02,097
> CassandraDaemon.java:217 - Exception in thread
> Thread[InternalResponseStage:4,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> ~[na:1.8.0_51]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.waitForFlushes(ColumnFamilyStore.java:930)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:892)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/2113926365.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> {code}
> Related failures:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/11/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_ttl_deletions/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/static_columns_with_distinct_test/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/refuse_in_with_indexes_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12364) dtest failure in upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
[
https://issues.apache.org/jira/browse/CASSANDRA-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12364:
---
Reviewer: Jim Witschey
> dtest failure in
> upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
>
>
> Key: CASSANDRA-12364
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12364
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_cell_TTL_expiry_during_paging
> {code}
> Error Message
> Unexpected error in log, see stdout
> {code}
> {code}
> Standard Output
> http://git-wip-us.apache.org/repos/asf/cassandra.git
> git:5051c0f6eb3f984600600c9577d6b5ece9038c74
> Unexpected error in node1 log, error:
> ERROR [InternalResponseStage:4] 2016-07-28 03:23:02,097
> CassandraDaemon.java:217 - Exception in thread
> Thread[InternalResponseStage:4,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> ~[na:1.8.0_51]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.waitForFlushes(ColumnFamilyStore.java:930)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:892)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/2113926365.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> {code}
> Related failures:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/11/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_ttl_deletions/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/static_columns_with_distinct_test/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/refuse_in_with_indexes_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12364) dtest failure in upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
[
https://issues.apache.org/jira/browse/CASSANDRA-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12364:
---
Status: Patch Available (was: Open)
> dtest failure in
> upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
>
>
> Key: CASSANDRA-12364
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12364
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_cell_TTL_expiry_during_paging
> {code}
> Error Message
> Unexpected error in log, see stdout
> {code}
> {code}
> Standard Output
> http://git-wip-us.apache.org/repos/asf/cassandra.git
> git:5051c0f6eb3f984600600c9577d6b5ece9038c74
> Unexpected error in node1 log, error:
> ERROR [InternalResponseStage:4] 2016-07-28 03:23:02,097
> CassandraDaemon.java:217 - Exception in thread
> Thread[InternalResponseStage:4,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> ~[na:1.8.0_51]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.waitForFlushes(ColumnFamilyStore.java:930)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:892)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/2113926365.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> {code}
> Related failures:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/11/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_ttl_deletions/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/static_columns_with_distinct_test/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/refuse_in_with_indexes_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12364) dtest failure in upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
[
https://issues.apache.org/jira/browse/CASSANDRA-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404772#comment-15404772
]
Russ Hatch commented on CASSANDRA-12364:
Thanks [~slebresne] -- I added a dtest pr to ignore the exception. We were
actually already ignoring this in one other place (the
upgrade_through_versions_tests.py).
https://github.com/riptano/cassandra-dtest/pull/1162
> dtest failure in
> upgrade_tests.paging_test.TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x.test_cell_TTL_expiry_during_paging
>
>
> Key: CASSANDRA-12364
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12364
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_cell_TTL_expiry_during_paging
> {code}
> Error Message
> Unexpected error in log, see stdout
> {code}
> {code}
> Standard Output
> http://git-wip-us.apache.org/repos/asf/cassandra.git
> git:5051c0f6eb3f984600600c9577d6b5ece9038c74
> Unexpected error in node1 log, error:
> ERROR [InternalResponseStage:4] 2016-07-28 03:23:02,097
> CassandraDaemon.java:217 - Exception in thread
> Thread[InternalResponseStage:4,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
> down
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> ~[na:1.8.0_51]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.waitForFlushes(ColumnFamilyStore.java:930)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:892)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/2113926365.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> ~[apache-cassandra-3.7.jar:3.7]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> {code}
> Related failures:
> http://cassci.datastax.com/job/trunk_dtest_upgrade/11/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes3RF3_Upgrade_current_3_x_To_indev_3_x/test_ttl_deletions/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/static_columns_with_distinct_test/
> http://cassci.datastax.com/job/trunk_dtest_upgrade/12/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/refuse_in_with_indexes_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-12341) dtest failure in hintedhandoff_test.TestHintedHandoffConfig.hintedhandoff_enabled_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch resolved CASSANDRA-12341.
Resolution: Cannot Reproduce
no repro on x100 multiplex.
> dtest failure in
> hintedhandoff_test.TestHintedHandoffConfig.hintedhandoff_enabled_test
> --
>
> Key: CASSANDRA-12341
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12341
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/440/testReport/hintedhandoff_test/TestHintedHandoffConfig/hintedhandoff_enabled_test
> {code}
> Error Message
> 29 Jul 2016 00:56:17 [node1] Missing: ['Finished hinted']:
> INFO [HANDSHAKE-/127.0.0.2] 2016-07-29 00:54:14,4.
> See system.log for remainder
> {code}
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 125, in
> hintedhandoff_enabled_test
> self._do_hinted_handoff(node1, node2, True)
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 61, in
> _do_hinted_handoff
> node1.watch_log_for(["Finished hinted"], from_mark=log_mark, timeout=120)
> File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + "
> [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" +
> reads[:50] + ".\nSee {} for remainder".format(filename))
> "29 Jul 2016 00:56:17 [node1] Missing: ['Finished hinted']:\nINFO
> [HANDSHAKE-/127.0.0.2] 2016-07-29 00:54:14,4.\nSee system.log for
> remainder
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12316) dtest failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_round_trip_random
[
https://issues.apache.org/jira/browse/CASSANDRA-12316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402980#comment-15402980
]
Russ Hatch edited comment on CASSANDRA-12316 at 8/2/16 6:22 PM:
This looks like possibly just an ephemeral env problem
{noformat}('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried
connecting to [('127.0.0.1', 9042)]. Last error: Connection
refused")}){noformat}
But multiplexing the test here to be sure:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/209/
was (Author: rhatch):
This looks like possibly just an ephemeral env problem
{noformat}('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried
connecting to [('127.0.0.1', 9042)]. Last error: Connection
refused")}){noformat}
But multiplexing the test here to be sure:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/205/
> dtest failure in
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_round_trip_random
> --
>
> Key: CASSANDRA-12316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12316
> Project: Cassandra
> Issue Type: Test
>Reporter: Craig Kodman
>Assignee: DS Test Eng
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_novnode_dtest/16/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_round_trip_random
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-12234) dtest failure in sstable_generation_loading_test.TestSSTableGenerationAndLoading.sstableloader_uppercase_keyspace_name_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch resolved CASSANDRA-12234.
Resolution: Cannot Reproduce
100 runs pass. resolving.
> dtest failure in
> sstable_generation_loading_test.TestSSTableGenerationAndLoading.sstableloader_uppercase_keyspace_name_test
> ---
>
> Key: CASSANDRA-12234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12234
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/430/testReport/sstable_generation_loading_test/TestSSTableGenerationAndLoading/sstableloader_uppercase_keyspace_name_test
> Failed on CassCI build trunk_novnode_dtest #430
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/sstable_generation_loading_test.py",
> line 229, in sstableloader_uppercase_keyspace_name_test
> self.load_sstable_with_configuration(ks='"Keyspace1"')
> File "/home/automaton/cassandra-dtest/sstable_generation_loading_test.py",
> line 187, in load_sstable_with_configuration
> "sstableloader exited with a non-zero status: {}".format(exit_status))
> File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
> assertion_func(first, second, msg=msg)
> File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
> raise self.failureException(msg)
> "sstableloader exited with a non-zero status: 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12352) dtest failure in upgrade_tests.storage_engine_upgrade_test.TestLoadLaCompactSStables.sstableloader_compression_snappy_to_snappy_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12352:
---
Description:
These are all similar looking test failures, which appear to be timeout issues
of some kind.
{noformat}('Unable to connect to any servers', {'127.0.0.1':
OperationTimedOut('errors=Timed out creating connection (10 seconds),
last_host=None',)}){noformat}
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.storage_engine_upgrade_test/TestLoadLaCompactSStables/sstableloader_compression_snappy_to_snappy_test
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.storage_engine_upgrade_test/TestLoadLaSStables/sstableloader_compression_deflate_to_deflate_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDataNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/test_paging_across_multi_wide_rows/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDataNodes3RF3_Upgrade_current_2_1_x_To_indev_3_0_x/test_paging_across_multi_wide_rows/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingSizeNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/test_undefined_page_size_default/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_2_1_x_To_indev_3_0_x/test_cell_TTL_expiry_during_paging/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingSizeNodes2RF1_Upgrade_current_2_2_x_To_indev_3_0_x/test_undefined_page_size_default/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/test_cell_TTL_expiry_during_paging/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/test_row_TTL_expiry_during_paging/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.storage_engine_upgrade_test/TestBootstrapAfterUpgrade/upgrade_with_unclustered_CQL_table_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_2_x_To_indev_3_0_x/range_tombstones_compaction_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/bug_5732_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_2_x_To_indev_3_0_x/collection_flush_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/large_count_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_0_x/conditional_delete_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/range_tombstones_compaction_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes2RF1_Upgrade_current_2_2_x_To_indev_3_0_x/test_failure_threshold_deletions/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/test_ttl_deletions/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_0_x/limit_multiget_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_1_x_To_indev_3_0_x/limit_multiget_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_2_2_x_To_indev_3_0_x/test_data_change_impacting_later_page/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_2_x_To_indev_3_0_x/expanded_list_item_conditional_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_2_x_To_indev_3_0_x/no_range_ghost_test/
http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_0_x/limit_compact_table_test/
[jira] [Issue Comment Deleted] (CASSANDRA-12352) dtest failure in upgrade_tests.storage_engine_upgrade_test.TestLoadLaCompactSStables.sstableloader_compression_snappy_to_snappy_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12352:
---
Comment: was deleted
(was: [~craig.kodman] could you clarify what the Decription above means? Are
those all failures for this issue?)
> dtest failure in
> upgrade_tests.storage_engine_upgrade_test.TestLoadLaCompactSStables.sstableloader_compression_snappy_to_snappy_test
>
>
> Key: CASSANDRA-12352
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12352
> Project: Cassandra
> Issue Type: Test
>Reporter: Craig Kodman
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node2.log, node2_debug.log
>
>
> These are all similar looking test failures, which appear to be timeout
> issues of some kind.
> {noformat}('Unable to connect to any servers', {'127.0.0.1':
> OperationTimedOut('errors=Timed out creating connection (10 seconds),
> last_host=None',)}){noformat}
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.storage_engine_upgrade_test/TestLoadLaCompactSStables/sstableloader_compression_snappy_to_snappy_test
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.storage_engine_upgrade_test/TestLoadLaSStables/sstableloader_compression_deflate_to_deflate_test/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDataNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/test_paging_across_multi_wide_rows/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDataNodes3RF3_Upgrade_current_2_1_x_To_indev_3_0_x/test_paging_across_multi_wide_rows/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingSizeNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/test_undefined_page_size_default/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_2_1_x_To_indev_3_0_x/test_cell_TTL_expiry_during_paging/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingSizeNodes2RF1_Upgrade_current_2_2_x_To_indev_3_0_x/test_undefined_page_size_default/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/test_cell_TTL_expiry_during_paging/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/test_row_TTL_expiry_during_paging/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.storage_engine_upgrade_test/TestBootstrapAfterUpgrade/upgrade_with_unclustered_CQL_table_test/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_2_x_To_indev_3_0_x/range_tombstones_compaction_test/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/bug_5732_test/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_2_x_To_indev_3_0_x/collection_flush_test/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/large_count_test/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_0_x/conditional_delete_test/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/range_tombstones_compaction_test/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes2RF1_Upgrade_current_2_2_x_To_indev_3_0_x/test_failure_threshold_deletions/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/test_ttl_deletions/
> http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_0_x/limit_multiget_test/
>
[jira] [Commented] (CASSANDRA-12352) dtest failure in upgrade_tests.storage_engine_upgrade_test.TestLoadLaCompactSStables.sstableloader_compression_snappy_to_snappy_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403043#comment-15403043 ] Russ Hatch commented on CASSANDRA-12352: [~craig.kodman] could you clarify what the Decription above means? Are those all failures for this issue? > dtest failure in > upgrade_tests.storage_engine_upgrade_test.TestLoadLaCompactSStables.sstableloader_compression_snappy_to_snappy_test > > > Key: CASSANDRA-12352 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12352 > Project: Cassandra > Issue Type: Test >Reporter: Craig Kodman >Assignee: DS Test Eng > Labels: dtest > Attachments: node1.log, node1_debug.log, node2.log, node2_debug.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.storage_engine_upgrade_test/TestLoadLaCompactSStables/sstableloader_compression_snappy_to_snappy_test > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.storage_engine_upgrade_test/TestLoadLaSStables/sstableloader_compression_deflate_to_deflate_test/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDataNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/test_paging_across_multi_wide_rows/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDataNodes3RF3_Upgrade_current_2_1_x_To_indev_3_0_x/test_paging_across_multi_wide_rows/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingSizeNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/test_undefined_page_size_default/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes3RF3_Upgrade_current_2_1_x_To_indev_3_0_x/test_cell_TTL_expiry_during_paging/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingSizeNodes2RF1_Upgrade_current_2_2_x_To_indev_3_0_x/test_undefined_page_size_default/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/test_cell_TTL_expiry_during_paging/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingDatasetChangesNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/test_row_TTL_expiry_during_paging/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.storage_engine_upgrade_test/TestBootstrapAfterUpgrade/upgrade_with_unclustered_CQL_table_test/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_2_x_To_indev_3_0_x/range_tombstones_compaction_test/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/bug_5732_test/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_2_x_To_indev_3_0_x/collection_flush_test/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/large_count_test/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_0_x/conditional_delete_test/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_2_1_x_To_indev_3_0_x/range_tombstones_compaction_test/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes2RF1_Upgrade_current_2_2_x_To_indev_3_0_x/test_failure_threshold_deletions/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.paging_test/TestPagingWithDeletionsNodes2RF1_Upgrade_current_3_0_x_To_indev_3_0_x/test_ttl_deletions/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_indev_3_0_x/limit_multiget_test/ > http://cassci.datastax.com/job/cassandra-3.0_dtest_upgrade/10/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_2_1_x_To_indev_3_0_x/limit_multiget_test/ >
[jira] [Commented] (CASSANDRA-12341) dtest failure in hintedhandoff_test.TestHintedHandoffConfig.hintedhandoff_enabled_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403040#comment-15403040
]
Russ Hatch commented on CASSANDRA-12341:
Single Ci failure, multiplexing x100 here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/206/
> dtest failure in
> hintedhandoff_test.TestHintedHandoffConfig.hintedhandoff_enabled_test
> --
>
> Key: CASSANDRA-12341
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12341
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/440/testReport/hintedhandoff_test/TestHintedHandoffConfig/hintedhandoff_enabled_test
> {code}
> Error Message
> 29 Jul 2016 00:56:17 [node1] Missing: ['Finished hinted']:
> INFO [HANDSHAKE-/127.0.0.2] 2016-07-29 00:54:14,4.
> See system.log for remainder
> {code}
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 125, in
> hintedhandoff_enabled_test
> self._do_hinted_handoff(node1, node2, True)
> File "/home/automaton/cassandra-dtest/hintedhandoff_test.py", line 61, in
> _do_hinted_handoff
> node1.watch_log_for(["Finished hinted"], from_mark=log_mark, timeout=120)
> File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + "
> [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" +
> reads[:50] + ".\nSee {} for remainder".format(filename))
> "29 Jul 2016 00:56:17 [node1] Missing: ['Finished hinted']:\nINFO
> [HANDSHAKE-/127.0.0.2] 2016-07-29 00:54:14,4.\nSee system.log for
> remainder
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12316) dtest failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_round_trip_random
[
https://issues.apache.org/jira/browse/CASSANDRA-12316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402980#comment-15402980
]
Russ Hatch commented on CASSANDRA-12316:
This looks like possibly just an ephemeral env problem
{noformat}('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried
connecting to [('127.0.0.1', 9042)]. Last error: Connection
refused")}){noformat}
But multiplexing the test here to be sure:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/205/
> dtest failure in
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_round_trip_random
> --
>
> Key: CASSANDRA-12316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12316
> Project: Cassandra
> Issue Type: Test
>Reporter: Craig Kodman
>Assignee: DS Test Eng
> Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-3.9_novnode_dtest/16/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_round_trip_random
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12267) dtest failure in materialized_views_test.TestMaterializedViews.add_dc_after_mv_simple_replication_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402949#comment-15402949
]
Russ Hatch commented on CASSANDRA-12267:
one flap in recent history, multiplexing x100 here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/204/
> dtest failure in
> materialized_views_test.TestMaterializedViews.add_dc_after_mv_simple_replication_test
> --
>
> Key: CASSANDRA-12267
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12267
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log,
> node4.log, node4_debug.log, node4_gc.log, node5.log, node5_debug.log,
> node5_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/432/testReport/materialized_views_test/TestMaterializedViews/add_dc_after_mv_simple_replication_test
> {code}
> Standard Output
> Unexpected error in node4 log, error:
> ERROR [main] 2016-07-21 01:57:17,951 MigrationManager.java:164 - Migration
> task failed to complete
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-12234) dtest failure in sstable_generation_loading_test.TestSSTableGenerationAndLoading.sstableloader_uppercase_keyspace_name_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch reassigned CASSANDRA-12234:
--
Assignee: Russ Hatch (was: DS Test Eng)
> dtest failure in
> sstable_generation_loading_test.TestSSTableGenerationAndLoading.sstableloader_uppercase_keyspace_name_test
> ---
>
> Key: CASSANDRA-12234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12234
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/430/testReport/sstable_generation_loading_test/TestSSTableGenerationAndLoading/sstableloader_uppercase_keyspace_name_test
> Failed on CassCI build trunk_novnode_dtest #430
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/sstable_generation_loading_test.py",
> line 229, in sstableloader_uppercase_keyspace_name_test
> self.load_sstable_with_configuration(ks='"Keyspace1"')
> File "/home/automaton/cassandra-dtest/sstable_generation_loading_test.py",
> line 187, in load_sstable_with_configuration
> "sstableloader exited with a non-zero status: {}".format(exit_status))
> File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
> assertion_func(first, second, msg=msg)
> File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
> raise self.failureException(msg)
> "sstableloader exited with a non-zero status: 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12234) dtest failure in sstable_generation_loading_test.TestSSTableGenerationAndLoading.sstableloader_uppercase_keyspace_name_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402866#comment-15402866
]
Russ Hatch commented on CASSANDRA-12234:
looks like only a single recent failure. multiplexing x100 here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/202/
> dtest failure in
> sstable_generation_loading_test.TestSSTableGenerationAndLoading.sstableloader_uppercase_keyspace_name_test
> ---
>
> Key: CASSANDRA-12234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12234
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/430/testReport/sstable_generation_loading_test/TestSSTableGenerationAndLoading/sstableloader_uppercase_keyspace_name_test
> Failed on CassCI build trunk_novnode_dtest #430
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/sstable_generation_loading_test.py",
> line 229, in sstableloader_uppercase_keyspace_name_test
> self.load_sstable_with_configuration(ks='"Keyspace1"')
> File "/home/automaton/cassandra-dtest/sstable_generation_loading_test.py",
> line 187, in load_sstable_with_configuration
> "sstableloader exited with a non-zero status: {}".format(exit_status))
> File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
> assertion_func(first, second, msg=msg)
> File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
> raise self.failureException(msg)
> "sstableloader exited with a non-zero status: 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-12116) dtest failure in snapshot_test.TestSnapshot.test_basic_snapshot_and_restore
[
https://issues.apache.org/jira/browse/CASSANDRA-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch resolved CASSANDRA-12116.
Resolution: Cannot Reproduce
200 iterations without a repro. Closing.
> dtest failure in snapshot_test.TestSnapshot.test_basic_snapshot_and_restore
> ---
>
> Key: CASSANDRA-12116
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12116
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/260/testReport/snapshot_test/TestSnapshot/test_basic_snapshot_and_restore
> Failed on CassCI build cassandra-2.1_novnode_dtest #260
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/snapshot_test.py", line 96, in
> test_basic_snapshot_and_restore
> self.restore_snapshot(snapshot_dir, node1, 'ks', 'cf')
> File "/home/automaton/cassandra-dtest/snapshot_test.py", line 68, in
> restore_snapshot
> (" ".join(args), exit_status, stdout, stderr))
> 'sstableloader command \'/home/automaton/cassandra/bin/sstableloader -d
> 127.0.0.1 /tmp/tmpGgR_dT/0/ks/cf\' failed; exit status: 1\'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12196) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402800#comment-15402800
]
Russ Hatch commented on CASSANDRA-12196:
We should re-check this failure against the dtest large job, now that the
upgrade tests have moved over there. Job here:
http://cassci.datastax.com/view/trunk/job/trunk_large_dtest/
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_1_x_To_indev_3_x.bootstrap_test
> --
>
> Key: CASSANDRA-12196
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12196
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log,
> node4.log, node4_debug.log, node4_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_1_x_To_indev_3_x/bootstrap_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 707, in bootstrap_test
> self.upgrade_scenario(after_upgrade_call=(self._bootstrap_new_node,))
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 383, in upgrade_scenario
> call()
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 688, in _bootstrap_new_node
> nnode.start(use_jna=True, wait_other_notice=True,
> wait_for_binary_proto=True)
> File "/home/automaton/ccm/ccmlib/node.py", line 634, in start
> node.watch_log_for_alive(self, from_mark=mark)
> File "/home/automaton/ccm/ccmlib/node.py", line 481, in watch_log_for_alive
> self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout,
> filename=filename)
> File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + "
> [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" +
> reads[:50] + ".\nSee {} for remainder".format(filename))
> "13 Jul 2016 02:23:05 [node2] Missing: ['127.0.0.4.* now UP']:\nINFO
> [HANDSHAKE-/127.0.0.4] 2016-07-13 02:21:00,2.\nSee system.log for
> remainder
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-12195) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch resolved CASSANDRA-12195.
Resolution: Not A Problem
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
> -
>
> Key: CASSANDRA-12195
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12195
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_2_x_To_indev_3_0_x/rolling_upgrade_with_internode_ssl_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 295, in rolling_upgrade_with_internode_ssl_test
> self.upgrade_scenario(rolling=True, internode_ssl=True)
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 352, in upgrade_scenario
> self._check_on_subprocs(self.subprocs)
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 409, in _check_on_subprocs
> raise RuntimeError(message)
> "A subprocess has terminated early. Subprocess statuses: Process-13
> (is_alive: False), Process-14 (is_alive: True), Process-15 (is_alive: True),
> Process-16 (is_alive: True), attempting to terminate remaining subprocesses
> now.
> {code}
> node2_debug.log is too large to attach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12195) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402690#comment-15402690
]
Russ Hatch commented on CASSANDRA-12195:
With this (or similar) test moved now to the dtest large job, I don't see any
recent failure. In any case, if it recurs I think we'll get better debug
output. I think this is safe to resolve for now and can be reopened later if it
happens again.
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
> -
>
> Key: CASSANDRA-12195
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12195
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_2_x_To_indev_3_0_x/rolling_upgrade_with_internode_ssl_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 295, in rolling_upgrade_with_internode_ssl_test
> self.upgrade_scenario(rolling=True, internode_ssl=True)
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 352, in upgrade_scenario
> self._check_on_subprocs(self.subprocs)
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 409, in _check_on_subprocs
> raise RuntimeError(message)
> "A subprocess has terminated early. Subprocess statuses: Process-13
> (is_alive: False), Process-14 (is_alive: True), Process-15 (is_alive: True),
> Process-16 (is_alive: True), attempting to terminate remaining subprocesses
> now.
> {code}
> node2_debug.log is too large to attach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12160) dtest failure in counter_tests.TestCounters.upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402683#comment-15402683
]
Russ Hatch commented on CASSANDRA-12160:
There was a single failure in the multiplex job above, but it appears to have
another cause (judging from the exception likely a routine failure). Resolving
this ticket.
> dtest failure in counter_tests.TestCounters.upgrade_test
>
>
> Key: CASSANDRA-12160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12160
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/493/testReport/counter_tests/TestCounters/upgrade_test
> Failed on CassCI build cassandra-2.1_dtest #493
> {code}
> Error Message
> 07 Jul 2016 21:07:28 [node1] Missing: ['127.0.0.2.* now UP']:
> .
> See system.log for remainder
> {code}
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/counter_tests.py", line 101, in
> upgrade_test
> rolling_restart()
> File "/home/automaton/cassandra-dtest/counter_tests.py", line 96, in
> rolling_restart
> nodes[i].start(wait_other_notice=True, wait_for_binary_proto=True)
> File "/home/automaton/ccm/ccmlib/node.py", line 634, in start
> node.watch_log_for_alive(self, from_mark=mark)
> File "/home/automaton/ccm/ccmlib/node.py", line 481, in watch_log_for_alive
> self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout,
> filename=filename)
> File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + "
> [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" +
> reads[:50] + ".\nSee {} for remainder".format(filename))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-12160) dtest failure in counter_tests.TestCounters.upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch resolved CASSANDRA-12160.
Resolution: Cannot Reproduce
> dtest failure in counter_tests.TestCounters.upgrade_test
>
>
> Key: CASSANDRA-12160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12160
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/493/testReport/counter_tests/TestCounters/upgrade_test
> Failed on CassCI build cassandra-2.1_dtest #493
> {code}
> Error Message
> 07 Jul 2016 21:07:28 [node1] Missing: ['127.0.0.2.* now UP']:
> .
> See system.log for remainder
> {code}
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/counter_tests.py", line 101, in
> upgrade_test
> rolling_restart()
> File "/home/automaton/cassandra-dtest/counter_tests.py", line 96, in
> rolling_restart
> nodes[i].start(wait_other_notice=True, wait_for_binary_proto=True)
> File "/home/automaton/ccm/ccmlib/node.py", line 634, in start
> node.watch_log_for_alive(self, from_mark=mark)
> File "/home/automaton/ccm/ccmlib/node.py", line 481, in watch_log_for_alive
> self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout,
> filename=filename)
> File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + "
> [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" +
> reads[:50] + ".\nSee {} for remainder".format(filename))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12116) dtest failure in snapshot_test.TestSnapshot.test_basic_snapshot_and_restore
[
https://issues.apache.org/jira/browse/CASSANDRA-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402677#comment-15402677
]
Russ Hatch commented on CASSANDRA-12116:
Looks like no failure since build #260. Multiplexing here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/201/
> dtest failure in snapshot_test.TestSnapshot.test_basic_snapshot_and_restore
> ---
>
> Key: CASSANDRA-12116
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12116
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/260/testReport/snapshot_test/TestSnapshot/test_basic_snapshot_and_restore
> Failed on CassCI build cassandra-2.1_novnode_dtest #260
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/snapshot_test.py", line 96, in
> test_basic_snapshot_and_restore
> self.restore_snapshot(snapshot_dir, node1, 'ks', 'cf')
> File "/home/automaton/cassandra-dtest/snapshot_test.py", line 68, in
> restore_snapshot
> (" ".join(args), exit_status, stdout, stderr))
> 'sstableloader command \'/home/automaton/cassandra/bin/sstableloader -d
> 127.0.0.1 /tmp/tmpGgR_dT/0/ks/cf\' failed; exit status: 1\'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12064) dtest failure in auth_test.TestAuth.login_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402619#comment-15402619
]
Russ Hatch commented on CASSANDRA-12064:
Looks like that last one was 100% pass -- can we close this?
> dtest failure in auth_test.TestAuth.login_test
> --
>
> Key: CASSANDRA-12064
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12064
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Fix For: 3.9
>
> Attachments: node1.log, node1_debug.log, node1_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1284/testReport/auth_test/TestAuth/login_test
> Failed on CassCI build trunk_dtest #1284
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/auth_test.py", line 82, in login_test
> assert isinstance(e.errors.values()[0], AuthenticationFailed)
> {code}
> Logs are attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11909) dtest failure in cql_tests.AbortedQueriesTester.remote_query_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402613#comment-15402613
]
Russ Hatch commented on CASSANDRA-11909:
looks like there's been a large number of passes on this tests since the last
failure, might not be a problem anymore?
> dtest failure in cql_tests.AbortedQueriesTester.remote_query_test
> -
>
> Key: CASSANDRA-11909
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11909
> Project: Cassandra
> Issue Type: Test
>Reporter: Craig Kodman
>Assignee: DS Test Eng
> Labels: dtest
> Fix For: 3.9
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1243/testReport/cql_tests/AbortedQueriesTester/remote_query_test
> Failed on CassCI build trunk_dtest #1243
> {code}
> ERROR [SharedPool-Worker-1] 2016-05-27 10:08:49,471 Keyspace.java:504 -
> Attempting to mutate non-existant table 01855840-23f3-11e6-912e-c5dc3b68cc6d
> (ks.test2)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400145#comment-15400145
]
Russ Hatch commented on CASSANDRA-12092:
[~Stefania] any idea what could be causing this test to fail (intermittently)
but on the same key when it does?
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1547#comment-1547
]
Russ Hatch commented on CASSANDRA-12092:
Interestingly both failures shown on this ticket happened at 'key 200', which
looks to be writing at quorum, reading back at one, with serial unset. For a
random-looking failure, the same key of 200 is a suspicious value.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1533#comment-1533
]
Russ Hatch commented on CASSANDRA-12092:
failure from recent multiplex:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/195/testReport/node_1_iter_059.consistency_test/TestAccuracy/test_simple_strategy_counters/
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1532#comment-1532
]
Russ Hatch commented on CASSANDRA-12092:
1 failure in 200 iterations. Either the test is bad or there's a bug here.
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12160) dtest failure in counter_tests.TestCounters.upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399932#comment-15399932
]
Russ Hatch commented on CASSANDRA-12160:
single flap in recent history, going to try a multiplex (x100) here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/196/
> dtest failure in counter_tests.TestCounters.upgrade_test
>
>
> Key: CASSANDRA-12160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12160
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/493/testReport/counter_tests/TestCounters/upgrade_test
> Failed on CassCI build cassandra-2.1_dtest #493
> {code}
> Error Message
> 07 Jul 2016 21:07:28 [node1] Missing: ['127.0.0.2.* now UP']:
> .
> See system.log for remainder
> {code}
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/counter_tests.py", line 101, in
> upgrade_test
> rolling_restart()
> File "/home/automaton/cassandra-dtest/counter_tests.py", line 96, in
> rolling_restart
> nodes[i].start(wait_other_notice=True, wait_for_binary_proto=True)
> File "/home/automaton/ccm/ccmlib/node.py", line 634, in start
> node.watch_log_for_alive(self, from_mark=mark)
> File "/home/automaton/ccm/ccmlib/node.py", line 481, in watch_log_for_alive
> self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout,
> filename=filename)
> File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + "
> [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" +
> reads[:50] + ".\nSee {} for remainder".format(filename))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-12160) dtest failure in counter_tests.TestCounters.upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch reassigned CASSANDRA-12160:
--
Assignee: Russ Hatch (was: DS Test Eng)
> dtest failure in counter_tests.TestCounters.upgrade_test
>
>
> Key: CASSANDRA-12160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12160
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/493/testReport/counter_tests/TestCounters/upgrade_test
> Failed on CassCI build cassandra-2.1_dtest #493
> {code}
> Error Message
> 07 Jul 2016 21:07:28 [node1] Missing: ['127.0.0.2.* now UP']:
> .
> See system.log for remainder
> {code}
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/counter_tests.py", line 101, in
> upgrade_test
> rolling_restart()
> File "/home/automaton/cassandra-dtest/counter_tests.py", line 96, in
> rolling_restart
> nodes[i].start(wait_other_notice=True, wait_for_binary_proto=True)
> File "/home/automaton/ccm/ccmlib/node.py", line 634, in start
> node.watch_log_for_alive(self, from_mark=mark)
> File "/home/automaton/ccm/ccmlib/node.py", line 481, in watch_log_for_alive
> self.watch_log_for(tofind, from_mark=from_mark, timeout=timeout,
> filename=filename)
> File "/home/automaton/ccm/ccmlib/node.py", line 449, in watch_log_for
> raise TimeoutError(time.strftime("%d %b %Y %H:%M:%S", time.gmtime()) + "
> [" + self.name + "] Missing: " + str([e.pattern for e in tofind]) + ":\n" +
> reads[:50] + ".\nSee {} for remainder".format(filename))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch reassigned CASSANDRA-12092:
--
Assignee: Russ Hatch (was: DS Test Eng)
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: Russ Hatch
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12092) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
[
https://issues.apache.org/jira/browse/CASSANDRA-12092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399825#comment-15399825
]
Russ Hatch commented on CASSANDRA-12092:
Since this is one isolated flap in recent history, testing with multiplex (200
iterations) here:
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/195/
> dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters
>
>
> Key: CASSANDRA-12092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12092
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node2.log, node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.1_dtest/484/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters
> Failed on CassCI build cassandra-2.1_dtest #484
> {code}
> Standard Error
> Traceback (most recent call last):
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 514, in run
> valid_fcn(v)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 497, in
> validate_counters
> check_all_sessions(s, n, c)
> File "/home/automaton/cassandra-dtest/consistency_test.py", line 490, in
> check_all_sessions
> "value of %s at key %d, instead got these values: %s" % (write_nodes,
> val, n, results)
> AssertionError: Failed to read value from sufficient number of nodes,
> required 2 nodes to have a counter value of 1 at key 200, instead got these
> values: [0, 0, 1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10848) Upgrade paging dtests involving deletion flap on CassCI
[
https://issues.apache.org/jira/browse/CASSANDRA-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399643#comment-15399643
]
Russ Hatch commented on CASSANDRA-10848:
I've had similar difficulty trying to get a test error to repro with 500
iterations. But the CI results still stand, so I'm not sure how we repro and
fix the test issue.
> Upgrade paging dtests involving deletion flap on CassCI
> ---
>
> Key: CASSANDRA-10848
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10848
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jim Witschey
>Assignee: DS Test Eng
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> A number of dtests in the {{upgrade_tests.paging_tests}} that involve
> deletion flap with the following error:
> {code}
> Requested pages were not delivered before timeout.
> {code}
> This may just be an effect of CASSANDRA-10730, but it's worth having a look
> at separately. Here are some examples of tests flapping in this way:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12236) RTE from new CDC column breaks in flight queries.
[
https://issues.apache.org/jira/browse/CASSANDRA-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396238#comment-15396238
]
Russ Hatch commented on CASSANDRA-12236:
With any luck we'll get results on the autojobs created job here:
http://cassci.datastax.com/view/All_Jobs/job/pcmanus-upgrade_12236-upgrade/1/
> RTE from new CDC column breaks in flight queries.
> -
>
> Key: CASSANDRA-12236
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12236
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeremiah Jordan
>Assignee: Sylvain Lebresne
> Fix For: 3.x
>
> Attachments: 12236.txt
>
>
> This RTE is not harmless. It will cause the internode connection to break
> which will cause all in flight requests between these nodes to die/timeout.
> {noformat}
> - Due to changes in schema migration handling and the storage format
> after 3.0, you will
> see error messages such as:
> "java.lang.RuntimeException: Unknown column cdc during
> deserialization"
> in your system logs on a mixed-version cluster during upgrades. This
> error message
> is harmless and due to the 3.8 nodes having cdc added to their schema
> tables while
> the <3.8 nodes do not. This message should cease once all nodes are
> upgraded to 3.8.
> As always, refrain from schema changes during cluster upgrades.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10848) Upgrade paging dtests involving deletion flap on CassCI
[
https://issues.apache.org/jira/browse/CASSANDRA-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-10848:
---
Assignee: DS Test Eng (was: Sylvain Lebresne)
> Upgrade paging dtests involving deletion flap on CassCI
> ---
>
> Key: CASSANDRA-10848
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10848
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jim Witschey
>Assignee: DS Test Eng
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> A number of dtests in the {{upgrade_tests.paging_tests}} that involve
> deletion flap with the following error:
> {code}
> Requested pages were not delivered before timeout.
> {code}
> This may just be an effect of CASSANDRA-10730, but it's worth having a look
> at separately. Here are some examples of tests flapping in this way:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12236) RTE from new CDC column breaks in flight queries.
[
https://issues.apache.org/jira/browse/CASSANDRA-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388685#comment-15388685
]
Russ Hatch edited comment on CASSANDRA-12236 at 7/22/16 12:38 AM:
--
technical difficulties -- trying another run here:
http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-joshupgrade12236-upgrade/3/
was (Author: rhatch):
technical difficulties -- trying another run here:
http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-joshupgrade12236-upgrade/2
> RTE from new CDC column breaks in flight queries.
> -
>
> Key: CASSANDRA-12236
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12236
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeremiah Jordan
>Assignee: Joshua McKenzie
> Fix For: 3.x
>
> Attachments: 12236.txt
>
>
> This RTE is not harmless. It will cause the internode connection to break
> which will cause all in flight requests between these nodes to die/timeout.
> {noformat}
> - Due to changes in schema migration handling and the storage format
> after 3.0, you will
> see error messages such as:
> "java.lang.RuntimeException: Unknown column cdc during
> deserialization"
> in your system logs on a mixed-version cluster during upgrades. This
> error message
> is harmless and due to the 3.8 nodes having cdc added to their schema
> tables while
> the <3.8 nodes do not. This message should cease once all nodes are
> upgraded to 3.8.
> As always, refrain from schema changes during cluster upgrades.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12236) RTE from new CDC column breaks in flight queries.
[
https://issues.apache.org/jira/browse/CASSANDRA-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388685#comment-15388685
]
Russ Hatch commented on CASSANDRA-12236:
technical difficulties -- trying another run here:
http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-joshupgrade12236-upgrade/2
> RTE from new CDC column breaks in flight queries.
> -
>
> Key: CASSANDRA-12236
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12236
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeremiah Jordan
>Assignee: Joshua McKenzie
> Fix For: 3.x
>
> Attachments: 12236.txt
>
>
> This RTE is not harmless. It will cause the internode connection to break
> which will cause all in flight requests between these nodes to die/timeout.
> {noformat}
> - Due to changes in schema migration handling and the storage format
> after 3.0, you will
> see error messages such as:
> "java.lang.RuntimeException: Unknown column cdc during
> deserialization"
> in your system logs on a mixed-version cluster during upgrades. This
> error message
> is harmless and due to the 3.8 nodes having cdc added to their schema
> tables while
> the <3.8 nodes do not. This message should cease once all nodes are
> upgraded to 3.8.
> As always, refrain from schema changes during cluster upgrades.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12236) RTE from new CDC column breaks in flight queries.
[
https://issues.apache.org/jira/browse/CASSANDRA-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388205#comment-15388205
]
Russ Hatch edited comment on CASSANDRA-12236 at 7/21/16 6:58 PM:
-
running here:
http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-joshupgrade12236-upgrade/1/
was (Author: rhatch):
running here:
http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-josh_upgrade_12236-upgrade/1/
> RTE from new CDC column breaks in flight queries.
> -
>
> Key: CASSANDRA-12236
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12236
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeremiah Jordan
>Assignee: Joshua McKenzie
> Fix For: 3.x
>
> Attachments: 12236.txt
>
>
> This RTE is not harmless. It will cause the internode connection to break
> which will cause all in flight requests between these nodes to die/timeout.
> {noformat}
> - Due to changes in schema migration handling and the storage format
> after 3.0, you will
> see error messages such as:
> "java.lang.RuntimeException: Unknown column cdc during
> deserialization"
> in your system logs on a mixed-version cluster during upgrades. This
> error message
> is harmless and due to the 3.8 nodes having cdc added to their schema
> tables while
> the <3.8 nodes do not. This message should cease once all nodes are
> upgraded to 3.8.
> As always, refrain from schema changes during cluster upgrades.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12236) RTE from new CDC column breaks in flight queries.
[
https://issues.apache.org/jira/browse/CASSANDRA-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388205#comment-15388205
]
Russ Hatch commented on CASSANDRA-12236:
running here:
http://cassci.datastax.com/view/Dev/view/knifewine/job/knifewine-josh_upgrade_12236-upgrade/1/
> RTE from new CDC column breaks in flight queries.
> -
>
> Key: CASSANDRA-12236
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12236
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeremiah Jordan
>Assignee: Joshua McKenzie
> Fix For: 3.x
>
> Attachments: 12236.txt
>
>
> This RTE is not harmless. It will cause the internode connection to break
> which will cause all in flight requests between these nodes to die/timeout.
> {noformat}
> - Due to changes in schema migration handling and the storage format
> after 3.0, you will
> see error messages such as:
> "java.lang.RuntimeException: Unknown column cdc during
> deserialization"
> in your system logs on a mixed-version cluster during upgrades. This
> error message
> is harmless and due to the 3.8 nodes having cdc added to their schema
> tables while
> the <3.8 nodes do not. This message should cease once all nodes are
> upgraded to 3.8.
> As always, refrain from schema changes during cluster upgrades.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12236) RTE from new CDC column breaks in flight queries.
[
https://issues.apache.org/jira/browse/CASSANDRA-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388121#comment-15388121
]
Russ Hatch commented on CASSANDRA-12236:
[~JoshuaMcKenzie] I'll take care of this. I'll link to the job when it's up and
running.
> RTE from new CDC column breaks in flight queries.
> -
>
> Key: CASSANDRA-12236
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12236
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeremiah Jordan
>Assignee: Joshua McKenzie
> Fix For: 3.x
>
> Attachments: 12236.txt
>
>
> This RTE is not harmless. It will cause the internode connection to break
> which will cause all in flight requests between these nodes to die/timeout.
> {noformat}
> - Due to changes in schema migration handling and the storage format
> after 3.0, you will
> see error messages such as:
> "java.lang.RuntimeException: Unknown column cdc during
> deserialization"
> in your system logs on a mixed-version cluster during upgrades. This
> error message
> is harmless and due to the 3.8 nodes having cdc added to their schema
> tables while
> the <3.8 nodes do not. This message should cease once all nodes are
> upgraded to 3.8.
> As always, refrain from schema changes during cluster upgrades.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382581#comment-15382581
]
Russ Hatch commented on CASSANDRA-12192:
That is to say we should set the class up with an ignore pattern for this.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12192) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382580#comment-15382580
]
Russ Hatch commented on CASSANDRA-12192:
>From what I understand this one is safe to ignore. See CASSANDRA-12026.
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk.map_keys_indexing_test
>
>
> Key: CASSANDRA-12192
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12192
> Project: Cassandra
> Issue Type: Bug
>Reporter: Sean McCarthy
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/tools.py", line 290, in wrapped
> f(obj)
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 3668, in map_keys_indexing_test
> cursor.execute("TRUNCATE test")
> File "cassandra/cluster.py", line 1941, in
> cassandra.cluster.Session.execute (cassandra/cluster.c:33642)
> return self.execute_async(query, parameters, trace, custom_payload,
> timeout, execution_profile).result()
> File "cassandra/cluster.py", line 3629, in
> cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:69369)
> raise self._final_exception
> '
> {code}
> Related failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.cql_tests/TestCQLNodes2RF1_Upgrade_current_3_0_x_To_head_trunk/map_keys_indexing_test/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12195) dtest failure in upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382568#comment-15382568
]
Russ Hatch commented on CASSANDRA-12195:
Most commonly this is an assertion error killing the process (unhandled
exception), but that should in theory be visible on stderr/stdout.
My work to transition upgrade tests over to regular test jobs will include
giving the subprocesses descriptive names, so at the very least in the future
we'll know what process died.
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_with_internode_ssl_test
> -
>
> Key: CASSANDRA-12195
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12195
> Project: Cassandra
> Issue Type: Test
>Reporter: Sean McCarthy
>Assignee: DS Test Eng
> Labels: dtest
> Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log,
> node2_gc.log, node3.log, node3_debug.log, node3_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all/59/testReport/upgrade_tests.upgrade_through_versions_test/TestUpgrade_current_2_2_x_To_indev_3_0_x/rolling_upgrade_with_internode_ssl_test
> Failed on CassCI build upgrade_tests-all #59
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 295, in rolling_upgrade_with_internode_ssl_test
> self.upgrade_scenario(rolling=True, internode_ssl=True)
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 352, in upgrade_scenario
> self._check_on_subprocs(self.subprocs)
> File
> "/home/automaton/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 409, in _check_on_subprocs
> raise RuntimeError(message)
> "A subprocess has terminated early. Subprocess statuses: Process-13
> (is_alive: False), Process-14 (is_alive: True), Process-15 (is_alive: True),
> Process-16 (is_alive: True), attempting to terminate remaining subprocesses
> now.
> {code}
> node2_debug.log is too large to attach.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11687) dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379884#comment-15379884
]
Russ Hatch commented on CASSANDRA-11687:
another run here (maybe the new assertion will help figure out what's going on)
http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/170/
> dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
> -
>
> Key: CASSANDRA-11687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11687
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> single failure on most recent run (3.0 no-vnode)
> {noformat}
> concurrent rebuild should not be allowed, but one rebuild command should have
> succeeded.
> {noformat}
> http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/217/testReport/rebuild_test/TestRebuild/simple_rebuild_test
> Failed on CassCI build cassandra-3.0_novnode_dtest #217
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11687) dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377341#comment-15377341
]
Russ Hatch commented on CASSANDRA-11687:
might help to know the actual value when we fail with this message:
https://github.com/riptano/cassandra-dtest/pull/1097
> dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
> -
>
> Key: CASSANDRA-11687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11687
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> single failure on most recent run (3.0 no-vnode)
> {noformat}
> concurrent rebuild should not be allowed, but one rebuild command should have
> succeeded.
> {noformat}
> http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/217/testReport/rebuild_test/TestRebuild/simple_rebuild_test
> Failed on CassCI build cassandra-3.0_novnode_dtest #217
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11393) dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373953#comment-15373953
]
Russ Hatch commented on CASSANDRA-11393:
Sorry I missed this earlier. I updated the dtest branch to point to your
3.0/3.9 branches, and I've got another test run kicked off over here:
http://cassci.datastax.com/view/Upgrades/job/upgrade_tests-all-custom_branch_runs/43/
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
> --
>
> Key: CASSANDRA-11393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11393
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination, Streaming and Messaging
>Reporter: Philip Thompson
>Assignee: Benjamin Lerer
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
> Attachments: 11393-3.0.txt
>
>
> We are seeing a failure in the upgrade tests that go from 2.1 to 3.0
> {code}
> node2: ERROR [SharedPool-Worker-2] 2016-03-10 20:05:17,865 Message.java:611 -
> Unexpected exception during request; channel = [id: 0xeb79b477,
> /127.0.0.1:39613 => /127.0.0.2:9042]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1208)
> ~[main/:na]
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1155)
> ~[main/:na]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[main/:na]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor$AlwaysSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:330)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[main/:na]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.execute(SinglePartitionReadCommand.java:302)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.SinglePartitionPager.fetchPage(SinglePartitionPager.java:34)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$Pager$NormalPager.fetchPage(SelectStatement.java:297)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:333)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:472)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:449)
> ~[main/:na]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
> ~[main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [main/:na]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
>
[jira] [Updated] (CASSANDRA-12171) counter mismatch during rolling upgrade from 2.2 to 3.0
[
https://issues.apache.org/jira/browse/CASSANDRA-12171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12171:
---
Reproduced In: 3.0.x
> counter mismatch during rolling upgrade from 2.2 to 3.0
> ---
>
> Key: CASSANDRA-12171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12171
> Project: Cassandra
> Issue Type: Bug
>Reporter: Russ Hatch
>Assignee: Aleksey Yeschenko
>
> This may occur on other versions, but 3.0 is where I observed it recently.
> N=RF=3, counter writes at quorum, reads at quorum.
> This is being seen on some upgrade tests I'm currently repairing here:
> https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix (this
> branch is to resolve an issue where counters were not being properly tested
> during rolling upgrade tests).
> The test runs a continuous counter incrementing process, as well as a
> continuous counter checking process. Once a counter value has been verified,
> the test code makes it eligible to be incremented again.
> The test is encountering the problem when trying to check an expected counter
> value and not matching expectations, for example:
> {noformat}
> Traceback (most recent call last):
> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> File
> "/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
> line 210, in counter_checker
> tester.assertEqual(expected_count, actual_count)
> File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
> assertion_func(first, second, msg=msg)
> File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
> raise self.failureException(msg)
> AssertionError: 1 != 2
> ERROR
> {noformat}
> To check if something else could be going on, I did an experiment where I
> changed the test to not upgrade nodes (just drain, stop, start) and the
> mismatch didn't occur in several attempts. So it appears something about
> upgrading is possibly the culprit.
> To run the test and repro locally:
> {noformat}
> grab my dtest branch at
> https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix
> export UPGRADE_TEST_RUN=true
> nosetests -v
> upgrade_tests/upgrade_through_versions_test.py:TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_test
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-12171) counter mismatch during rolling upgrade from 2.2 to 3.0
Russ Hatch created CASSANDRA-12171:
--
Summary: counter mismatch during rolling upgrade from 2.2 to 3.0
Key: CASSANDRA-12171
URL: https://issues.apache.org/jira/browse/CASSANDRA-12171
Project: Cassandra
Issue Type: Bug
Reporter: Russ Hatch
Assignee: Aleksey Yeschenko
This may occur on other versions, but 3.0 is where I observed it recently.
N=RF=3, counter writes at quorum, reads at quorum.
This is being seen on some upgrade tests I'm currently repairing here:
https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix (this
branch is to resolve an issue where counters were not being properly tested
during rolling upgrade tests).
The test runs a continuous counter incrementing process, as well as a
continuous counter checking process. Once a counter value has been verified,
the test code makes it eligible to be incremented again.
The test is encountering the problem when trying to check an expected counter
value and not matching expectations, for example:
{noformat}
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File
"/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
line 210, in counter_checker
tester.assertEqual(expected_count, actual_count)
File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: 1 != 2
ERROR
{noformat}
To check if something else could be going on, I did an experiment where I
changed the test to not upgrade nodes (just drain, stop, start) and the
mismatch didn't occur in several attempts. So it appears something about
upgrading is possibly the culprit.
To run the test and repro locally:
{noformat}
grab my dtest branch at
https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix
export UPGRADE_TEST_RUN=true
nosetests -v
upgrade_tests/upgrade_through_versions_test.py:TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_test
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11668) InterruptedException in HintsDispatcher
[
https://issues.apache.org/jira/browse/CASSANDRA-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371213#comment-15371213
]
Russ Hatch commented on CASSANDRA-11668:
If it's not appeared on any other tests (I'm not certain) I think we're ok to
close this.
> InterruptedException in HintsDispatcher
> ---
>
> Key: CASSANDRA-11668
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11668
> Project: Cassandra
> Issue Type: Bug
>Reporter: Russ Hatch
> Labels: dtest
>
> This exception was seen when trying to repro a test problem. The original
> issue test problem appears to be a non-issue, but the exception seems like it
> could be worth investigation.
> This happened on upgrade from 3.2.1 to 3.3 HEAD (a soon to be retired
> test-case).
> The test does a rolling upgrade where nodes are one by one stopped, upgraded,
> and started on the new version.
> The exception occurred some time after starting node1 on the upgraded
> version, and upgrading/starting node2 on the new version. Node2 logged the
> exception.
> {noformat}
> node2: ERROR [HintsDispatcher:2] 2016-05-09 23:37:45,816
> CassandraDaemon.java:195 - Exception in thread
> Thread[HintsDispatcher:2,1,main]
> java.lang.AssertionError: java.lang.InterruptedException
> at
> org.apache.cassandra.hints.HintsDispatcher$Callback.await(HintsDispatcher.java:205)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:146)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:121)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:93)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:247)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:219)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:198)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> Caused by: java.lang.InterruptedException: null
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.checkInterrupted(WaitQueue.java:313)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUntil(WaitQueue.java:301)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.utils.concurrent.SimpleCondition.await(SimpleCondition.java:63)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher$Callback.await(HintsDispatcher.java:201)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> ... 11 common frames omitted
> Unexpected error in node2 log, error:
> ERROR [HintsDispatcher:2] 2016-05-09 23:37:45,816 CassandraDaemon.java:195 -
> Exception in thread Thread[HintsDispatcher:2,1,main]
> java.lang.AssertionError: java.lang.InterruptedException
> at
> org.apache.cassandra.hints.HintsDispatcher$Callback.await(HintsDispatcher.java:205)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:146)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:121)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:93)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:247)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:219)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:198)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at
[jira] [Updated] (CASSANDRA-11668) InterruptedException in HintsDispatcher
[
https://issues.apache.org/jira/browse/CASSANDRA-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-11668:
---
Assignee: (was: Russ Hatch)
> InterruptedException in HintsDispatcher
> ---
>
> Key: CASSANDRA-11668
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11668
> Project: Cassandra
> Issue Type: Bug
>Reporter: Russ Hatch
> Labels: dtest
>
> This exception was seen when trying to repro a test problem. The original
> issue test problem appears to be a non-issue, but the exception seems like it
> could be worth investigation.
> This happened on upgrade from 3.2.1 to 3.3 HEAD (a soon to be retired
> test-case).
> The test does a rolling upgrade where nodes are one by one stopped, upgraded,
> and started on the new version.
> The exception occurred some time after starting node1 on the upgraded
> version, and upgrading/starting node2 on the new version. Node2 logged the
> exception.
> {noformat}
> node2: ERROR [HintsDispatcher:2] 2016-05-09 23:37:45,816
> CassandraDaemon.java:195 - Exception in thread
> Thread[HintsDispatcher:2,1,main]
> java.lang.AssertionError: java.lang.InterruptedException
> at
> org.apache.cassandra.hints.HintsDispatcher$Callback.await(HintsDispatcher.java:205)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:146)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:121)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:93)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:247)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:219)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:198)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
> Caused by: java.lang.InterruptedException: null
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.checkInterrupted(WaitQueue.java:313)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUntil(WaitQueue.java:301)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.utils.concurrent.SimpleCondition.await(SimpleCondition.java:63)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher$Callback.await(HintsDispatcher.java:201)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> ... 11 common frames omitted
> Unexpected error in node2 log, error:
> ERROR [HintsDispatcher:2] 2016-05-09 23:37:45,816 CassandraDaemon.java:195 -
> Exception in thread Thread[HintsDispatcher:2,1,main]
> java.lang.AssertionError: java.lang.InterruptedException
> at
> org.apache.cassandra.hints.HintsDispatcher$Callback.await(HintsDispatcher.java:205)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:146)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:121)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:93)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:247)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:219)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:198)
> ~[apache-cassandra-3.2.1.jar:3.2.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_51]
> at
>
[jira] [Commented] (CASSANDRA-12123) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_next_2_1_x_To_current_3_x.cql3_non_compound_range_tombstones_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358045#comment-15358045
]
Russ Hatch commented on CASSANDRA-12123:
For sure. Job running over here:
http://cassci.datastax.com/view/Upgrades/job/upgrade_tests-all-custom_branch_runs/42/
https://github.com/riptano/cassandra-dtest/compare/archive/upgrade_test_tag_2.1.15...test_cassandra_12123?expand=1
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_next_2_1_x_To_current_3_x.cql3_non_compound_range_tombstones_test
> ---
>
> Key: CASSANDRA-12123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12123
> Project: Cassandra
> Issue Type: Test
>Reporter: Philip Thompson
>Assignee: Tyler Hobbs
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all-custom_branch_runs/37/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_next_2_1_x_To_current_3_x/cql3_non_compound_range_tombstones_test
> Failed on CassCI build upgrade_tests-all-custom_branch_runs #37
> Failing here:
> {code}
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 1667, in cql3_non_compound_range_tombstones_test
> self.assertEqual(6, len(row), row)
> {code}
> As we see, the row has more data returned. This implies that data isn't
> properly being shadowed by the tombstone. As such, I'm filing this directly
> as a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12123) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_next_2_1_x_To_current_3_x.cql3_non_compound_range_tombstones_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358015#comment-15358015
]
Russ Hatch commented on CASSANDRA-12123:
Sure thing, sounds like we need to run the suite on:
2.1.same_as_before -> your 3.0 branch
2.1.same_as_before -> your 3.9 branch
2.1.same_as_before -> your trunk branch
That sound correct?
> dtest failure in
> upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_next_2_1_x_To_current_3_x.cql3_non_compound_range_tombstones_test
> ---
>
> Key: CASSANDRA-12123
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12123
> Project: Cassandra
> Issue Type: Test
>Reporter: Philip Thompson
>Assignee: Tyler Hobbs
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
>
> example failure:
> http://cassci.datastax.com/job/upgrade_tests-all-custom_branch_runs/37/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_next_2_1_x_To_current_3_x/cql3_non_compound_range_tombstones_test
> Failed on CassCI build upgrade_tests-all-custom_branch_runs #37
> Failing here:
> {code}
> File "/home/automaton/cassandra-dtest/upgrade_tests/cql_tests.py", line
> 1667, in cql3_non_compound_range_tombstones_test
> self.assertEqual(6, len(row), row)
> {code}
> As we see, the row has more data returned. This implies that data isn't
> properly being shadowed by the tombstone. As such, I'm filing this directly
> as a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11701) [windows] dtest failure in cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_reading_with_skip_and_max_rows
[
https://issues.apache.org/jira/browse/CASSANDRA-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353613#comment-15353613
]
Russ Hatch commented on CASSANDRA-11701:
Sorry I missed that earlier, will try and test it out soon.
> [windows] dtest failure in
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest.test_reading_with_skip_and_max_rows
> -
>
> Key: CASSANDRA-11701
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11701
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: Stefania
> Labels: dtest, windows
>
> looks to be an assertion problem, so could be test or cassandra related:
> e.g.:
> {noformat}
> 1 != 331
> {noformat}
> http://cassci.datastax.com/job/trunk_dtest_win32/404/testReport/cqlsh_tests.cqlsh_copy_tests/CqlshCopyTest/test_reading_with_skip_and_max_rows
> Failed on CassCI build trunk_dtest_win32 #404
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11393) dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352106#comment-15352106
]
Russ Hatch commented on CASSANDRA-11393:
dtest branch here:
https://github.com/riptano/cassandra-dtest/compare/ca11393?expand=1
upgrade job queued here:
http://cassci.datastax.com/view/Upgrades/job/upgrade_tests-all-custom_branch_runs/35/
don't panic if the job above fails, I've been having some intermittent problems
with that job failing lately, can re-run if needed.
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
> --
>
> Key: CASSANDRA-11393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11393
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination, Streaming and Messaging
>Reporter: Philip Thompson
>Assignee: Benjamin Lerer
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
> Attachments: 11393-3.0.txt
>
>
> We are seeing a failure in the upgrade tests that go from 2.1 to 3.0
> {code}
> node2: ERROR [SharedPool-Worker-2] 2016-03-10 20:05:17,865 Message.java:611 -
> Unexpected exception during request; channel = [id: 0xeb79b477,
> /127.0.0.1:39613 => /127.0.0.2:9042]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1208)
> ~[main/:na]
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1155)
> ~[main/:na]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[main/:na]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor$AlwaysSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:330)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[main/:na]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.execute(SinglePartitionReadCommand.java:302)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.SinglePartitionPager.fetchPage(SinglePartitionPager.java:34)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$Pager$NormalPager.fetchPage(SelectStatement.java:297)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:333)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:472)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:449)
> ~[main/:na]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
> ~[main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [main/:na]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>
[jira] [Commented] (CASSANDRA-11393) dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351449#comment-15351449
]
Russ Hatch commented on CASSANDRA-11393:
OK, I'll put together a dtest branch to test:
2.1.latest -> your 3.0 branch
2.2.latest -> your 3.0 branch
2.1.latest -> your 3.7 branch
2.2.latest -> your 3.7 branch
Any other useful cases?
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
> --
>
> Key: CASSANDRA-11393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11393
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination, Streaming and Messaging
>Reporter: Philip Thompson
>Assignee: Benjamin Lerer
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
> Attachments: 11393-3.0.txt
>
>
> We are seeing a failure in the upgrade tests that go from 2.1 to 3.0
> {code}
> node2: ERROR [SharedPool-Worker-2] 2016-03-10 20:05:17,865 Message.java:611 -
> Unexpected exception during request; channel = [id: 0xeb79b477,
> /127.0.0.1:39613 => /127.0.0.2:9042]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1208)
> ~[main/:na]
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1155)
> ~[main/:na]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[main/:na]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor$AlwaysSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:330)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[main/:na]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.execute(SinglePartitionReadCommand.java:302)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.SinglePartitionPager.fetchPage(SinglePartitionPager.java:34)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$Pager$NormalPager.fetchPage(SelectStatement.java:297)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:333)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:472)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:449)
> ~[main/:na]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
> ~[main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [main/:na]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>
[jira] [Commented] (CASSANDRA-11393) dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351420#comment-15351420
]
Russ Hatch commented on CASSANDRA-11393:
I'm not familiar with that setting, is it something we should have set by
default on the regular upgrade jobs?
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
> --
>
> Key: CASSANDRA-11393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11393
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination, Streaming and Messaging
>Reporter: Philip Thompson
>Assignee: Benjamin Lerer
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
> Attachments: 11393-3.0.txt
>
>
> We are seeing a failure in the upgrade tests that go from 2.1 to 3.0
> {code}
> node2: ERROR [SharedPool-Worker-2] 2016-03-10 20:05:17,865 Message.java:611 -
> Unexpected exception during request; channel = [id: 0xeb79b477,
> /127.0.0.1:39613 => /127.0.0.2:9042]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1208)
> ~[main/:na]
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1155)
> ~[main/:na]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[main/:na]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor$AlwaysSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:330)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[main/:na]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.execute(SinglePartitionReadCommand.java:302)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.SinglePartitionPager.fetchPage(SinglePartitionPager.java:34)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$Pager$NormalPager.fetchPage(SelectStatement.java:297)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:333)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:472)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:449)
> ~[main/:na]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
> ~[main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [main/:na]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
>
[jira] [Commented] (CASSANDRA-11393) dtest failure in upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348521#comment-15348521
]
Russ Hatch commented on CASSANDRA-11393:
[~blerer] If you'd like, I can build a custom dtest branch to test these
changes on the upgrade suite. I just need to know what version combinations you
want tested and can take it from there. For point to point upgrades (A->B) we
get the most tests, but they can also be the form of multiple upgrades
(A->B->N), but that type of test is much more limited in scope. Both
varieties is fine too, they aren't mutually exclusive.
> dtest failure in
> upgrade_tests.upgrade_through_versions_test.ProtoV3Upgrade_2_1_UpTo_3_0_HEAD.rolling_upgrade_test
> --
>
> Key: CASSANDRA-11393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11393
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination, Streaming and Messaging
>Reporter: Philip Thompson
>Assignee: Benjamin Lerer
> Labels: dtest
> Fix For: 3.0.x, 3.x
>
> Attachments: 11393-3.0.txt
>
>
> We are seeing a failure in the upgrade tests that go from 2.1 to 3.0
> {code}
> node2: ERROR [SharedPool-Worker-2] 2016-03-10 20:05:17,865 Message.java:611 -
> Unexpected exception during request; channel = [id: 0xeb79b477,
> /127.0.0.1:39613 => /127.0.0.2:9042]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1208)
> ~[main/:na]
> at
> org.apache.cassandra.db.ReadCommand$LegacyReadCommandSerializer.serializedSize(ReadCommand.java:1155)
> ~[main/:na]
> at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166)
> ~[main/:na]
> at
> org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:72)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:609)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:758)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:701)
> ~[main/:na]
> at
> org.apache.cassandra.net.MessagingService.sendRRWithFailure(MessagingService.java:684)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:110)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.makeDataRequests(AbstractReadExecutor.java:85)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor$AlwaysSpeculatingReadExecutor.executeAsync(AbstractReadExecutor.java:330)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.doInitialQueries(StorageProxy.java:1699)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1654)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1601)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1520)
> ~[main/:na]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.execute(SinglePartitionReadCommand.java:302)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.service.pager.SinglePartitionPager.fetchPage(SinglePartitionPager.java:34)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement$Pager$NormalPager.fetchPage(SelectStatement.java:297)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:333)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:209)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:472)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:449)
> ~[main/:na]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
> ~[main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [main/:na]
> at
>
[jira] [Updated] (CASSANDRA-12050) per-patch smoke suites as an early/fast testing tier
[
https://issues.apache.org/jira/browse/CASSANDRA-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-12050:
---
Description:
Coverage data offers a unique opportunity to build metadata about tests and
related Cassandra code.
Using the jacoco coverage api we should be able to build a simple index of
tests (dtest, unit) to the Cassandra code they touch (and the reverse).
When a new patch is introduced, we do a lookup in that index, based on java
source files touched (and possibly lines within those files), and use that to
infer most relevant dtests or unit tests. Patch authors can then run that small
test subset as a first testing pass.
In this way we can build small, focused, test suites that are custom to each
patch. Once this small custom smoke test appears successful things would of
course need to be vetted across a more complete test run on CI.
I think the best interface would simply be ant targets. One target would be
used to build/refresh the test:source code index (run occasionally and saved
somewhere; index building would be time consuming since it will require full
job runs). A second target looks at files touched and does the index lookups,
then outputs a list of tests to run.
The dev user experience might looking something like this:
{noformat}
ant get-custom-smoke -Dcoverage_index=./trunk_coverage_index_SHA_foo.idx
Generating test lists
Please run the following two scripts to vet your changes:
./custom_smoke_junit.sh
./custom_smoke_dtest.sh
{noformat}
was:
Coverage data offers a unique opportunity to build metadata about tests and
related Cassandra code.
Using the jacoco coverage api we should be able to build a simple index of
tests (dtest, unit) to the Cassandra code they touch (and the reverse).
When a new patch is introduced, we do a lookup in that index, based on java
source files touched (and possibly lines within those files), and use that to
infer most relevant dtests or unit tests. Patch authors can then run that small
test subset as a first testing pass.
In this way we can build small, focused, test suites that are custom to each
patch. Once this small custom smoke test appears successful things would of
course need to be vetted across a more complete test run on CI.
I think the best interface would simply be ant targets. One target would be
used to build/refresh the test:source code index (run occasionally and saved
somewhere (index building would be time consuming since it will require full
job runs). A second target looks at files touched and does the index lookups,
then outputs a list of tests to run.
The dev user experience might looking something like this:
{noformat}
ant get-custom-smoke -Dcoverage_index=./trunk_coverage_index_SHA_foo.idx
Generating test lists
Please run the following two scripts to vet your changes:
./custom_smoke_junit.sh
./custom_smoke_dtest.sh
{noformat}
> per-patch smoke suites as an early/fast testing tier
>
>
> Key: CASSANDRA-12050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12050
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Russ Hatch
>Assignee: Russ Hatch
>
> Coverage data offers a unique opportunity to build metadata about tests and
> related Cassandra code.
> Using the jacoco coverage api we should be able to build a simple index of
> tests (dtest, unit) to the Cassandra code they touch (and the reverse).
> When a new patch is introduced, we do a lookup in that index, based on java
> source files touched (and possibly lines within those files), and use that to
> infer most relevant dtests or unit tests. Patch authors can then run that
> small test subset as a first testing pass.
> In this way we can build small, focused, test suites that are custom to each
> patch. Once this small custom smoke test appears successful things would of
> course need to be vetted across a more complete test run on CI.
> I think the best interface would simply be ant targets. One target would be
> used to build/refresh the test:source code index (run occasionally and saved
> somewhere; index building would be time consuming since it will require full
> job runs). A second target looks at files touched and does the index lookups,
> then outputs a list of tests to run.
> The dev user experience might looking something like this:
> {noformat}
> ant get-custom-smoke -Dcoverage_index=./trunk_coverage_index_SHA_foo.idx
> Generating test lists
> Please run the following two scripts to vet your changes:
> ./custom_smoke_junit.sh
> ./custom_smoke_dtest.sh
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-12050) per-patch smoke suites as an early/fast testing tier
Russ Hatch created CASSANDRA-12050:
--
Summary: per-patch smoke suites as an early/fast testing tier
Key: CASSANDRA-12050
URL: https://issues.apache.org/jira/browse/CASSANDRA-12050
Project: Cassandra
Issue Type: Improvement
Reporter: Russ Hatch
Assignee: Russ Hatch
Coverage data offers a unique opportunity to build metadata about tests and
related Cassandra code.
Using the jacoco coverage api we should be able to build a simple index of
tests (dtest, unit) to the Cassandra code they touch (and the reverse).
When a new patch is introduced, we do a lookup in that index, based on java
source files touched (and possibly lines within those files), and use that to
infer most relevant dtests or unit tests. Patch authors can then run that small
test subset as a first testing pass.
In this way we can build small, focused, test suites that are custom to each
patch. Once this small custom smoke test appears successful things would of
course need to be vetted across a more complete test run on CI.
I think the best interface would simply be ant targets. One target would be
used to build/refresh the test:source code index (run occasionally and saved
somewhere (index building would be time consuming since it will require full
job runs). A second target looks at files touched and does the index lookups,
then outputs a list of tests to run.
The dev user experience might looking something like this:
{noformat}
ant get-custom-smoke -Dcoverage_index=./trunk_coverage_index_SHA_foo.idx
Generating test lists
Please run the following two scripts to vet your changes:
./custom_smoke_junit.sh
./custom_smoke_dtest.sh
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-12026) update NEWS.txt to explain system schema exceptions during partial cluster upgrade
Russ Hatch created CASSANDRA-12026:
--
Summary: update NEWS.txt to explain system schema exceptions
during partial cluster upgrade
Key: CASSANDRA-12026
URL: https://issues.apache.org/jira/browse/CASSANDRA-12026
Project: Cassandra
Issue Type: Bug
Reporter: Russ Hatch
Upgrade tests found this exception occuring during upgrades:
{noformat}
node2: ERROR [MessagingService-Incoming-/127.0.0.1] 2016-06-16 20:14:59,268
CassandraDaemon.java:217 - Exception in thread
Thread[MessagingService-Incoming-/127.0.0.1,5,main]
java.lang.RuntimeException: Unknown column cdc during deserialization
at
org.apache.cassandra.db.Columns$Serializer.deserialize(Columns.java:433)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.SerializationHeader$Serializer.deserializeForMessaging(SerializationHeader.java:407)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.deserializeHeader(UnfilteredRowIteratorSerializer.java:192)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:668)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:656)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:341)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:350)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:610)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.service.MigrationManager$MigrationsSerializer.deserialize(MigrationManager.java:593)
~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.7.jar:3.7]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.7.jar:3.7]
{noformat}
Which is apparently normal and should subside after full cluster upgrade to
post-3.0 versions.
NEWS.txt needs an update to let users know this is not a problem during their
upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10664) Fix failing tests
[ https://issues.apache.org/jira/browse/CASSANDRA-10664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323143#comment-15323143 ] Russ Hatch commented on CASSANDRA-10664: +1 > Fix failing tests > - > > Key: CASSANDRA-10664 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10664 > Project: Cassandra > Issue Type: Test >Reporter: Sylvain Lebresne > Labels: dtest > Fix For: 3.0.x > > > This is the continuation of CASSANDRA-10166, just a meta-ticket to group all > tickets related to fixing any unit test or dtest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11687) dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308224#comment-15308224
]
Russ Hatch commented on CASSANDRA-11687:
This might be worth one more trial of like 200, because it was discovered that
the multiplexer was running a smaller heap than the standard job run
(multiplexer has now been fixed), which could maybe affect the results.
> dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
> -
>
> Key: CASSANDRA-11687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11687
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> single failure on most recent run (3.0 no-vnode)
> {noformat}
> concurrent rebuild should not be allowed, but one rebuild command should have
> succeeded.
> {noformat}
> http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/217/testReport/rebuild_test/TestRebuild/simple_rebuild_test
> Failed on CassCI build cassandra-3.0_novnode_dtest #217
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11687) dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
[
https://issues.apache.org/jira/browse/CASSANDRA-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russ Hatch updated CASSANDRA-11687:
---
Assignee: DS Test Eng (was: Russ Hatch)
> dtest failure in rebuild_test.TestRebuild.simple_rebuild_test
> -
>
> Key: CASSANDRA-11687
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11687
> Project: Cassandra
> Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
> Labels: dtest
>
> single failure on most recent run (3.0 no-vnode)
> {noformat}
> concurrent rebuild should not be allowed, but one rebuild command should have
> succeeded.
> {noformat}
> http://cassci.datastax.com/job/cassandra-3.0_novnode_dtest/217/testReport/rebuild_test/TestRebuild/simple_rebuild_test
> Failed on CassCI build cassandra-3.0_novnode_dtest #217
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11195) paging may returns incomplete results on small page size
[ https://issues.apache.org/jira/browse/CASSANDRA-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301259#comment-15301259 ] Russ Hatch commented on CASSANDRA-11195: this is still reprodocing in CI, with the symptom being an assertion error of data not matching. Failed 37/100 times. > paging may returns incomplete results on small page size > > > Key: CASSANDRA-11195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11195 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey >Assignee: Benjamin Lerer > Labels: dtest > Attachments: allfiles.tar.gz, node1.log, node1_debug.log, node2.log, > node2_debug.log > > > This was found through a flapping test, and running that test is still the > easiest way to repro the issue. On CI we're seeing a 40-50% failure rate, but > locally this test fails much less frequently. > If I attach a python debugger and re-query the "bad" query, it continues to > return incomplete data indefinitely. If I go directly to cqlsh I can see all > rows just fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11195) paging may returns incomplete results on small page size
[ https://issues.apache.org/jira/browse/CASSANDRA-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301127#comment-15301127 ] Russ Hatch edited comment on CASSANDRA-11195 at 5/25/16 11:53 PM: -- Going to check if this still repros (will report back). One small update: since the upgrade tests have been reworked a little bit, the most analogous test is probably now: upgrade_tests/paging_test.py:TestPagingDataNodes2RF1_Upgrade_current_2_2_x_To_indev_3_x.static_columns_paging_test trying to repro on CI here: http://cassci.datastax.com/view/Parameterized/job/parameterized_dtest_multiplexer/121/ was (Author: rhatch): Going to check if this still repros (will report back). One small update: since the upgrade tests have been reworked a little bit, the most analogous test is probably now: upgrade_tests/paging_test.py:TestPagingDataNodes2RF1_Upgrade_current_2_2_x_To_indev_3_x.static_columns_paging_test > paging may returns incomplete results on small page size > > > Key: CASSANDRA-11195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11195 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey >Assignee: Benjamin Lerer > Labels: dtest > Attachments: allfiles.tar.gz, node1.log, node1_debug.log, node2.log, > node2_debug.log > > > This was found through a flapping test, and running that test is still the > easiest way to repro the issue. On CI we're seeing a 40-50% failure rate, but > locally this test fails much less frequently. > If I attach a python debugger and re-query the "bad" query, it continues to > return incomplete data indefinitely. If I go directly to cqlsh I can see all > rows just fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
