[jira] [Commented] (CASSANDRA-12251) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432596#comment-15432596 ] Aleksey Yeschenko commented on CASSANDRA-12251: --- Committed as [465bb5d45ccef337382592127e214a0ca16a3d88|https://github.com/apache/cassandra/commit/465bb5d45ccef337382592127e214a0ca16a3d88] to 2.2, and merged with 3.0, 3.0, and trunk. Thanks. > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test > -- > > Key: CASSANDRA-12251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12251 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Alex Petrov > Labels: dtest > Fix For: 2.2.8, 3.0.9, 3.8 > > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/whole_list_conditional_test > Failed on CassCI build cassandra-3.8_dtest_upgrade #1 > Relevant error in logs is > {code} > Unexpected error in node1 log, error: > ERROR [InternalResponseStage:2] 2016-07-20 04:58:45,876 > CassandraDaemon.java:217 - Exception in thread > Thread[InternalResponseStage:2,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > ~[na:1.8.0_51] > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > ~[na:1.8.0_51] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:842) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:822) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:891) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/1129213153.accept(Unknown > Source) ~[na:na] > at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51] > at > org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_51] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > {code} > This is on a mixed 3.0.8, 3.8-tentative cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12251) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431277#comment-15431277 ] Joel Knighton commented on CASSANDRA-12251: --- +1 - lgtm. This should merge forward cleanly from 2.2. > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test > -- > > Key: CASSANDRA-12251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12251 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Alex Petrov > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/whole_list_conditional_test > Failed on CassCI build cassandra-3.8_dtest_upgrade #1 > Relevant error in logs is > {code} > Unexpected error in node1 log, error: > ERROR [InternalResponseStage:2] 2016-07-20 04:58:45,876 > CassandraDaemon.java:217 - Exception in thread > Thread[InternalResponseStage:2,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > ~[na:1.8.0_51] > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > ~[na:1.8.0_51] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:842) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:822) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:891) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/1129213153.accept(Unknown > Source) ~[na:na] > at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51] > at > org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_51] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > {code} > This is on a mixed 3.0.8, 3.8-tentative cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12251) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417881#comment-15417881 ] Alex Petrov commented on CASSANDRA-12251: - You're right. In fact, it all merges cleanly from 2.2 upwards. I've triggered CI as well: |[2.2|https://github.com/ifesdjeen/cassandra/tree/12251-upgrade-2.2]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12251-upgrade-2.2-dtest/]|[testall|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12251-upgrade-2.2-testall/]|[upgrade|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12251-upgrade-2.2-upgrade/]| |[3.0|https://github.com/ifesdjeen/cassandra/tree/12251-upgrade-3.0]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12251-upgrade-3.0-dtest/]|[testall|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12251-upgrade-3.0-testall/]|[upgrade|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12251-upgrade-3.0-upgrade/]| |[trunk|https://github.com/ifesdjeen/cassandra/tree/12251-upgrade-trunk]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12251-upgrade-trunk-dtest/]|[testall|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12251-upgrade-trunk-testall/]|[upgrade|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12251-upgrade-trunk-upgrade/]| > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test > -- > > Key: CASSANDRA-12251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12251 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Alex Petrov > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/whole_list_conditional_test > Failed on CassCI build cassandra-3.8_dtest_upgrade #1 > Relevant error in logs is > {code} > Unexpected error in node1 log, error: > ERROR [InternalResponseStage:2] 2016-07-20 04:58:45,876 > CassandraDaemon.java:217 - Exception in thread > Thread[InternalResponseStage:2,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > ~[na:1.8.0_51] > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > ~[na:1.8.0_51] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:842) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:822) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:891) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/1129213153.accept(Unknown > Source) ~[na:na] > at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51] > at > org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_51]
[jira] [Commented] (CASSANDRA-12251) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415570#comment-15415570 ] Joel Knighton commented on CASSANDRA-12251: --- Patch and CI look good to me. Is there a reason you only ran CI on trunk [~ifesdjeen]? I think this should go into 2.2+. > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test > -- > > Key: CASSANDRA-12251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12251 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Alex Petrov > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/whole_list_conditional_test > Failed on CassCI build cassandra-3.8_dtest_upgrade #1 > Relevant error in logs is > {code} > Unexpected error in node1 log, error: > ERROR [InternalResponseStage:2] 2016-07-20 04:58:45,876 > CassandraDaemon.java:217 - Exception in thread > Thread[InternalResponseStage:2,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > ~[na:1.8.0_51] > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > ~[na:1.8.0_51] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:842) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:822) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:891) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/1129213153.accept(Unknown > Source) ~[na:na] > at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51] > at > org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_51] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > {code} > This is on a mixed 3.0.8, 3.8-tentative cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12251) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408013#comment-15408013 ] Joel Knighton commented on CASSANDRA-12251: --- [~ifesdjeen] - good point, they aren't really duplicates in their symptoms since instead of hitting an exception, those two issues deadlock for a minute. They're the same source though, which is that the ordering of the nonPeriodicTasks/postFlush executors with other parts of the shutdown are not correct. I have a patch that I'll be posting on [CASSANDRA-12260] today which changes the order of executor shutdown. This ticket probably still needs to change the executor for the migration task (I haven't thought this through fully yet). > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test > -- > > Key: CASSANDRA-12251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12251 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Alex Petrov > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/whole_list_conditional_test > Failed on CassCI build cassandra-3.8_dtest_upgrade #1 > Relevant error in logs is > {code} > Unexpected error in node1 log, error: > ERROR [InternalResponseStage:2] 2016-07-20 04:58:45,876 > CassandraDaemon.java:217 - Exception in thread > Thread[InternalResponseStage:2,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > ~[na:1.8.0_51] > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > ~[na:1.8.0_51] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:842) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:822) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:891) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/1129213153.accept(Unknown > Source) ~[na:na] > at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51] > at > org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_51] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > {code} > This is on a mixed 3.0.8, 3.8-tentative cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12251) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407330#comment-15407330 ] Alex Petrov commented on CASSANDRA-12251: - [~jkni] I've checked the two other issues. Most likely I'm missing something, but I could not see similar exception there (as regards the duplicate). But we can / should definitely centralise the review. > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test > -- > > Key: CASSANDRA-12251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12251 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Alex Petrov > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/whole_list_conditional_test > Failed on CassCI build cassandra-3.8_dtest_upgrade #1 > Relevant error in logs is > {code} > Unexpected error in node1 log, error: > ERROR [InternalResponseStage:2] 2016-07-20 04:58:45,876 > CassandraDaemon.java:217 - Exception in thread > Thread[InternalResponseStage:2,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > ~[na:1.8.0_51] > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > ~[na:1.8.0_51] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:842) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:822) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:891) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/1129213153.accept(Unknown > Source) ~[na:na] > at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51] > at > org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_51] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > {code} > This is on a mixed 3.0.8, 3.8-tentative cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12251) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402186#comment-15402186 ] Joel Knighton commented on CASSANDRA-12251: --- [~ifesdjeen] - I actually need to make a superset of these changes for [CASSANDRA-12260] and [CASSANDRA-12313]. It might make sense to just centralize review there and link this one as a duplicate of [CASSANDRA-12260]. > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test > -- > > Key: CASSANDRA-12251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12251 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Alex Petrov > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/whole_list_conditional_test > Failed on CassCI build cassandra-3.8_dtest_upgrade #1 > Relevant error in logs is > {code} > Unexpected error in node1 log, error: > ERROR [InternalResponseStage:2] 2016-07-20 04:58:45,876 > CassandraDaemon.java:217 - Exception in thread > Thread[InternalResponseStage:2,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > ~[na:1.8.0_51] > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > ~[na:1.8.0_51] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:842) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:822) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:891) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/1129213153.accept(Unknown > Source) ~[na:na] > at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51] > at > org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_51] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > {code} > This is on a mixed 3.0.8, 3.8-tentative cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12251) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401718#comment-15401718 ] Alex Petrov commented on CASSANDRA-12251: - Unfortunately, I could not reproduce this locally. However, given the exception it's more or less clear what has happened. It's just much harder to land nodes into the same state. The tasks executed by {{MigrationManager}} are landing in the {{optional}} tasks of scheduled executor [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/MigrationManager.java]. Optional tasks are not being waited for during shutdown or drain. Although even putting the task into the {{nonPeriodicTasks}} executor won't fix the problem since post-flush executor is being shut down before the periodic executor [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L4251]. In order to fix the issue, we need to both move the migration task to non-periodic and make sure that shutdown order is correct. Since mutation stage is already shut down, I left commit log shutdown on ints place and only moved the post flush executor shutdown after the periodic tasks. > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test > -- > > Key: CASSANDRA-12251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12251 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Alex Petrov > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/whole_list_conditional_test > Failed on CassCI build cassandra-3.8_dtest_upgrade #1 > Relevant error in logs is > {code} > Unexpected error in node1 log, error: > ERROR [InternalResponseStage:2] 2016-07-20 04:58:45,876 > CassandraDaemon.java:217 - Exception in thread > Thread[InternalResponseStage:2,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > ~[na:1.8.0_51] > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > ~[na:1.8.0_51] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:842) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:822) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:891) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/1129213153.accept(Unknown > Source) ~[na:na] > at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51] > at > org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_51] > at >
[jira] [Commented] (CASSANDRA-12251) dtest failure in upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test
[ https://issues.apache.org/jira/browse/CASSANDRA-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400277#comment-15400277 ] Joel Knighton commented on CASSANDRA-12251: --- This looks to me like in drain/StorageShutdownHook, the schema stage is not shutdown, so if you have a task submitted to that executor that doesn't execute until after the postflush executor has been terminated in drain, you'll hit this exception. This is a C* fix for sure. > dtest failure in > upgrade_tests.cql_tests.TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x.whole_list_conditional_test > -- > > Key: CASSANDRA-12251 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12251 > Project: Cassandra > Issue Type: Bug >Reporter: Philip Thompson >Assignee: Alex Petrov > Labels: dtest > Attachments: node1.log, node1_debug.log, node1_gc.log, node2.log, > node2_debug.log, node2_gc.log, node3.log, node3_debug.log, node3_gc.log > > > example failure: > http://cassci.datastax.com/job/cassandra-3.8_dtest_upgrade/1/testReport/upgrade_tests.cql_tests/TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x/whole_list_conditional_test > Failed on CassCI build cassandra-3.8_dtest_upgrade #1 > Relevant error in logs is > {code} > Unexpected error in node1 log, error: > ERROR [InternalResponseStage:2] 2016-07-20 04:58:45,876 > CassandraDaemon.java:217 - Exception in thread > Thread[InternalResponseStage:2,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > ~[na:1.8.0_51] > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > ~[na:1.8.0_51] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:842) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.switchMemtableIfCurrent(ColumnFamilyStore.java:822) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:891) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$1(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace$$Lambda$200/1129213153.accept(Unknown > Source) ~[na:na] > at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_51] > at > org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:279) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1271) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1253) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > ~[na:1.8.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_51] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] > {code} > This is on a mixed 3.0.8, 3.8-tentative cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)