[jira] [Commented] (CASSANDRA-10612) Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted
[
https://issues.apache.org/jira/browse/CASSANDRA-10612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205221#comment-15205221
]
Russ Hatch commented on CASSANDRA-10612:
not completely sure [~mambocab] . The impression I had from this was that it
was maybe benign to begin with (and so thought maybe I coded the tests to
ignore it, but that doesn't seem to be the case).
> Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted
> --
>
> Key: CASSANDRA-10612
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10612
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Russ Hatch
> Labels: dtest
> Fix For: 3.0.x
>
>
> The following tests in the upgrade_through_versions dtest suite fail:
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_HEAD.rolling_upgrade_test
> See this report:
> http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/
> They fail with the following error:
> {code}
> A subprocess has terminated early. Subprocess statuses: Process-41 (is_alive:
> True), Process-42 (is_alive: False), Process-43 (is_alive: True), Process-44
> (is_alive: False), attempting to terminate remaining subprocesses now.
> {code}
> and with logs that look like this:
> {code}
> Unexpected error in node1 node log: ['ERROR [SecondaryIndexManagement:1]
> 2015-10-27 00:06:52,335 CassandraDaemon.java:195 - Exception in thread
> Thread[SecondaryIndexManagement:1,5,main] java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.buildBlocking(CassandraIndex.java:688)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.lambda$getBuildIndexTask$206(CassandraIndex.java:658)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex$$Lambda$151/1841229245.call(Unknown
> Source) ~[na:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf,
> 578160/{code}1663620)bytes
> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_51]
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:364)
> ~[main/:na]
> ... 7 common frames omitted Caused by:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at
> org.apache.cassandra.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1269)
> ~[main/:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> ... 4 common frames omitted', 'ERROR [HintsDispatcher:2] 2015-10-27
> 00:08:48,520 CassandraDaemon.java:195 - Exception in thread
> Thread[HintsDispatcher:2,1,main]', 'ERROR [HintsDispatcher:2]
[jira] [Commented] (CASSANDRA-10612) Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted
[
https://issues.apache.org/jira/browse/CASSANDRA-10612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204673#comment-15204673
]
Jim Witschey commented on CASSANDRA-10612:
--
bq. This does not appear to be manifesting any longer.
Any idea why?
> Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted
> --
>
> Key: CASSANDRA-10612
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10612
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Russ Hatch
> Labels: dtest
> Fix For: 3.0.x
>
>
> The following tests in the upgrade_through_versions dtest suite fail:
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_HEAD.rolling_upgrade_test
> See this report:
> http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/
> They fail with the following error:
> {code}
> A subprocess has terminated early. Subprocess statuses: Process-41 (is_alive:
> True), Process-42 (is_alive: False), Process-43 (is_alive: True), Process-44
> (is_alive: False), attempting to terminate remaining subprocesses now.
> {code}
> and with logs that look like this:
> {code}
> Unexpected error in node1 node log: ['ERROR [SecondaryIndexManagement:1]
> 2015-10-27 00:06:52,335 CassandraDaemon.java:195 - Exception in thread
> Thread[SecondaryIndexManagement:1,5,main] java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.buildBlocking(CassandraIndex.java:688)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.lambda$getBuildIndexTask$206(CassandraIndex.java:658)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex$$Lambda$151/1841229245.call(Unknown
> Source) ~[na:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf,
> 578160/{code}1663620)bytes
> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_51]
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:364)
> ~[main/:na]
> ... 7 common frames omitted Caused by:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at
> org.apache.cassandra.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1269)
> ~[main/:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> ... 4 common frames omitted', 'ERROR [HintsDispatcher:2] 2015-10-27
> 00:08:48,520 CassandraDaemon.java:195 - Exception in thread
> Thread[HintsDispatcher:2,1,main]', 'ERROR [HintsDispatcher:2] 2015-10-27
> 00:11:58,336 CassandraDaemon.java:195 - Exception in thread
> Thread[HintsDispatcher:2,1,main]']
> {code}
--
This
[jira] [Commented] (CASSANDRA-10612) Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted
[
https://issues.apache.org/jira/browse/CASSANDRA-10612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196312#comment-15196312
]
Russ Hatch commented on CASSANDRA-10612:
This does not appear to be manifesting any longer.
> Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted
> --
>
> Key: CASSANDRA-10612
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10612
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Russ Hatch
> Labels: dtest
> Fix For: 3.0.x
>
>
> The following tests in the upgrade_through_versions dtest suite fail:
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_HEAD.rolling_upgrade_test
> See this report:
> http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/
> They fail with the following error:
> {code}
> A subprocess has terminated early. Subprocess statuses: Process-41 (is_alive:
> True), Process-42 (is_alive: False), Process-43 (is_alive: True), Process-44
> (is_alive: False), attempting to terminate remaining subprocesses now.
> {code}
> and with logs that look like this:
> {code}
> Unexpected error in node1 node log: ['ERROR [SecondaryIndexManagement:1]
> 2015-10-27 00:06:52,335 CassandraDaemon.java:195 - Exception in thread
> Thread[SecondaryIndexManagement:1,5,main] java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.buildBlocking(CassandraIndex.java:688)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.lambda$getBuildIndexTask$206(CassandraIndex.java:658)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex$$Lambda$151/1841229245.call(Unknown
> Source) ~[na:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf,
> 578160/{code}1663620)bytes
> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_51]
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:364)
> ~[main/:na]
> ... 7 common frames omitted Caused by:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at
> org.apache.cassandra.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1269)
> ~[main/:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> ... 4 common frames omitted', 'ERROR [HintsDispatcher:2] 2015-10-27
> 00:08:48,520 CassandraDaemon.java:195 - Exception in thread
> Thread[HintsDispatcher:2,1,main]', 'ERROR [HintsDispatcher:2] 2015-10-27
> 00:11:58,336 CassandraDaemon.java:195 - Exception in thread
> Thread[HintsDispatcher:2,1,main]']
> {code}
--
This message was sent by
[jira] [Commented] (CASSANDRA-10612) Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted
[
https://issues.apache.org/jira/browse/CASSANDRA-10612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153387#comment-15153387
]
Russ Hatch commented on CASSANDRA-10612:
At one point I asked and I think [~brandon.williams] said it could be feasible
that this is a normal failure when a node is stopped during compaction.
I don't remember at this point if it's causing nodes to stop abnormally, or
just log noise.
One option on the dtest side might be to try and wait for compactions to
complete before shutting down nodes -- then again, this would likely diverge
from typical user behavior during an upgrade, so there's an argument to be made
against that as well.
> Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted
> --
>
> Key: CASSANDRA-10612
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10612
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Russ Hatch
> Labels: dtest
> Fix For: 3.0.x
>
>
> The following tests in the upgrade_through_versions dtest suite fail:
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_HEAD.rolling_upgrade_test
> See this report:
> http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/
> They fail with the following error:
> {code}
> A subprocess has terminated early. Subprocess statuses: Process-41 (is_alive:
> True), Process-42 (is_alive: False), Process-43 (is_alive: True), Process-44
> (is_alive: False), attempting to terminate remaining subprocesses now.
> {code}
> and with logs that look like this:
> {code}
> Unexpected error in node1 node log: ['ERROR [SecondaryIndexManagement:1]
> 2015-10-27 00:06:52,335 CassandraDaemon.java:195 - Exception in thread
> Thread[SecondaryIndexManagement:1,5,main] java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.buildBlocking(CassandraIndex.java:688)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.lambda$getBuildIndexTask$206(CassandraIndex.java:658)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex$$Lambda$151/1841229245.call(Unknown
> Source) ~[na:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf,
> 578160/{code}1663620)bytes
> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_51]
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:364)
> ~[main/:na]
> ... 7 common frames omitted Caused by:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at
> org.apache.cassandra.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1269)
>
