[
https://issues.apache.org/jira/browse/CASSANDRA-10612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153387#comment-15153387
]
Russ Hatch commented on CASSANDRA-10612:
----------------------------------------
At one point I asked and I think [~brandon.williams] said it could be feasible
that this is a normal failure when a node is stopped during compaction.
I don't remember at this point if it's causing nodes to stop abnormally, or
just log noise.
One option on the dtest side might be to try and wait for compactions to
complete before shutting down nodes -- then again, this would likely diverge
from typical user behavior during an upgrade, so there's an argument to be made
against that as well.
> Protocol v3 upgrade tests on 2.1->3.0 path fail when compaction is interrupted
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-10612
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10612
> Project: Cassandra
> Issue Type: Sub-task
> Reporter: Jim Witschey
> Assignee: Russ Hatch
> Labels: dtest
> Fix For: 3.0.x
>
>
> The following tests in the upgrade_through_versions dtest suite fail:
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestRandomPartitionerUpgrade.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgradeThroughVersions.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_2_1_latest_tag_to_cassandra_3_0_HEAD.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_latest_tag.rolling_upgrade_with_internode_ssl_test
> *
> upgrade_through_versions_test.py:TestUpgrade_from_cassandra_2_1_HEAD_to_cassandra_3_0_HEAD.rolling_upgrade_test
> See this report:
> http://cassci.datastax.com/view/Upgrades/job/cassandra_upgrade_2.1_to_3.0_proto_v3/10/testReport/
> They fail with the following error:
> {code}
> A subprocess has terminated early. Subprocess statuses: Process-41 (is_alive:
> True), Process-42 (is_alive: False), Process-43 (is_alive: True), Process-44
> (is_alive: False), attempting to terminate remaining subprocesses now.
> {code}
> and with logs that look like this:
> {code}
> Unexpected error in node1 node log: ['ERROR [SecondaryIndexManagement:1]
> 2015-10-27 00:06:52,335 CassandraDaemon.java:195 - Exception in thread
> Thread[SecondaryIndexManagement:1,5,main] java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:368)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.buildBlocking(CassandraIndex.java:688)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.lambda$getBuildIndexTask$206(CassandraIndex.java:658)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex$$Lambda$151/1841229245.call(Unknown
> Source) ~[na:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_51]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_51]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51] Caused by:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf,
> 578160/{code}1663620)bytes
> at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_51]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_51]
> at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:364)
> ~[main/:na]
> ... 7 common frames omitted Caused by:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Secondary index
> build@41202370-7c3e-11e5-9331-6bb6e58f8b1b(upgrade, cf, 578160/1663620)bytes
> at
> org.apache.cassandra.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:67)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1269)
> ~[main/:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_51]
> ... 4 common frames omitted', 'ERROR [HintsDispatcher:2] 2015-10-27
> 00:08:48,520 CassandraDaemon.java:195 - Exception in thread
> Thread[HintsDispatcher:2,1,main]', 'ERROR [HintsDispatcher:2] 2015-10-27
> 00:11:58,336 CassandraDaemon.java:195 - Exception in thread
> Thread[HintsDispatcher:2,1,main]']
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
