[
https://issues.apache.org/jira/browse/CASSANDRA-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950774#comment-15950774
]
Alex Petrov commented on CASSANDRA-13229:
-----------------------------------------
bq. If there are more disks than vnode ranges, I think we should fall back to
splitting vnodes over the available disks, otherwise we would leave some disks
unused
But wouldn't this lead to the problems with disk blacklisting and resurrecting
data like Paulo said in [CASSANDRA-6696]?.. If not, we could go with "try
dividing with {{splitOwnedRangesNoPartialRanges}} first, if some disks are
still unutilized, fall back to splitting vnode ranges.
> dtest failure in topology_test.TestTopology.size_estimates_multidc_test
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-13229
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13229
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
> Reporter: Sean McCarthy
> Assignee: Alex Petrov
> Labels: dtest, test-failure
> Fix For: 4.0
>
> Attachments: node1_debug.log, node1_gc.log, node1.log,
> node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log,
> node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/508/testReport/topology_test/TestTopology/size_estimates_multidc_test
> {code}
> Standard Output
> Unexpected error in node1 log, error:
> ERROR [MemtablePostFlush:1] 2017-02-15 16:07:33,837 CassandraDaemon.java:211
> - Exception in thread Thread[MemtablePostFlush:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> at
> org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
> ~[main/:na]
> at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
> ~[main/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
> [main/:na]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
> Source) [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Unexpected error in node1 log, error:
> ERROR [MigrationStage:1] 2017-02-15 16:07:33,853 CassandraDaemon.java:211 -
> Exception in thread Thread[MigrationStage:1,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
> ~[main/:na]
> at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> at
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
> ~[main/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
> [main/:na]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
> Source) [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397)
> ~[main/:na]
> ... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> at
> org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
> ~[main/:na]
> at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
> ~[main/:na]
> ... 5 common frames omitted
> Unexpected error in node1 log, error:
> ERROR [main] 2017-02-15 16:07:33,857 CassandraDaemon.java:663 - Exception
> encountered during startup
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager.announce(MigrationManager.java:384)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager.announceNewKeyspace(MigrationManager.java:176)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.maybeAddKeyspace(StorageService.java:1066)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.maybeAddOrUpdateKeyspace(StorageService.java:1091)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1048)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1043)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:966)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:649)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:364)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:646)
> [main/:na]
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397)
> ~[main/:na]
> ... 12 common frames omitted
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
> Index: 3, Size: 3
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
> ~[main/:na]
> at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> at
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
> ~[main/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
> ~[main/:na]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
> Source) ~[na:na]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397)
> ~[main/:na]
> ... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> at
> org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
> ~[main/:na]
> at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
> ~[main/:na]
> ... 5 common frames omitted
> Unexpected error in node1 log, error:
> ERROR [StorageServiceShutdownHook] 2017-02-15 16:07:35,972
> AbstractCommitLogSegmentManager.java:311 - Failed to force-recycle all
> segments; at least one segment is still in use with dirty CFs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
