[
https://issues.apache.org/jira/browse/CASSANDRA-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950698#comment-15950698
]
Marcus Eriksson commented on CASSANDRA-13229:
---------------------------------------------
bq. it's expected that a single vnode range should not span more than 1 disk
If there are more disks than vnode ranges, I think we should fall back to
splitting vnodes over the available disks, otherwise we would leave some disks
unused. This should be very rare in real production scenarios. I would assume
that this is less surprising to a user than the fact that a vnode was split
over several disks? Maybe even do something like avoiding to split vnode ranges
onto multiple ranges if there are less than 16 tokens on the node (reason being
that it is very hard to get a good balance). So, instead of checking if we run
vnodes or single-token, we check if there are less than 16 tokens?
The optimisation we wanted to do in the future was to be able to take a bunch
of vnode ranges offline if the disk backing the ranges fails, but not sure that
is planned anymore.
> dtest failure in topology_test.TestTopology.size_estimates_multidc_test
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-13229
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13229
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
> Reporter: Sean McCarthy
> Assignee: Alex Petrov
> Labels: dtest, test-failure
> Fix For: 4.0
>
> Attachments: node1_debug.log, node1_gc.log, node1.log,
> node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log,
> node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/508/testReport/topology_test/TestTopology/size_estimates_multidc_test
> {code}
> Standard Output
> Unexpected error in node1 log, error:
> ERROR [MemtablePostFlush:1] 2017-02-15 16:07:33,837 CassandraDaemon.java:211
> - Exception in thread Thread[MemtablePostFlush:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> at
> org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
> ~[main/:na]
> at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
> ~[main/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
> [main/:na]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
> Source) [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Unexpected error in node1 log, error:
> ERROR [MigrationStage:1] 2017-02-15 16:07:33,853 CassandraDaemon.java:211 -
> Exception in thread Thread[MigrationStage:1,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
> ~[main/:na]
> at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> at
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
> ~[main/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
> [main/:na]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
> Source) [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397)
> ~[main/:na]
> ... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> at
> org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
> ~[main/:na]
> at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
> ~[main/:na]
> ... 5 common frames omitted
> Unexpected error in node1 log, error:
> ERROR [main] 2017-02-15 16:07:33,857 CassandraDaemon.java:663 - Exception
> encountered during startup
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager.announce(MigrationManager.java:384)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager.announceNewKeyspace(MigrationManager.java:176)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.maybeAddKeyspace(StorageService.java:1066)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.maybeAddOrUpdateKeyspace(StorageService.java:1091)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1048)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1043)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:966)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:649)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:364)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:646)
> [main/:na]
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397)
> ~[main/:na]
> ... 12 common frames omitted
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
> Index: 3, Size: 3
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
> ~[main/:na]
> at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> at
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
> ~[main/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
> ~[main/:na]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
> Source) ~[na:na]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397)
> ~[main/:na]
> ... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> at
> org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
> ~[main/:na]
> at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
> ~[main/:na]
> ... 5 common frames omitted
> Unexpected error in node1 log, error:
> ERROR [StorageServiceShutdownHook] 2017-02-15 16:07:35,972
> AbstractCommitLogSegmentManager.java:311 - Failed to force-recycle all
> segments; at least one segment is still in use with dirty CFs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
