[
https://issues.apache.org/jira/browse/CASSANDRA-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951175#comment-15951175
]
Paulo Motta commented on CASSANDRA-13229:
-----------------------------------------
Thanks for the feedback Marcus!
bq. But wouldn't this lead to the problems with disk blacklisting and
resurrecting data like Paulo said in CASSANDRA-6696?
There's no actual problem from what Marcus said, we just wanted to keep vnodes
tied to a single disk to be able to take vnodes offline if a specific disk
fails but this hasn't been implemented yet. Furthermore CASSANDRA-10540 will
probably benefit if vnodes are kept in a single disk.
With this said, we should probably keep your approach of falling back to split
evenly if not all disks are used and also raise the minimum number of tokens to
split by vnode from 2 to 8 to ensure a better balance (Marcus suggested 16 but
8 should be good already since there are 24 local ranges with RF=3 and
CASSANDRA-7032 should also improve load balancing). We can probably reconsider
this if we ever implement the optimization of blacklisting specific vnodes when
disks fail.
In your initial approach, I'd just move the fallback out of
{{splitOwnedRanges}} because it's a bit wrong to pass {{dontSplitRanges=true}}
and have your ranges splitted. :-)
> dtest failure in topology_test.TestTopology.size_estimates_multidc_test
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-13229
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13229
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
> Reporter: Sean McCarthy
> Assignee: Alex Petrov
> Labels: dtest, test-failure
> Fix For: 4.0
>
> Attachments: node1_debug.log, node1_gc.log, node1.log,
> node2_debug.log, node2_gc.log, node2.log, node3_debug.log, node3_gc.log,
> node3.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_novnode_dtest/508/testReport/topology_test/TestTopology/size_estimates_multidc_test
> {code}
> Standard Output
> Unexpected error in node1 log, error:
> ERROR [MemtablePostFlush:1] 2017-02-15 16:07:33,837 CassandraDaemon.java:211
> - Exception in thread Thread[MemtablePostFlush:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> at
> org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
> ~[main/:na]
> at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
> ~[main/:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
> [main/:na]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
> Source) [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Unexpected error in node1 log, error:
> ERROR [MigrationStage:1] 2017-02-15 16:07:33,853 CassandraDaemon.java:211 -
> Exception in thread Thread[MigrationStage:1,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
> ~[main/:na]
> at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> at
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
> ~[main/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
> [main/:na]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
> Source) [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397)
> ~[main/:na]
> ... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> at
> org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
> ~[main/:na]
> at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
> ~[main/:na]
> ... 5 common frames omitted
> Unexpected error in node1 log, error:
> ERROR [main] 2017-02-15 16:07:33,857 CassandraDaemon.java:663 - Exception
> encountered during startup
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager.announce(MigrationManager.java:384)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager.announceNewKeyspace(MigrationManager.java:176)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.maybeAddKeyspace(StorageService.java:1066)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.maybeAddOrUpdateKeyspace(StorageService.java:1091)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1048)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1043)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:966)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:649)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
> ~[main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:364)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:646)
> [main/:na]
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397)
> ~[main/:na]
> ... 12 common frames omitted
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
> Index: 3, Size: 3
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:401)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$496(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace$$Lambda$222/1949434065.accept(Unknown
> Source) ~[na:na]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
> at
> org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:284)
> ~[main/:na]
> at
> org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1265)
> ~[main/:na]
> at org.apache.cassandra.schema.Schema.merge(Schema.java:577) ~[main/:na]
> at
> org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:564)
> ~[main/:na]
> at
> org.apache.cassandra.schema.MigrationManager$1.runMayThrow(MigrationManager.java:402)
> ~[main/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$290(NamedThreadFactory.java:81)
> ~[main/:na]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$5/1321203216.run(Unknown
> Source) ~[na:na]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[na:1.8.0_45]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[na:1.8.0_45]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397)
> ~[main/:na]
> ... 16 common frames omitted
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_45]
> at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_45]
> at
> org.apache.cassandra.dht.Splitter.splitOwnedRangesNoPartialRanges(Splitter.java:92)
> ~[main/:na]
> at org.apache.cassandra.dht.Splitter.splitOwnedRanges(Splitter.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.getDiskBoundaries(StorageService.java:5180)
> ~[main/:na]
> at
> org.apache.cassandra.db.Memtable.createFlushRunnables(Memtable.java:312)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:304)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1150)
> ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1115)
> ~[main/:na]
> ... 5 common frames omitted
> Unexpected error in node1 log, error:
> ERROR [StorageServiceShutdownHook] 2017-02-15 16:07:35,972
> AbstractCommitLogSegmentManager.java:311 - Failed to force-recycle all
> segments; at least one segment is still in use with dirty CFs.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
