[
https://issues.apache.org/jira/browse/CASSANDRA-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitrij Koniajev updated CASSANDRA-6845:
----------------------------------------
Description:
Originally we had Cassandra cluster with the following configuration:
{quote}
Initial tokens:
10.1.11.51: -9223372036854775808
10.1.11.52: 0
10.2.11.51: -9223372036854775807
10.2.11.52: 1
Endpoint snitch: PropertyFileSnitch
10.1.11.51=DC1:RAC1
10.1.11.52=DC1:RAC1
10.2.11.51=DC2:RAC1
10.2.11.52=DC2:RAC1
{quote}
We have created keyspace with network topology strategy and replication factor
2:
{code}
CREATE KEYSPACE our_keyspace WITH replication = {
'class': 'NetworkTopologyStrategy',
'DC2': '2',
'DC1': '2'
};
{code}
All column families have been created with SizeTiered compaction strategy.
Аfter a while column families have been migrated to LeveledCompactionStrategy
and then replication factor was decreased to 1.
We started cleanup, but on non-seed nodes (10.1.11.52,10.2.11.52) it failed
with assertion error. Assertion stacktrace was the same as described in
[CASSANDRA-6774|https://issues.apache.org/jira/browse/CASSANDRA-6774] issue.
Then we've stopped all compactions, started cleanup again and got another
exception:
{quote}
Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 1, Size: 1
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:227)
at
org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:265)
at
org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
at
org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.cassandra.db.compaction.CompactionManager.needsCleanup(CompactionManager.java:502)
at
org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:540)
at
org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:274)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
{quote}
It seems there is typo in *needsCleanup* method of *CompactionManager* class
and the supplied patch should fix the issue.
Get method on the line *502* (*CompactionManager.java*) throws exception,
because of the wrong comparison on the line *496*: there should be
*sortedRanges* size check, not *ownedRanges*.
was:
Originally we had Cassandra cluster with the following configuration:
{quote}
Initial tokens:
10.1.11.51: -9223372036854775808
10.1.11.52: 0
10.2.11.51: -9223372036854775807
10.2.11.52: 1
Endpoint snitch: PropertyFileSnitch
10.1.11.51=DC1:RAC1
10.1.11.52=DC1:RAC1
10.2.11.51=DC2:RAC1
10.2.11.52=DC2:RAC1
{quote}
We have created keyspace with network topology strategy and replication factor
2:
{quote}
CREATE KEYSPACE our_keyspace WITH replication = {
'class': 'NetworkTopologyStrategy',
'DC2': '2',
'DC1': '2'
};
{quote}
All column families have been created with SizeTiered compaction strategy.
Аfter a while column families have been migrated to LeveledCompactionStrategy
and then replication factor was decreased to 1.
We started cleanup, but on non-seed nodes (10.1.11.52,10.2.11.52) it failed
with assertion error. Assertion stacktrace was the same as described in
[CASSANDRA-6774|https://issues.apache.org/jira/browse/CASSANDRA-6774] issue.
Then we've stopped all compactions, started cleanup again and got another
exception:
{quote}
Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
Index: 1, Size: 1
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:227)
at
org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:265)
at
org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
at
org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.cassandra.db.compaction.CompactionManager.needsCleanup(CompactionManager.java:502)
at
org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:540)
at
org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:274)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
{quote}
It seems there is typo in *needsCleanup* method of *CompactionManager* class
and the supplied patch should fix the issue.
Get method on the line *502* (*CompactionManager.java*) throws exception,
because of the wrong comparison on the line *496*: there should be
*sortedRanges* size check, not *ownedRanges*.
> Cleanup fails with IndexOutOfBoundsException exception
> ------------------------------------------------------
>
> Key: CASSANDRA-6845
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6845
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: CentOS Linux 2.6.32-431.el6.x86_64
> Datastax Cassandra 2.0.5-1
> Reporter: Dmitrij Koniajev
> Assignee: Dmitrij Koniajev
> Attachments: 0001-fixed-bug-in-CompactionManager.needsCleanup.patch
>
>
> Originally we had Cassandra cluster with the following configuration:
> {quote}
> Initial tokens:
> 10.1.11.51: -9223372036854775808
> 10.1.11.52: 0
> 10.2.11.51: -9223372036854775807
> 10.2.11.52: 1
> Endpoint snitch: PropertyFileSnitch
> 10.1.11.51=DC1:RAC1
> 10.1.11.52=DC1:RAC1
> 10.2.11.51=DC2:RAC1
> 10.2.11.52=DC2:RAC1
> {quote}
> We have created keyspace with network topology strategy and replication
> factor 2:
> {code}
> CREATE KEYSPACE our_keyspace WITH replication = {
> 'class': 'NetworkTopologyStrategy',
> 'DC2': '2',
> 'DC1': '2'
> };
> {code}
> All column families have been created with SizeTiered compaction strategy.
> Аfter a while column families have been migrated to LeveledCompactionStrategy
> and then replication factor was decreased to 1.
> We started cleanup, but on non-seed nodes (10.1.11.52,10.2.11.52) it failed
> with assertion error. Assertion stacktrace was the same as described in
> [CASSANDRA-6774|https://issues.apache.org/jira/browse/CASSANDRA-6774] issue.
> Then we've stopped all compactions, started cleanup again and got another
> exception:
> {quote}
> Error occurred during cleanup
> java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException:
> Index: 1, Size: 1
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:227)
> at
> org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:265)
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115)
> at
> org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2152)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
> at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at sun.rmi.transport.Transport$1.run(Transport.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.cassandra.db.compaction.CompactionManager.needsCleanup(CompactionManager.java:502)
> at
> org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:540)
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:62)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:274)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:222)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ... 3 more
> {quote}
> It seems there is typo in *needsCleanup* method of *CompactionManager* class
> and the supplied patch should fix the issue.
> Get method on the line *502* (*CompactionManager.java*) throws exception,
> because of the wrong comparison on the line *496*: there should be
> *sortedRanges* size check, not *ownedRanges*.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
