[
https://issues.apache.org/jira/browse/CASSANDRA-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005209#comment-13005209
]
Sylvain Lebresne commented on CASSANDRA-2307:
---------------------------------------------
2.5GB, ouch :)
I'll try to have a look to those sstables. In the meantime, if you have only
one node having problem and unless your replication factor is 1, you safest and
probably quicker way to go back to a correct state may well be to replace the
node with the potentially bad disk.
> IndexOutOfBoundsException during compaction
> -------------------------------------------
>
> Key: CASSANDRA-2307
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2307
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.2, 0.7.3
> Environment: Debian 5.0.8, Linux 2.6.26-2-amd64, 4GB of ram assigned
> to Cassandra, JRE 1.6.0_24
> Reporter: Mike Nadeau
>
> We're getting an IndexOutOfBounds exception when compacting.
> Here's the detailed error we get on-screen when running "nodetool -h
> 10.3.133.10 compact":
> Error occured while compacting keyspace test
> java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException
> at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
> at java.util.concurrent.FutureTask.get(Unknown Source)
> at
> org.apache.cassandra.db.CompactionManager.performMajor(CompactionManager.java:186)
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1678)
> at
> org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1248)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown
> Source)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown
> Source)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown
> Source)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
> at sun.rmi.transport.Transport$1.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Unknown Source)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Unknown Source)
> at java.nio.HeapByteBuffer.getInt(Unknown Source)
> at
> org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
> at
> org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
> at
> org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
> at
> org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
> at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:94)
> at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
> at
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217)
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> ... 3 more
> And here's the error I'm getting in my log file:
> ERROR [CompactionExecutor:1] 2011-03-09 19:16:52,299
> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
> Thread[CompactionExecutor:1,1,main]
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Unknown Source)
> at java.nio.HeapByteBuffer.getInt(Unknown Source)
> at
> org.apache.cassandra.db.DeletedColumn.getLocalDeletionTime(DeletedColumn.java:57)
> at
> org.apache.cassandra.db.ColumnFamilyStore.removeDeletedStandard(ColumnFamilyStore.java:822)
> at
> org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:809)
> at
> org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:800)
> at org.apache.cassandra.io.PrecompactedRow.<init>(PrecompactedRow.java:94)
> at
> org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:139)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:108)
> at
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:43)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427)
> at
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:217)
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> We run Cassandra 0.7.3, but we had the issue with Cassandra 0.7.2 as well. We
> have 8 machines in the cluster, the error happens only on one machine, at
> least for now.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
