CASSANDRA-3712 has not been applied to 0.8.X. If I understand the problem correctly the issue is 0.8.10.
You may be able to avoid the race condition by: 1) Isolating the node from the cluster to stop write activity. You can either start the node with the -Dcassandra.join_ring=false JVM option or use nodetool disablethrift and disablegossip to stop writes. Note that this will not stop existing Hinted Handoff sessions which target the node. 2) Flushing all memtables with nodetool flush 3) Running nodetool cleanup Hope that helps. ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 20/05/2012, at 10:20 PM, Boris Yen wrote: > Hi, > > We are currently running 0.8.10 with 4 nodes. We tried to re-balance the > range each node owns by using "nodetool move". After moving the node to the > assigned token, we run the cleanup command, then we saw the exceptions: > > Error occured during cleanup > java.util.concurrent.ExecutionException: java.lang.AssertionError > > at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) > > at java.util.concurrent.FutureTask.get(Unknown Source) > > at > org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:226) > > at > org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1073) > > at > org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1423) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) > > at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) > > at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source) > > at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source) > > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source) > > at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown > Source) > > at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) > > at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source) > > at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source) > > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown > Source) > > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown > Source) > > at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source) > > at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) > > at sun.rmi.transport.Transport$1.run(Unknown Source) > > at java.security.AccessController.doPrivileged(Native Method) > > at sun.rmi.transport.Transport.serviceCall(Unknown Source) > > at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) > > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) > > at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > Caused by: java.lang.AssertionError > > at > org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:762) > > at > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:844) > > at > org.apache.cassandra.db.ColumnFamilyStore.forceBlockingFlush(ColumnFamilyStore.java:849) > > at > org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:928) > > at > org.apache.cassandra.db.compaction.CompactionManager.access$500(CompactionManager.java:66) > > at > org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:205) > > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > > at java.util.concurrent.FutureTask.run(Unknown Source) > > ... 3 more > > > > It seems to be relative to CASSANDRA-3712. However, it only got fixed on 1.0. > I wonder if this will also be fixed on 0.8? One other question, if we do not > run the cleanup, is there any other way that the unnecessary data can get > cleaned? > > > > Regards. > > Boris >
