[
https://issues.apache.org/jira/browse/CASSANDRA-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-2968:
----------------------------------------
Attachment: 2968.patch
This is actually a pretty stupid bug (not that there is smart bug): the old
NodeId for the local node were read from the system table in reversed order
while they shouldn't. The wrong path was then taken based on that mistake. No
data was lost due to that (i.e, the total value of the counters is preserved),
but non-sensical counter context were created (hence triggering the assertion).
Fixing the root cause is pretty straightforward. Fixing the nonsensical counter
contexts is more subtle, but it is doable up to the fact that the local NodeId
on the node(s) where the assertion is triggered will have to be renewed.
Attaching a patch that does both (fixing root cause and repairing the bad
data). Also add two unit tests, one for the root cause and one to check that
the bad data repair code does what it is supposed to do.
After applying that patch (or upgrading on a release shipping it), you will
(potentially) need to restart the node with the
-Dcassandra.renew_counter_id=true (compaction will still fail if you don't but
with a message saying that you should restart with the startup flag).
> AssertionError during compaction of CF with counter columns
> -----------------------------------------------------------
>
> Key: CASSANDRA-2968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2968
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.2
> Environment: CentOS release 5.6
> Reporter: Taras Puchko
> Assignee: Sylvain Lebresne
> Fix For: 0.8.3
>
> Attachments: 2968.patch, AffiliateActivity-g-147-Data.db,
> AffiliateActivity-g-147-Index.db, AffiliateActivity-g-195-Data.db,
> AffiliateActivity-g-195-Index.db
>
>
> Having upgraded from 0.8.0 to 0.8.2 we ran nodetool compact and got
> Error occured during compaction
> java.util.concurrent.ExecutionException: java.lang.AssertionError
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.db.compaction.CompactionManager.performMajor(CompactionManager.java:277)
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1762)
> at
> org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:1358)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.AssertionError
>
>
>
> at
> org.apache.cassandra.db.context.CounterContext.removeOldShards(CounterContext.java:593)
>
>
> at
> org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:237)
>
>
> at
> org.apache.cassandra.db.CounterColumn.removeOldShards(CounterColumn.java:256)
>
>
> at
> org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:88)
>
>
> at
> org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:140)
>
>
> at
> org.apache.cassandra.db.compaction.CompactionIterator.getReduced(CompactionIterator.java:123)
>
>
> at
> org.apache.cassandra.db.compaction.CompactionIterator.getReduced(CompactionIterator.java:43)
>
>
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:74)
>
>
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>
>
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>
>
> at
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
>
>
> at
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
>
>
> at
> org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:569)
>
>
> at
> org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:506)
>
>
> at
> org.apache.cassandra.db.compaction.CompactionManager$4.call(CompactionManager.java:319)
>
>
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>
>
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>
>
>
> ... 3 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
