1. Not necessarily data corruption, but it seems compaction is trying to write data in the wrong order most likely due to a temporary race condition/bug a la #9935, but since the compaction fails your original data is probably safe (you can try running scrub to verify/fix corruptions). 2. This is pretty tricky to reproduce because it will depend on which sstables were picked for compaction at a particular instant, but you could try running a user-defined compaction or scrub on the sstables that contain this key, see CASSANDRA-11337and https://gist.github.com/jeromatron/e238e5795b3e79866b83 3. clone the cassandra repository of your current version, git-cherry-pick the commit of CASSANDRA-9935, ant jar, replace the cassandra jar with the generated SNAPSHOT.jar and restart the node`
2016-06-28 7:55 GMT-03:00 Prakash Chauhan <prakash.chau...@ericsson.com>: > Hello, > > > > Recently we changed compaction strategy fora table to LCS from the default > STCS. While bootstrapping a node , we are getting following Exception in > the logs: > > > > ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java > (line 258) Exception in thread Thread[CompactionExecutor:81,1,main] > > java.lang.RuntimeException: Last written key > DecoratedKey(-2711050270696519088, 623330333a313a35) >= current key > DecoratedKey(-8631371593982690738, 623437393a313a30) writing into > /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db > > at > org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) > > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) > > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167) > > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957 > ColumnFamilyStore.java (line 795) Enqueuing flush of > Memtable-compactions_in_progress@541886922(0/0 serialized/live bytes, 1 > ops) > > > > Questions: > > *1. *Are these exceptions due to data corruption ? > > *2. *I am unable to reproduce the problem. How can I reproduce the > Exception? Is there any specific case when such exceptions are raised? > > *3. *Without reproducing the Exception, how can I test the patch > available at related JIRA : > https://issues.apache.org/jira/browse/CASSANDRA-9935 > > > > > > Thanks, > > Prakash Chauhan. > > > > > > > > > > >
