Hi,
I'm running a 6 node Cassandra 1.1.5 cluster on EC2. We have switched to
leveled compaction a couple of weeks ago,
this has been successful. Some days ago 3 of the nodes start to log the
following exception during compaction of
a particular column family:
ERROR [CompactionExecutor:726] 2013-02-11 13:02:26,582
AbstractCassandraDaemon.java (line 135) Exception in thread
Thread[CompactionExecutor:726,1,main]
java.lang.RuntimeException: Last written key
DecoratedKey(84590743047470232854915142878708713938,
31333535333333383530323237303130313030303232313537303030303132393832)
>= current key DecoratedKey(28357704665244162161305918843747894551,
31333430313336313830333831303130313030303230313632303030303036363338)
writing into
/var/cassandra/data/AdServer/EventHistory/Adserver-EventHistory-tmp-he-68638-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:134)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:153)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
at
org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50)
at
org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Compaction does not happen any more for the column family and read
performance gets worse because of the growing
number of data files accessed during reads. Looks like one or more of the
data files are corrupt and have keys
that are stored out of order.
Any help to resolve this situation would be greatly appreciated.
Thanks
Andre
