Which version of Cassandra has your data been created initially with? A bug in Cassandra 1.1.2 and earlier could cause out-of-order sstables and inter-level overlaps in CFs with Leveled Compaction. Your sstables generated with 1.1.3 and later should not have this issue [1] [2].
In case you have old Leveled-compacted sstables (generated with 1.1.2 or earlier. including 1.0.x) you need to run offline scrub using Cassandra 1.1.4 or later via /bin/sstablescrub command so it'll fix out-of-order sstables and inter-level overlaps caused by previous versions of LCS. You need to take nodes down in order to run offline scrub. > After 3 hours the job is done and there are 11390 compaction tasks pending. > My question: Can these assertions be ignored? Or do I need to worry about > it? They can't be ignored since pending compactions elevate the upper bound on number of disk seeks you need to make to read a row and you don't get the nice guarantees of leveled compaction. Cheers, Omid [1] https://issues.apache.org/jira/browse/CASSANDRA-4411 [2] https://issues.apache.org/jira/browse/CASSANDRA-4321 On Mon, Sep 10, 2012 at 6:37 PM, Rudolf van der Leeden <rudolf.vanderlee...@scoreloop.com> wrote: > Hi, > > I'm getting 5 identical assertions while running 'nodetool cleanup' on a > Cassandra 1.1.4 node with Load=104G and 80m keys. > From system.log : > > ERROR [CompactionExecutor:576] 2012-09-10 11:25:50,265 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[CompactionExecutor:576,1,main] > java.lang.AssertionError > at > org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:214) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:158) > at > org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:531) > at > org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:254) > at > org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:992) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:200) > at > org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50) > at > org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > After 3 hours the job is done and there are 11390 compaction tasks pending. > My question: Can these assertions be ignored? Or do I need to worry about > it? > > Thanks for your help and best regards, > -Rudolf. >
