Hi there, I have a cluster of three nodes running Cassandra 1.2.0
I received alerts from my monitoring, and then discovered this huge spike in cluster latency: https://dl.dropbox.com/u/3444322/Screen%20Shot%202013-02-12%20at%205.07.49%20PM.png Investigating what is going on, there is no load on any node, iostat shows nothing more than idle operations, and I've restarted all nodes. In the system.log I keep noticing this on ONE node only: ==> /var/log/cassandra/system.log <== ERROR [ReadStage:563] 2013-02-12 04:31:30,013 CassandraDaemon.java (line 133) Exception in thread Thread[ReadStage:563,5,main] java.lang.AssertionError: Interval min > max at org.apache.cassandra.utils.IntervalTree$IntervalNode.<init>(IntervalTree.java:250) at org.apache.cassandra.utils.IntervalTree.<init>(IntervalTree.java:72) at org.apache.cassandra.utils.IntervalTree.build(IntervalTree.java:81) at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:175) at org.apache.cassandra.db.AbstractThreadUnsafeSortedColumns.delete(AbstractThreadUnsafeSortedColumns.java:40) at org.apache.cassandra.db.AbstractColumnContainer.delete(AbstractColumnContainer.java:51) at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:224) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:182) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143) at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:86) at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:45) at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:134) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:286) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1362) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1134) at org.apache.cassandra.db.Table.getRow(Table.java:348) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1048) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1506) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) -- Cheers, Drew Broadley *Broadley Speaking :)* e: [email protected] p: +64 (0)21 519 711 m: P O Box 488, Wellington, New Zealand w: http://blog.drew.broadley.org.nz/ ln: http://nz.linkedin.com/in/drewbroadley
