Re: RuntimeException during leveled compaction

2013-02-16 Thread aaron morton
That sounds like something wrong with the way the rows are merged during compaction then. Can you run the compaction with DEBUG logging and raise a ticket? You may want to do this with the node not in the ring. Five minutes after it starts it will run pending compactions, so if there if

Re: RuntimeException during leveled compaction

2013-02-12 Thread aaron morton
snapshot all nodes so you have a backup: nodetool snapshot -t corrupt run nodetool scrub on the errant CF. Look for messages such as: Out of order row detected… %d out of order rows found while scrubbing %s; Those have been written (in order) to a new sstable (%s) In the logs. Cheers

Re: RuntimeException during leveled compaction

2013-02-12 Thread Andre Sprenger
Aaron, thanks for your help. I ran 'nodetool scrub' and it finished after a couple of hours. But there are no infos about out of order rows in the logs and the compaction on the column family still raises the same exception. With the row key I could identify some of the errant SSTables and

RuntimeException during leveled compaction

2013-02-11 Thread Andre Sprenger
Hi, I'm running a 6 node Cassandra 1.1.5 cluster on EC2. We have switched to leveled compaction a couple of weeks ago, this has been successful. Some days ago 3 of the nodes start to log the following exception during compaction of a particular column family: ERROR [CompactionExecutor:726]