Limited joy I would say :)  No long term damage at least.

 

I ended up deleting (moving to another disk) all the sstables which fixed the 
problem. I ran in to even more problems during repair (detailed in another 
recent email) but it seems to have worked regardless. Just to be safe, I am in 
the process of starting a ‘manual repair’ (copying SSTables from other nodes 
for this particular CF then restarting and running a cleanup + major 
compaction).

 

Any thoughts on what the root cause of this problem could be? It is somewhat 
worrying that a CF can randomly become corrupt bringing down the whole node. 
Cassandras handling of a corrupt CF (regardless of how rare an occurrence) is 
less than elegant. 

 

Dan

 

From: Aaron Morton [mailto:aa...@thelastpickle.com] 
Sent: January-25-11 16:03
To: user@cassandra.apache.org
Subject: Re: Errors During Compaction

 

Dan how did you go with this? More joy, less joy or a continuation of the 
current level of joy?

 

Aaron

 


On 24/01/2011, at 9:38 AM, Dan Hendry <dan.hendry.j...@gmail.com> wrote:

I have run into a strange problem and was hoping for suggestions on how to fix 
it (0.7.0). When compaction occurs on one node for what appears to be one 
specific column family, the following error pops up the Cassandra log. 
Compaction apparently fails and temp files don’t get cleaned up. After a while 
and what seems to be multiple failed compactions on the CF, the node runs out 
of disk space and crashes. Not sure if it is a related problem or a function of 
this being a heavily used column family but after failing to compact, 
compaction restarts on the same CF exacerbating the issue.

 

Problems with this specific node started earlier this weekend when it crashed 
with and OOM error. This is quite surprising since my memtable thresholds and 
GC settings have been tuned to run with quite a bit of overhead during normal 
operation (max heap usage usually <= 10 GB on a 12 GB heap, average usage of 
6-8 GB). I could not find anything abnormal in the logs which would prompt an 
OOM.

 

I will look things over tomorrow and try to provide a bit more information on 
the problem but as a solution, I was going to wipe out all SSTables for this CF 
on this node and then run a repair. Far from ideal, is this a reasonable 
solution?

 

 

ERROR [CompactionExecutor:1] 2011-01-23 14:10:29,855 
AbstractCassandraDaemon.java (line 91) Fatal exception in thread 
Thread[CompactionExecutor:1,1,RMI Runtime]

java.io.IOError: java.io.EOFException: attempted to skip -1983579368 bytes but 
only skipped 0

        at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:78)

        at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:178)

        at 
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:143)

        at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:135)

        at 
org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)

        at 
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)

        at 
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)

        at 
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)

        at 
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)

        at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)

        at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)

        at 
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)

        at 
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)

        at 
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)

        at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)

        at 
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.io.EOFException: attempted to skip -1983579368 bytes but only 
skipped 0

        at 
org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:52)

        at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69)

        ... 20 more

 

Dan Hendry

(403) 660-2297

 

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.872 / Virus Database: 271.1.1/3402 - Release Date: 01/25/11 
02:34:00

Reply via email to