Looks like it is harmless -- Scrub would write a zero-length row when tombstones expire and there is nothing left, instead of writing no row at all. Fix attached to the jira ticket.
On Tue, Mar 8, 2011 at 8:58 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > It *may* be harmless depending on where those zero-length rows are > coming from. I've added asserts to 0.7 branch that fire if we attempt > to write a zero-length row, so if the bug is still present in 0.7.3+ > that should catch it. > > On Tue, Mar 8, 2011 at 7:31 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >> alienth on irc is reporting the same error. His path was 0.6.8 to >> 0.7.1 to 0.7.3. >> >> It's probably a bug in scrub. If we can get an sstable exhibiting the >> problem posted here or on Jira that would help troubleshoot. >> >> On Tue, Mar 8, 2011 at 10:31 AM, Karl Hiramoto <k...@hiramoto.org> wrote: >>> On 08/03/2011 17:09, Jonathan Ellis wrote: >>>> >>>> No. >>>> >>>> What is the history of your cluster? >>> >>> It started out as 0.7.0 - RC3 And I've upgraded 0.7.0, 0.7.1, 0.7.2, >>> 0.7.3 within a few days after each was released. >>> >>> I have 6 nodes about 10GB of data each RF=2. Only one CF every >>> row/column has a TTL of 24 hours. >>> I do a staggered repair/compact/cleanup across every node in a cronjob. >>> >>> >>> After upgrading to 0.7.3 I had a lot of nodes crashing due to OOM. I >>> reduced the key cache from the default 200000 to 1000 and increased the heap >>> size from 8GB to 12GB and the OOM crashes went away. >>> >>> >>> Anyway to fix this without throwing away all the data? >>> >>> Since i only keep data 24 hours, I could insert into two CF for the next 24 >>> hours than after only valid data in new CF remove the old CF. >>> >>> >>> >>>> On Tue, Mar 8, 2011 at 5:34 AM, Karl Hiramoto<k...@hiramoto.org> wrote: >>>>> >>>>> I have 1000's of these in the log is this normal? >>>>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be longer >>>>> than >>>>> entire row size >>>>> at >>>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117) >>>>> at >>>>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:590) >>>>> at >>>>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) >>>>> at >>>>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) >>>>> at >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>>>> at java.lang.Thread.run(Thread.java:636) >>>>> Caused by: java.io.EOFException: bloom filter claims to be longer than >>>>> entire row size >>>>> at >>>>> >>>>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:113) >>>>> at >>>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87) >>>>> ... 8 more >>>>> WARN [CompactionExecutor:1] 2011-03-08 11:32:35,615 >>>>> CompactionManager.java >>>>> (line 625) Row is unreadable; skipping to next >>>>> WARN [CompactionExecutor:1] 2011-03-08 11:32:35,615 >>>>> CompactionManager.java >>>>> (line 599) Non-fatal error reading row (stacktrace follows) >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be longer >>>>> than >>>>> entire row size >>>>> at >>>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117) >>>>> at >>>>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:590) >>>>> at >>>>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) >>>>> at >>>>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) >>>>> at >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>>>> at java.lang.Thread.run(Thread.java:636) >>>>> Caused by: java.io.EOFException: bloom filter claims to be longer than >>>>> entire row size >>>>> at >>>>> >>>>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:113) >>>>> at >>>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87) >>>>> ... 8 more >>>>> WARN [CompactionExecutor:1] 2011-03-08 11:32:35,615 >>>>> CompactionManager.java >>>>> (line 625) Row is unreadable; skipping to next >>>>> WARN [CompactionExecutor:1] 2011-03-08 11:32:35,615 >>>>> CompactionManager.java >>>>> (line 599) Non-fatal error reading row (stacktrace follows) >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be longer >>>>> than >>>>> entire row size >>>>> at >>>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117) >>>>> at >>>>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:590) >>>>> at >>>>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) >>>>> at >>>>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) >>>>> at >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>>>> at java.lang.Thread.run(Thread.java:636) >>>>> Caused by: java.io.EOFException: bloom filter claims to be longer than >>>>> entire row size >>>>> at >>>>> >>>>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:113) >>>>> at org.apa >>>>> >>>> >>>> >>> >>> >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of DataStax, the source for professional Cassandra support >> http://www.datastax.com >> > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
