https://issues.apache.org/jira/browse/CASSANDRA-3482
I restarted the node and the problem has cropped up again. Is it possible to downgrade back to 0.8? Is there any way to convert 'h' version SSTables to the old 'g' version? Any other data file changes to be aware of? Dan -----Original Message----- From: Sylvain Lebresne [mailto:sylv...@datastax.com] Sent: November-10-11 15:23 To: user@cassandra.apache.org Subject: Re: 1.0.2 Assertion Error That would be a bug (as any assertion error would be), likely some race condition. Could you open a ticket? The fact that this block the MemtablePostFlusher is unfortunately related. Restarting the node would fix but we need to make that more solid too. -- Sylvain On Thu, Nov 10, 2011 at 9:04 PM, Dan Hendry <dan.hendry.j...@gmail.com> wrote: > Just happened again, seems to be with the same column family (at least on a > flusher thread for which the last activity was flushing a memtable for that > CF). > > > > It also looks like MemtablePostFlusher tasks blocked (and not getting > cleared) as evidenced by tpstats: > > > > Pool Name Active Pending Completed Blocked All > time blocked > > MemtablePostFlusher 1 18 16 > 0 0 > > > > Dan > > > > From: Dan Hendry [mailto:dan.hendry.j...@gmail.com] > Sent: November-10-11 14:49 > To: 'user@cassandra.apache.org' > Subject: 1.0.2 Assertion Error > > > > Just saw this weird assertion after upgrading one of my nodes from 0.8.6 to > 1.0.2 (its been running fine for a few hours now): > > > > INFO [FlushWriter:9] 2011-11-10 13:08:58,882 Memtable.java (line 237) > Writing Memtable-Data@1388955390(25676955/430716097 serialized/live bytes, > 478913 ops) > > ERROR [FlushWriter:9] 2011-11-10 13:08:59,513 AbstractCassandraDaemon.java > (line 133) Fatal exception in thread Thread[FlushWriter:9,5,main] > > java.lang.AssertionError: CF size changed during serialization: was 4 > initially but 3 written > > at > org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFam ilySerializer.java:94) > > at > org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFa milySerializer.java:112) > > at > org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:177) > > at > org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:264) > > at org.apache.cassandra.db.Memtable.access$400(Memtable.java:47) > > at org.apache.cassandra.db.Memtable$4.runMayThrow(Memtable.java:289) > > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) > > at java.lang.Thread.run(Thread.java:662) > > > > > > After taking a quick peek at the code, it looks like the numbers (4 > initially but 3 written) refer to the number of columns (in the memtable?). > Given the byte size of the memtable being flushed, there are *certainly* > more than 4 columns. Besides this error, there are no other unexpected or > unusual log entries and the node seems to be behaving normally. > > > > Thoughts? > > > > Dan Hendry No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.920 / Virus Database: 271.1.1/4007 - Release Date: 11/10/11 02:34:00
