[
https://issues.apache.org/jira/browse/CASSANDRA-5890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis resolved CASSANDRA-5890.
---------------------------------------
Resolution: Cannot Reproduce
> Exception: CorruptSSTableException after nightly 'enqueuing flush'
> ------------------------------------------------------------------
>
> Key: CASSANDRA-5890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5890
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Release: 1.2.5
> 2 Cassandra nodes in a cluster
> OS: Ubuntu 12.04.2 LTS
> CQL: 3.0.2
> Reporter: Danh Kieu
>
> The 2 Cassandra nodes have worked well for 2 days. However, after a nightly
> 'enqueuing flush' process, a few column families (in all keyspaces) are
> corrupted and the following error is observed in log file while trying to
> access the CF:
> ERROR [ReadStage:413277] 2013-08-14 10:16:13,322 CassandraDaemon.java (line
> 175) Exception in thread Thread[ReadStage:413277,5,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:65)
> at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
> at
> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
> at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
> at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:380)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:116)
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
> ... 14 more
> ERROR [ReadStage:413292] 2013-08-14 10:16:53,700 CassandraDaemon.java (line
> 175) Exception in thread Thread[ReadStage:413292,5,main]
> java.lang.AssertionError: DecoratedKey(-8619398030348476976,
> 796177616431406d6e75626f2e636f6d) != DecoratedKey(-8430385117828588592,
> 79624079622e636f6d) in
> /var/lib/cassandra/data/newui/user/newui-user-ic-1-Data.db
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:119)
> at
> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
> at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
> at
> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
> at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Here the traces of the 'enqueuing flush' process:
> INFO [OptionalTasks:1] 2013-08-14 04:01:50,586 ColumnFamilyStore.java (line
> 631) Enqueuing flush of Memtable-user@848766585(3130/3130 serialized/live
> bytes, 96 ops)
> INFO [FlushWriter:381] 2013-08-14 04:01:50,632 Memtable.java (line 461)
> Writing Memtable-user@848766585(3130/3130 serialized/live bytes, 96 ops)
> INFO [FlushWriter:381] 2013-08-14 04:01:50,641 Memtable.java (line 495)
> Completed flushing /var/lib/cassandra/data/newui/user/newui-user-ic-1-Data.db
> (1513 bytes) for commitlog position ReplayPosition(segmentId=1373063890581,
> position=9254)
> The issue had happened quite regularly and on a random CF. Please note that
> we don't have a lot of data (max 20 records per CF).
> We've to run 'nodetool scrub' and 'nodetool repair' to get out of troubles.
> Please let me know if this issue is related to a configuration, a bad use of
> Cassandra or a real problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
