[
https://issues.apache.org/jira/browse/CASSANDRA-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis resolved CASSANDRA-6929.
---------------------------------------
Resolution: Not A Problem
Unless you can reproduce from specific Cassandra requests, this is most likely
a hardware problem (ram, disk, raid controller are suspects).
FTR, scrub will rebuild your index for you.
> Corrupted Index File: read 8599 but expected 8600 chunks.
> ---------------------------------------------------------
>
> Key: CASSANDRA-6929
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6929
> Project: Cassandra
> Issue Type: Bug
> Reporter: Donald Smith
>
> I have a 3 node cassandra cluster running 2.0.6 (we started at 2.0.1). It has
> several terabytes of data. We've been seeing exceptions in system.log like
> "Corrupted Index File ... read 21478 but expected 21479 chunks."
> Here's a stack trace from one server:
> {noformat}
> INFO [CompactionExecutor:9109] 2014-03-24 06:55:28,148
> ColumnFamilyStore.java (line 785) Enqueuing flush of
> Memtable-compactions_in_progress@1299803435(0/0 serialized/live bytes, 1 ops)
> INFO [FlushWriter:496] 2014-03-24 06:55:28,148 Memtable.java (line 331)
> Writing Memtable-compactions_in_progress@1299803435(0/0 serialized/live
> bytes, 1 ops)
> INFO [FlushWriter:496] 2014-03-24 06:55:28,299 Memtable.java (line 371)
> Completed flushing
> /mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12862-Data.db
> (42 bytes) for commitlog position ReplayPosition(segmentId=1395195644764,
> position=17842243)
> INFO [CompactionExecutor:9142] 2014-03-24 06:55:28,299 CompactionTask.java
> (line 115) Compacting
> [SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12861-Data.db'),
>
> SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12860-Data.db'),
>
> SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12858-Data.db'),
>
> SSTableReader(path='/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12862-Data.db')]
> ERROR [CompactionExecutor:9109] 2014-03-24 06:55:28,302 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:9109,1,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
> read 21478 but expected 21479 chunks.
> at
> org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:152)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
> at
> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:330)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
> read 21478 but expected 21479 chunks.
> ... 16 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(Unknown Source)
> at java.io.DataInputStream.readLong(Unknown Source)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:146)
> ... 15 more
> INFO [CompactionExecutor:9142] 2014-03-24 06:55:28,739 CompactionTask.java
> (line 275) Compacted 4 sstables to
> [/mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-12863,].
> 571 bytes to 42 (~7% of original) in 439ms = 0.000091MB/s. 4 total
> partitions merged to 1. Partition merge counts were {2:2, }
> {noformat}
> Here's another example:
> {noformat}
> INFO [CompactionExecutor:9566] 2014-03-25 06:32:02,234
> ColumnFamilyStore.java (line 785) Enqueuing flush of
> Memtable-compactions_in_progress@1216289160(0/0 serialized/live bytes, 1 ops)
> INFO [FlushWriter:474] 2014-03-25 06:32:02,234 Memtable.java (line 331)
> Writing Memtable-compactions_in_progress@1216289160(0/0 serialized/live
> bytes, 1 ops)
> INFO [FlushWriter:474] 2014-03-25 06:32:02,445 Memtable.java (line 371)
> Completed flushing
> /mnt/cassandra-storage/data/system/compactions_in_progress/system-compactions_in_progress-jb-33701-Data.db
> (42 bytes) for commitlog position ReplayPosition(segmentId=1395185650610,
> position=4461144)
> ERROR [CompactionExecutor:9566] 2014-03-25 06:32:02,447 CassandraDaemon.java
> (line 196) Exception in thread Thread[CompactionExecutor:9566,1,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_report_details/as_reports-data_report_details-jb-23168-CompressionInfo.db:
> read 8599 but expected 8600 chunks.
> at
> org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:152)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
> at
> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:330)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_report_details/as_reports-data_report_details-jb-23168-CompressionInfo.db:
> read 8599 but expected 8600 chunks.
> ... 16 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readFully(Unknown Source)
> at java.io.DataInputStream.readLong(Unknown Source)
> at
> org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:146)
> ... 15 more
> {noformat}
> Here are multiple examples (f is an alias for "grep -i"):
> {noformat}
> cass1 /var/log/cassandra> f exception system.log | f corrupted | f chunks
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-51913-CompressionInfo.db:
> read 18848 but expected 18849 chunks.
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-51913-CompressionInfo.db:
> read 18848 but expected 18849 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-53537-CompressionInfo.db:
> read 19930 but expected 19931 chunks.
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-53537-CompressionInfo.db:
> read 19930 but expected 19931 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54063-CompressionInfo.db:
> read 19607 but expected 19608 chunks.
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54063-CompressionInfo.db:
> read 19607 but expected 19608 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54562-CompressionInfo.db:
> read 18425 but expected 18426 chunks.
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54562-CompressionInfo.db:
> read 18425 but expected 18426 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54767-CompressionInfo.db:
> read 18648 but expected 18649 chunks.
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54767-CompressionInfo.db:
> read 18648 but expected 18649 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54920-CompressionInfo.db:
> read 18882 but expected 18883 chunks.
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-54920-CompressionInfo.db:
> read 18882 but expected 18883 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55077-CompressionInfo.db:
> read 18949 but expected 18950 chunks.
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55077-CompressionInfo.db:
> read 18949 but expected 18950 chunks.
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
> read 21478 but expected 21479 chunks.
> Caused by: java.io.IOException: Corrupted Index File
> /mnt/cassandra-storage/data/as_reports/data_hierarchy_details/as_reports-data_hierarchy_details-jb-55104-CompressionInfo.db:
> read 21478 but expected 21479 chunks.
> {noformat}
> Let me know if you need more info. I wouldn't know how to reproduce this from
> a simple example.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
