What version of Cassandra are you running? It sounds like the disk failure policy is incorrectly being applied to scrub, which kind of defeats the purpose of it. I'd recommend opening a ticket ( https://issues.apache.org/jira/browse/CASSANDRA) with the information you posted here.
On Fri, Oct 2, 2015 at 3:01 AM, George Sigletos <sigle...@textkernel.nl> wrote: > I'm also facing problems regarding corrupt sstables and also couldn't run > sstablescrub successfully. > > I restarted my nodes with disk failure policy "best_effort", then I run > the "nodetool scrub <keyspace>" > > Once done I removed the corrupt tables manually and started repair > > > On Thu, Oct 1, 2015 at 7:27 PM, John Anderson <son...@gmail.com> wrote: > >> I have a 25 node cluster and we lost power on one of the racks last night >> and now 6 of our nodes will not start up and we are getting the following >> error: >> >> INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is >> enabled to receive remote connections on port: 7199 >> INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - Initializing >> key cache with capacity of 100 MBs. >> INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - Initializing >> row cache with capacity of 0 MBs >> INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - Initializing >> counter cache with capacity of 50 MBs >> INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling >> counter cache save to every 7200 seconds (going to save all keys). >> INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - >> Initializing system.sstable_activity >> INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 SSTableReader.java:478 >> - Opening >> /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 >> (805 bytes) >> ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - >> Exiting forcefully due to file system exception on startup, disk failure >> policy "stop" >> org.apache.cassandra.io.sstable.CorruptSSTableException: >> java.io.EOFException >> at >> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_60] >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_60] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >> Caused by: java.io.EOFException: null >> at >> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) >> ~[na:1.8.0_60] >> at java.io.DataInputStream.readUTF(DataInputStream.java:589) >> ~[na:1.8.0_60] >> at java.io.DataInputStream.readUTF(DataInputStream.java:564) >> ~[na:1.8.0_60] >> at >> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> ... 14 common frames omitted >> >> >> I found some people recommending scrubbing the sstable so I attempted >> that and got the following error: >> >> bin/sstablescrub system sstable_activity -v >> >> >> ERROR 17:26:03 Exiting forcefully due to file system exception on >> startup, disk failure policy "stop" >> org.apache.cassandra.io.sstable.CorruptSSTableException: >> java.io.EOFException >> at >> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_60] >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_60] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >> Caused by: java.io.EOFException: null >> at >> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) >> ~[na:1.8.0_60] >> at java.io.DataInputStream.readUTF(DataInputStream.java:589) >> ~[na:1.8.0_60] >> at java.io.DataInputStream.readUTF(DataInputStream.java:564) >> ~[na:1.8.0_60] >> at >> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> ... 14 common frames omitted >> >> >> Is there a fix for this? >> > > -- Tyler Hobbs DataStax <http://datastax.com/>
