I'm running 2.1.9 On Fri, Oct 2, 2015 at 8:40 AM, Tyler Hobbs <ty...@datastax.com> wrote:
> What version of Cassandra are you running? It sounds like the disk > failure policy is incorrectly being applied to scrub, which kind of defeats > the purpose of it. I'd recommend opening a ticket ( > https://issues.apache.org/jira/browse/CASSANDRA) with the information you > posted here. > > On Fri, Oct 2, 2015 at 3:01 AM, George Sigletos <sigle...@textkernel.nl> > wrote: > >> I'm also facing problems regarding corrupt sstables and also couldn't run >> sstablescrub successfully. >> >> I restarted my nodes with disk failure policy "best_effort", then I run >> the "nodetool scrub <keyspace>" >> >> Once done I removed the corrupt tables manually and started repair >> >> >> On Thu, Oct 1, 2015 at 7:27 PM, John Anderson <son...@gmail.com> wrote: >> >>> I have a 25 node cluster and we lost power on one of the racks last >>> night and now 6 of our nodes will not start up and we are getting the >>> following error: >>> >>> INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is >>> enabled to receive remote connections on port: 7199 >>> INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - >>> Initializing key cache with capacity of 100 MBs. >>> INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - >>> Initializing row cache with capacity of 0 MBs >>> INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - >>> Initializing counter cache with capacity of 50 MBs >>> INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling >>> counter cache save to every 7200 seconds (going to save all keys). >>> INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - >>> Initializing system.sstable_activity >>> INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 >>> SSTableReader.java:478 - Opening >>> /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 >>> (805 bytes) >>> ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - >>> Exiting forcefully due to file system exception on startup, disk failure >>> policy "stop" >>> org.apache.cassandra.io.sstable.CorruptSSTableException: >>> java.io.EOFException >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> [na:1.8.0_60] >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> [na:1.8.0_60] >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>> Caused by: java.io.EOFException: null >>> at >>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) >>> ~[na:1.8.0_60] >>> at java.io.DataInputStream.readUTF(DataInputStream.java:589) >>> ~[na:1.8.0_60] >>> at java.io.DataInputStream.readUTF(DataInputStream.java:564) >>> ~[na:1.8.0_60] >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> ... 14 common frames omitted >>> >>> >>> I found some people recommending scrubbing the sstable so I attempted >>> that and got the following error: >>> >>> bin/sstablescrub system sstable_activity -v >>> >>> >>> ERROR 17:26:03 Exiting forcefully due to file system exception on >>> startup, disk failure policy "stop" >>> org.apache.cassandra.io.sstable.CorruptSSTableException: >>> java.io.EOFException >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> [na:1.8.0_60] >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> [na:1.8.0_60] >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>> Caused by: java.io.EOFException: null >>> at >>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) >>> ~[na:1.8.0_60] >>> at java.io.DataInputStream.readUTF(DataInputStream.java:589) >>> ~[na:1.8.0_60] >>> at java.io.DataInputStream.readUTF(DataInputStream.java:564) >>> ~[na:1.8.0_60] >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> ... 14 common frames omitted >>> >>> >>> Is there a fix for this? >>> >> >> > > > -- > Tyler Hobbs > DataStax <http://datastax.com/> >
