I have a nagging memory of reading about issues with virtualization and not actually having durable versions of your data even after an fsync (within the VM). Googling around lead me to this post: http://petercai.com/virtualization-is-bad-for-database-integrity/
It's possible you're hitting this issue, with with the virtualization layer, or with EBS itself. Just a shot in the dark though, other people would likely know much more than I. On Fri, Mar 28, 2014 at 12:50 PM, Russ Lavoie <ussray...@yahoo.com> wrote: > Robert, > > That is what I thought as well. But apparently something is happening. > The only way I can get away with doing this is adding a sleep 60 right > after the nodetool snapshot is executed. I can reproduce this 100% of the > time by not issuing a sleep after nodetool snapshot. > > This is the error. > > ERROR [SSTableBatchOpen:1] 2014-03-28 17:08:14,290 CassandraDaemon.java > (line 191) Exception in thread Thread[SSTableBatchOpen:1,5,main] > org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.EOFException > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108) > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63) > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42) > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:407) > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:198) > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157) > at > org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.io.EOFException > at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) > at java.io.DataInputStream.readUTF(DataInputStream.java:589) > at java.io.DataInputStream.readUTF(DataInputStream.java:564) > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83) > ... 11 more > > > On Friday, March 28, 2014 2:38 PM, Robert Coli <rc...@eventbrite.com> > wrote: > On Fri, Mar 28, 2014 at 12:21 PM, Russ Lavoie <ussray...@yahoo.com>wrote: > > Thank you for your quick response. > > Is there a way to tell when a snapshot is completely done? > > > IIRC, the JMX call blocks until the snapshot completes. It should be done > when nodetool returns. > > =Rob > > > -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
