[
https://issues.apache.org/jira/browse/CASSANDRA-12113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405841#comment-15405841
]
Paulo Motta commented on CASSANDRA-12113:
-----------------------------------------
[~serhatd] unfortunately it's not possible to reproduce the problem with the
provided logs. Are you still able to reproduce this? There were a few streaming
problems fixed recently, could you try upgrading to 3.0.8 and check if this
still happens? If you can still reproduce it in that version, it would be nice
if you could attach full debug logs of source and destination nodes for further
investigation. Thanks!
> Cassandra 3.5.0 Repair Error
> ----------------------------
>
> Key: CASSANDRA-12113
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12113
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Production
> Reporter: Serhat Rıfat Demircan
> Assignee: Paulo Motta
>
> I got the following error while repairing nodes with the "nodetool repair"
> command. Error occured on 2 nodes in the cluster which have 9 nodes.
>
> Interesting thing is corrupted sstable is no more exists one of 2 nodes.
> Copied existing one to test cluster and restored table from that sstable. No
> error occured on test cluster.
> {noformat}
> ERROR [StreamReceiveTask:6] 2016-06-16 02:56:47,480
> StreamReceiveTask.java:215 - Error applying streamed data:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
> at
> org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:50)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:372)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.buildIndexesBlocking(SecondaryIndexManager.java:375)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.buildAllIndexesBlocking(SecondaryIndexManager.java:262)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:182)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_91]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> [na:1.8.0_91]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> [na:1.8.0_91]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:365)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 8 common frames omitted
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> Corrupted: /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:367)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 5 common frames omitted
> Caused by: java.io.IOException: Error building row with data deserialized
> from /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db - chunk
> length 65536, data length 85398996.
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:446)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.AssertionError: Memory was freed
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.SerializationHeader.readLocalDeletionTime(SerializationHeader.java:210)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:455)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:433)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 27 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
