[
https://issues.apache.org/jira/browse/CASSANDRA-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406000#comment-15406000
]
Paulo Motta commented on CASSANDRA-12255:
-----------------------------------------
Do you still have the logs? Do you see any other streaming errors before this?
Was this a one-off error or can you reproduce it reliably?
Don't worry as there was no data corruption, what happened is that an sstable
was closed while it was being processed, so there's problem some race going on
there.
This looks somewhat similar to CASSANDRA-12113.
> 3.5.0 error while rebuilding: java.lang.AssertionError: Attempted to use
> closed region
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-12255
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12255
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Production
> Reporter: Serhat Rıfat Demircan
>
> I encountered following error while rebuilding a node in new datacenter.
> {noformat}
> ERROR [StreamReceiveTask:9] 2016-07-21 02:04:37,131
> StreamReceiveTask.java:215 - Error applying streamed data:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db
> at
> org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:50)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:372)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.buildIndexesBlocking(SecondaryIndexManager.java:375)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.buildAllIndexesBlocking(SecondaryIndexManager.java:262)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:182)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_91]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> [na:1.8.0_91]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> [na:1.8.0_91]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:365)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 8 common frames omitted
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> Corrupted: /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:367)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 5 common frames omitted
> Caused by: java.io.IOException: Error building row with data deserialized
> from /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db - chunk
> length 65536, data length 81867738.
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:446)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.AssertionError: Attempted to use closed region
> at
> org.apache.cassandra.io.util.MmappedRegions.floor(MmappedRegions.java:190)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:172)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.SerializationHeader.readLocalDeletionTime(SerializationHeader.java:210)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:455)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:433)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 27 common frames omitted
> ERROR [StreamReceiveTask:9] 2016-07-21 02:04:37,134 StreamSession.java:519 -
> [Stream #91549300-4ecd-11e6-8fe6-0deed292eef4] Streaming error occurred
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db
> at
> org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:50)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:372)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.buildIndexesBlocking(SecondaryIndexManager.java:375)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.buildAllIndexesBlocking(SecondaryIndexManager.java:262)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:182)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_91]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> [na:1.8.0_91]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> [na:1.8.0_91]
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:365)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 8 common frames omitted
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> Corrupted: /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:367)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 5 common frames omitted
> Caused by: java.io.IOException: Error building row with data deserialized
> from /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db - chunk
> length 65536, data length 81867738.
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:446)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.AssertionError: Attempted to use closed region
> at
> org.apache.cassandra.io.util.MmappedRegions.floor(MmappedRegions.java:190)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:172)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.SerializationHeader.readLocalDeletionTime(SerializationHeader.java:210)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:455)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:433)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 27 common frames omitted
> ERROR [CompactionExecutor:5] 2016-07-21 02:04:37,141 CassandraDaemon.java:195
> - Exception in thread Thread[CompactionExecutor:5,1,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:367)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_91]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Error building row with data deserialized
> from /var/lib/cassandra/data/keyspace/table_name/ma-18-big-Data.db - chunk
> length 65536, data length 81867738.
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:446)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.AssertionError: Attempted to use closed region
> at
> org.apache.cassandra.io.util.MmappedRegions.floor(MmappedRegions.java:190)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:172)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.SerializationHeader.readLocalDeletionTime(SerializationHeader.java:210)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:455)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:433)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 27 common frames omitted
> {noformat}
> At the end of rebuilding processes:
> {noformat}
> WARN [StreamReceiveTask:51] 2016-07-21 03:03:17,781
> StreamResultFuture.java:212 - [Stream #91549300-4ecd-11e6-8fe6-0deed292eef4]
> Stream failed
> ERROR [RMI TCP Connection(24)-127.0.0.1] 2016-07-21 03:03:17,783
> StorageService.java:1171 - Error while rebuilding node
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:706)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:667)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:211)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_91]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
> {noformat}
> I tried to run rebuild command again and it is completed successfully without
> giving any error. I wonder if there is data corruption on this node
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
