Has anyone seen this error or might have an idea what is causing this? Version: 3.5
java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170) at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) at org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88) at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66) at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) at org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338) at org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:391) at org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeOne(ClusteringPrefix.java:480) at org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeAll(ClusteringPrefix.java:486) at org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeNextClustering(ClusteringPrefix.java:502) at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:210) at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125) at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149) at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549) at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570) at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70) at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I ran `nodetool rebuild_index -- keyspace table index` on 6 nodes of a 6 node cluster. After several hours of processing, it failed on 3 of the nodes with the following additional call stacks: java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170) at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144) at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63) at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209) at org.apache.cassandra.db.Columns$Serializer.deserializeSubset(Columns.java:494) at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:429) at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211) at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125) at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149) at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549) at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570) at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70) at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170) at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144) at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:73) at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209) at org.apache.cassandra.db.SerializationHeader.readTimestamp(SerializationHeader.java:205) at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:420) at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211) at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125) at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149) at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549) at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570) at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70) at org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Is something timing out and freeing up the memory? Mark Bryant [Sumo-Logo] [Tel] +44 (0)114 2426766 [Web]www.sumo-digital.com [Address]32 Jessops Riverside, 800 Brightside Lane, Sheffield, S9 2RX, United Kingdom [------------------------------------------------------------------------------------------------------------------------------] [Facebook]<https://www.facebook.com/SumoDigital> [Twitter] <https://twitter.com/sumodigitalltd> [LinkedIn] <https://www.linkedin.com/company/sumo-digital> The contents of this email are confidential and may not be copied, disclosed to other parties or used by anyone other than the intended recipient. If you received this email in error, please contact Sumo Digital Ltd by email: i...@sumo-digital.com or by calling +44 (0)114 2426766. Quote the name of the sender, your email address and then delete this email from your system. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org