3.5 is probably not a version you should be using in production in 2018 - it was a feature release and has had no bug fixes for years. Going up to 3.11.3 will likely fix many serious bugs you’re not noticing, and maybe the bug below you are noticing
-- Jeff Jirsa > On Oct 24, 2018, at 5:03 AM, Mark Bryant <mbry...@sumo-digital.com> wrote: > > Has anyone seen this error or might have an idea what is causing this? > > Version: 3.5 > > java.lang.AssertionError: Memory was freed > at > org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170) > at > org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) > at > org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88) > at > org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66) > at > org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338) > at > org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:391) > at > org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeOne(ClusteringPrefix.java:480) > at > org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeAll(ClusteringPrefix.java:486) > at > org.apache.cassandra.db.ClusteringPrefix$Deserializer.deserializeNextClustering(ClusteringPrefix.java:502) > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:210) > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125) > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149) > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) > at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) > at > org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549) > at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570) > at > org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70) > at > org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > I ran `nodetool rebuild_index -- keyspace table index` on 6 nodes of a 6 node > cluster. After several hours of processing, it failed on 3 of the nodes with > the following additional call stacks: > > java.lang.AssertionError: Memory was freed > at > org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170) > at > org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) > at > org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144) > at > org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63) > at > org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209) > at > org.apache.cassandra.db.Columns$Serializer.deserializeSubset(Columns.java:494) > at > org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:429) > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211) > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125) > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149) > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) > at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) > at > org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549) > at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570) > at > org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70) > at > org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > java.lang.AssertionError: Memory was freed > at > org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170) > at > org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) > at > org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144) > at > org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:73) > at > org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209) > at > org.apache.cassandra.db.SerializationHeader.readTimestamp(SerializationHeader.java:205) > at > org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:420) > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211) > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125) > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149) > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) > at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) > at > org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549) > at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570) > at > org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70) > at > org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Is something timing out and freeing up the memory? > Mark Bryant > [Sumo-Logo] [Tel] +44 (0)114 2426766 > [Web]www.sumo-digital.com > [Address]32 Jessops Riverside, 800 Brightside Lane, Sheffield, S9 2RX, United > Kingdom > [------------------------------------------------------------------------------------------------------------------------------] > [Facebook]<https://www.facebook.com/SumoDigital> [Twitter] > <https://twitter.com/sumodigitalltd> [LinkedIn] > <https://www.linkedin.com/company/sumo-digital> > The contents of this email are confidential and may not be copied, disclosed > to other parties or used by anyone other than the intended recipient. If you > received this email in error, please contact Sumo Digital Ltd by email: > i...@sumo-digital.com or by calling +44 (0)114 2426766. Quote the name of the > sender, your email address and then delete this email from your system. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
