[
https://issues.apache.org/jira/browse/CASSANDRA-18556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-18556:
-----------------------------------------
Resolution: Invalid
Status: Resolved (was: Triage Needed)
This jira is for the development of Apache Cassandra, and as such makes for a
poor vehicle for support. For support I recommend contacting the community on
the ML or slack: https://cassandra.apache.org/_/community.html
Also note that 3.11.1 is nearly 6 years old at this point; you should probably
upgrade to the latest 3.11 release and see if your problems persist.
> Leak detected issue in cassandra 3.11.1
> ---------------------------------------
>
> Key: CASSANDRA-18556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18556
> Project: Cassandra
> Issue Type: Bug
> Reporter: Purushottam baghel
> Priority: Normal
> Attachments: Screenshot 2023-05-30 at 10.50.32 AM.png
>
>
> We are running 36 nodes Cassandra cluster. All nodes have the some config,
> Cassandra version is 3.11.1. Suddenly, we are seeing fluctuation in storage
> available graphs for the 2 nodes. Attaching the last 7 days used storage
> graph for the Cassandra nodes.
> Further investigation, we saw errors in the logs of these 2 Cassandra node
> logs.
> {code:java}
> ERROR [CompactionExecutor:164360] 2023-05-04 00:30:58,094
> CassandraDaemon.java:228 - Exception in thread
> Thread[CompactionExecutor:164360,1,main]
> java.lang.IllegalArgumentException: null
> at java.nio.Buffer.position(Buffer.java:244) ~[na:1.8.0_131]
> at
> org.apache.cassandra.io.util.SafeMemoryWriter.reallocate(SafeMemoryWriter.java:59)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.util.SafeMemoryWriter.setCapacity(SafeMemoryWriter.java:68)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.sstable.IndexSummaryBuilder.prepareToCommit(IndexSummaryBuilder.java:250)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.doPrepare(BigTableWriter.java:524)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:364)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.java:281)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:379)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_131]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_131]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.1.jar:3.11.1]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
> WARN [CompactionExecutor:164909] 2023-05-04 00:31:54,573
> IndexSummaryBuilder.java:115 - min_index_interval of 128 is too low for
> 4298508892 expected keys of avg size 64; using interval of 145 instead
> ERROR [Reference-Reaper:1] 2023-05-04 00:32:02,326 Ref.java:224 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@4f374614) to class
> org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1106289026:Memory@[7fa6ab9ec010..7fa72ad798d0)
> was not released before the reference was garbage collected {code}
> Compactions also running continually run on these nodes.
> I am unable to understand the root cause of it. Any help is appreciated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
