[
https://issues.apache.org/jira/browse/CASSANDRA-21486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093489#comment-18093489
]
Dmitry Konstantinov commented on CASSANDRA-21486:
-------------------------------------------------
[~smarsching] if you can capture a heap dump and analyze it - it would be very
helpful: you can use Eclipse Memory Analyzer tool to find mapped byte buffers
as objects in it and check paths to GC roots for them, to clarify owners for
these buffers.
Do not share the heap dump itself in public, it may contain sensitive
information.
Also please clarify what disk_access_mode option do you use and have you
changed it recently?
> Possible NIO mapped byte buffer leak
> ------------------------------------
>
> Key: CASSANDRA-21486
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21486
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Sebastian Marsching
> Priority: Normal
>
> After upgrading two clusters from C* 5.0.5 to 5.0.7, we have noticed the
> following behavior:
> Where previously, memory (RAM) usage was mostly stable over time, it is now
> increasing constantly over time, with the consequence that we have hit alarm
> limits on some of the nodes. A restart of the Cassandra daemon decreases the
> usage only slightly. It is still much higher than before.
> This behavior seems to correlate with the NIO mapped byte buffers. On one of
> the nodes, we noticed that the estimated number of these buffers sharply
> increased (about fourfold from about 4000 to about 16000) about three weeks
> ago. This was accompanied by a sharp increase of the estimated capacity and
> used bytes (from about 9 GiB to about 12 GiB). While the number of buffers
> dropped to about 900 after restarting the Cassandra daemon, the capacity and
> used bytes did not drop at all and are steadily increasing.
> This sharp rise happened around a time where the number of live SSTables for
> our main keyspace rose from about 290 to about 320, and the live disk space
> bytes dropped slightly. This might hint at the fact that the sharp increase
> in the number of NIO buffers might be associated with compaction.
> For context, I should add that a while ago, we switched from STCS to UCS, but
> this was rather long ago (in October 2025), while the problem with memory
> usage only started in early April 2026, after upgrading from 5.0.5 to 5.0.7.
> The problem seems somewhat similar to CASSANDRA-20753, but as this has been
> fixed in 5.0.7, it cannot be the cause. I wonder whether the fix could
> possibly have introduced a new problem, which would explain why we only
> started seeing the problem after upgrading to 5.0.7.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
