[
https://issues.apache.org/jira/browse/CASSANDRA-21486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093536#comment-18093536
]
Sebastian Marsching commented on CASSANDRA-21486:
-------------------------------------------------
[~dnk] Thank you for your suggestion. I might do this at a later point in time,
but first I have to apologize, because based on my latest research, I think
that I blamed C* for something that wasn’t even caused by Cassandra.
Around the same time that we upgraded to C* 5.0.7, we installed another service
(Pacemaker). I had checked that service and it looked fine and {{top}}
primarily showed me the high memory usage of C*, so this is what I blamed. As
it turns out, Pacemaker pulled in {{pcsd}} as a dependency (not even a true one
just a “recommends” dependency, but it was still installed automatically). As
we do not use {{{}pcsd{}}}, I didn’t even consider it at first.
Well, turns out that even though it wasn’t properly configured and wasn’t doing
anything it had acquired 20 GB of memory over time, and this is what caused the
alarms for high memory usage. After stopping this (unused) service, all nodes
immediately recovered.
So, C* 5.0.7 did not actually introduce any problems, and the somewhat strange
behavior with regards to mapped byte buffers was a red herring. What we are
seeing with the byte buffers might not even be a bug but expected behavior. I
was able to recover older data from our metrics storage, and using this
information, I was able to get a better idea what was happening there:
Around last October, we made several changes in a rather short time period, so
I cannot say for sure which of the changes introduced this:
* We switched from STCS to UCS.
* We switched from the BIG to the BTI format for SSTables.
* We switches must settings from the legacy to the new “recommended” settings
(trie memtables, {{memtable_allocation_type}} changed from {{heap_buffers}} to
{{{}offheap_objects{}}}, {{commitlog_disk_access_mode}} from {{legacy}} to
{{{}auto{}}}).
This is the point in time where the size occupied by mapped byte buffers
started growing constantly. Thinking about it, this might really be just the
combined effect to changing these options and having a growing amount of data
over time. In particular, it did not lead to a significant increase in overall
memory consumption, so it probably is not a bug.
For the other thing that we recently saw on a few nodes, I am not quite so sure
whether it is expected. I think it can most easily be explained by showing a
few metrics:
!image-2026-07-03-22-53-29-737.png!
!image-2026-07-03-22-53-37-284.png!
!image-2026-07-03-22-55-11-607.png!
As seen in the diagrams, the load of the node and the used disk space of the
main keyspace (in our case, almost all data is stored in a single table within
a single keyspace) are growing constantly. That’s expected due to new data
coming in with old data rarely being deleted. The big drop in October 2025 was
probably due to the change in compaction strategy and the subsequent
compactions removing some deleted data.
The estimated number of mapped buffers grows constantly while C* is running and
drops when the daemon is restarted. I am not sure whether this behavior is
expected. What I find very suspicous is the sharp rise in June 2026, associated
with a very sharp rise in the space used. I also find it suspicious that the
space occupied by the buffers grows much more quickly than the load of the
node. While the load grew from about 1.5 TiB to about 1.7 TiB, so maybe a bit
more than 10 %, the space occupied by the mapped byte buffers grew from about 7
GiB to about 12 GiB, so it almost doubled. Is this expected? Also, unlike the
number of buffers, the space occupied by them does not drop when restarting the
C* daemon.
So, I have two questions:
# Is the sudden sharp rise in the number of space occupied by the mapped byte
buffers that seems to correlate with a rather small spike in the number of
SSTables expected and if so, what is causing it?
# Is the constant rise in the space occupied by the mapped byte buffers that
grows much more quickly than the node’s load expected?
For the moment, I do not claim that there is a bug, I just want to understand
what is happening there, and whether this is completely harmless behavior, we
might have to tune our configuration, or this might actually be a bug.
> Possible NIO mapped byte buffer leak
> ------------------------------------
>
> Key: CASSANDRA-21486
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21486
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Sebastian Marsching
> Priority: Normal
> Attachments: image-2026-07-03-22-53-29-737.png,
> image-2026-07-03-22-53-37-284.png, image-2026-07-03-22-55-11-607.png
>
>
> After upgrading two clusters from C* 5.0.5 to 5.0.7, we have noticed the
> following behavior:
> Where previously, memory (RAM) usage was mostly stable over time, it is now
> increasing constantly over time, with the consequence that we have hit alarm
> limits on some of the nodes. A restart of the Cassandra daemon decreases the
> usage only slightly. It is still much higher than before.
> This behavior seems to correlate with the NIO mapped byte buffers. On one of
> the nodes, we noticed that the estimated number of these buffers sharply
> increased (about fourfold from about 4000 to about 16000) about three weeks
> ago. This was accompanied by a sharp increase of the estimated capacity and
> used bytes (from about 9 GiB to about 12 GiB). While the number of buffers
> dropped to about 900 after restarting the Cassandra daemon, the capacity and
> used bytes did not drop at all and are steadily increasing.
> This sharp rise happened around a time where the number of live SSTables for
> our main keyspace rose from about 290 to about 320, and the live disk space
> bytes dropped slightly. This might hint at the fact that the sharp increase
> in the number of NIO buffers might be associated with compaction.
> For context, I should add that a while ago, we switched from STCS to UCS, but
> this was rather long ago (in October 2025), while the problem with memory
> usage only started in early April 2026, after upgrading from 5.0.5 to 5.0.7.
> The problem seems somewhat similar to CASSANDRA-20753, but as this has been
> fixed in 5.0.7, it cannot be the cause. I wonder whether the fix could
> possibly have introduced a new problem, which would explain why we only
> started seeing the problem after upgrading to 5.0.7.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
