Sebastian Marsching created CASSANDRA-21486:
-----------------------------------------------
Summary: Possible NIO mapped byte buffer leak
Key: CASSANDRA-21486
URL: https://issues.apache.org/jira/browse/CASSANDRA-21486
Project: Apache Cassandra
Issue Type: Bug
Reporter: Sebastian Marsching
After upgrading two clusters from C* 5.0.5 to 5.0.7, we have noticed the
following behavior:
Where previously, memory (RAM) usage was mostly stable over time, it is now
increasing constantly over time, with the consequence that we have hit alarm
limits on some of the nodes. A restart of the Cassandra daemon decreases the
usage only slightly. It is still much higher than before.
This behavior seems to correlate with the NIO mapped byte buffers. On one of
the nodes, we noticed that the estimated number of these buffers sharply
increased (about fourfold from about 4000 to about 16000) about three weeks
ago. This was accompanied by a sharp increase of the estimated capacity and
used bytes (from about 9 GiB to about 12 GiB). While the number of buffers
dropped to about 900 after restarting the Cassandra daemon, the capacity and
used bytes did not drop at all and are steadily increasing.
This sharp rise happened around a time where the number of live SSTables for
our main keyspace rose from about 290 to about 320, and the live disk space
bytes dropped slightly. This might hint at the fact that the sharp increase in
the number of NIO buffers might be associated with compaction.
For context, I should add that a while ago, we switched from STCS to UCS, but
this was rather long ago (in October 2025), while the problem with memory usage
only started in early April 2026, after upgrading from 5.0.5 to 5.0.7.
The problem seems somewhat similar to CASSANDRA-20753, but as this has been
fixed in 5.0.7, it cannot be the cause. I wonder whether the fix could possibly
have introduced a new problem, which would explain why we only started seeing
the problem after upgrading to 5.0.7.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
