[
https://issues.apache.org/jira/browse/CASSANDRA-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18089623#comment-18089623
]
Dmitry Konstantinov edited comment on CASSANDRA-21390 at 6/19/26 11:21 AM:
---------------------------------------------------------------------------
Interim summary:
* based on inputs from [~samueldlightfoot] (see also: [^findings_mask.md])
the issue is reproduced on several clusters with the following configuration
(it does not mean that it is only reproduced in such conditions):
version: 5.0.6
allocation type: offheap_objects
memtable type: SkipListMemtable
affected tables: cluster (Cassandra Reaper)
* we have found 3 potential scenarios but all of them due to different reasons
cannot be considered as really happened on the reported envs:
** CASSANDRA-21469 - suggested by [~smiklosovic] : override of clustering row
with out of order timestamp, the updated row is incorrectly accounted into heap
usage while it is not actually stored (similar to CASSANDRA-18125), MR:
[https://github.com/apache/cassandra/pull/4886/changes]
:( Reaper cluster table does not have clustering keys.
** suggested by [~samueldlightfoot] : if an exception is thrown from
tryUpdateData then we may miss an update of memtable but increment usage in
finally block.
:( In theory it is possible but we have not seen any such exceptions in the log
on the affected clusters.
** CASSANDRA-21472 - suggested by [~dnk]: we have asymmetry in BTree heap
usage measurement: when we remove a complex column we measure heap size for
inner BTree cells using org.apache.cassandra.utils.btree.BTree#sizeOnHeapOf
which includes size maps. When we update a BTree by adding elements into it
then we do not include size maps.
:( It requires more than 31 element in set to reproduce and usage of +/- set
operations which are not present in Reaper code
So, the mystery still remains :)
was (Author: dnk):
Interim summary:
* based on inputs from [~samueldlightfoot] (see also: [^findings_mask.md])
the issue is reproduced on several clusters with the following configuration
(it does not mean that it is only reproduced in such conditions):
version: 5.0.6
allocation type: offheap_objects
memtable type: SkipListMemtable
affected tables: cluster (Cassandra Reaper)
* we have found 3 potential scenarios but all of them due to different reasons
cannot be considered as really happened on the reported envs:
** suggested by [~smiklosovic] : override of clustering row with out of order
timestamp, the updated row is incorrectly accounted into heap usage while it is
not actually stored (similar to CASSANDRA-18125), MR:
[https://github.com/apache/cassandra/pull/4886/changes]
:( Reaper cluster table does not have clustering keys.
** suggested by [~samueldlightfoot] : if an exception is thrown from
tryUpdateData then we may miss an update of memtable but increment usage in
finally block.
:( In theory it is possible but we have not seen any such exceptions in the log
on the affected clusters.
** suggested by [~dnk]: we have asymmetry in BTree heap usage measurement:
when we remove a complex column we measure heap size for inner BTree cells
using org.apache.cassandra.utils.btree.BTree#sizeOnHeapOf which includes size
maps. When we update a BTree by adding elements into it then we do not include
size maps.
:( It requires more than 31 element in set to reproduce and usage of +/- set
operations which are not present in Reaper code
So, the mystery still remains :)
> TrieMemtable MemtableReclaimMemory AssertionError: Negative released in
> MemtablePool$SubPool
> --------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21390
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Memtable
> Reporter: Praveen Reddy Arra
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Attachments: cassandra.yaml, findings_mask.md,
> image-2026-05-21-09-17-49-716.png, reaper_cluster_output-1.txt, sstabledump
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We have started seeing this fatal exception in Apache Cassandra 5.0.6 on one
> of our clusters.
> {code:java}
> [ERROR] [MemtableReclaimMemory:1] cluster_id=xxx ip_address=xxx.xxx.xxx.xxx
> JVMStabilityInspector.java:70 - Exception in thread
> Thread[MemtableReclaimMemory:1,5,MemtableReclaimMemory]
> java.lang.AssertionError: Negative released: -4332
> at
> org.apache.cassandra.utils.memory.MemtablePool$SubPool.released(MemtablePool.java:194)
> at
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.releaseAll(MemtableAllocator.java:153)
> at
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.setDiscarded(MemtableAllocator.java:144)
> at
> org.apache.cassandra.utils.memory.MemtableAllocator.setDiscarded(MemtableAllocator.java:95)
> at
> org.apache.cassandra.utils.memory.NativeAllocator.setDiscarded(NativeAllocator.java:205)
> at
> org.apache.cassandra.db.memtable.AbstractAllocatorMemtable.discard(AbstractAllocatorMemtable.java:171)
> at
> org.apache.cassandra.db.memtable.TrieMemtable.discard(TrieMemtable.java:163)
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1398)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:842)
> {code}
> {code:yaml}
> memtable_allocation_type: offheap_objects
> file_cache_enabled: true
> file_cache_size: 2048MiB
> memtable:
> configurations:
> skiplist:
> class_name: SkipListMemtable
> trie:
> class_name: TrieMemtable
> default:
> inherits: trie
> {code}
> it looks similar to the open MemtableReclaimMemory assert issue in
> CASSANDRA-18159
> Notes:
> * RHEL 8.10 with OpenJDK 17
> * 16GB heap
> * -ea enabled
> * Reaper v3.8
> * SkipListMemtable and TrieMemtable
> * memtable_allocation_type: offheap_objects ({*}only{*})
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
