[
https://issues.apache.org/jira/browse/CASSANDRA-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18085792#comment-18085792
]
Brad Schoening commented on CASSANDRA-21390:
--------------------------------------------
Also seeing it today on this same cluster having converted back to SkipLists
(with Cassandra 5.0.6):
2026-06-02 07:06:49,108 [ERROR] [MemtableReclaimMemory:1] cluster_id=xxx
ip_address=10.0.0.1 JVMStabilityInspector.java:70 - Exception in thread
Thread[MemtableReclaimMemory:1,5,MemtableReclaimMemory]
java.lang.AssertionError: Negative released: -10764
at
org.apache.cassandra.utils.memory.MemtablePool$SubPool.released(MemtablePool.java:194)
at
org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.releaseAll(MemtableAllocator.java:153)
at
org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.setDiscarded(MemtableAllocator.java:144)
at
org.apache.cassandra.utils.memory.MemtableAllocator.setDiscarded(MemtableAllocator.java:95)
at
org.apache.cassandra.utils.memory.NativeAllocator.setDiscarded(NativeAllocator.java:205)
at
org.apache.cassandra.db.memtable.AbstractAllocatorMemtable.discard(AbstractAllocatorMemtable.java:171)
at
org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1398)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
at
org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:842)
2026-06-02 07:06:49,153 [INFO ] [CompactionExecutor:2360] cluster_id=xxx
ip_address=10.0.0.1 CompactionTask.java:268 - Compacted
(9fc55e30-5e51-11f1-88ef-15b009575214) 2 sstables to
[/opt/nosql/data/cluster_xxx
/data/caas_repair/cluster-dcae655014bb11ed8414b52584c330bb/da-26210-bti,] to
level=0. 1.962KiB to 1008B (~50% of original) in 45ms. Read Throughput =
43.574KiB/s, Write Throughput = 21.863KiB/s, Row Throughput = ~2/s. 2 total
partitions merged to 1. Partition merge counts were \{2:1, }. Time spent
writing keys = 11ms
2026-06-02 07:06:49,169 [INFO ] [NonPeriodicTasks:1] cluster_id=xxx
ip_address=10.0.0.1 BtiFormat.java:216 - Deleting sstable:
/opt/nosql/data/cluster_xxx
/data/caas_repair/cluster-dcae655014bb11ed8414b52584c330bb/da-26208-bti
2026-06-02 07:06:49,178 [INFO ] [NonPeriodicTasks:1] cluster_id=xxx
ip_address=10.0.0.1 BtiFormat.java:216 - Deleting sstable:
/opt/nosql/data/cluster_xxx
/data/caas_repair/cluster-dcae655014bb11ed8414b52584c330bb/da-26209-bti
2026-06-02 07:07:34,705 [INFO ] [Repair-Task:16] cluster_id=xxx
ip_address=10.0.0.1 RepairCoordinator.java:366 - Starting repair command #36
(baf2eb00-5e51-11f1-88ef-15b009575214), repairing keyspace system_auth with
repair options (parallelism: parallel, primary range: false, incremental:
false, job threads: 1, ColumnFamilies: [cidr_groups, role_permissions,
identity_to_role, network_permissions, resource_role_permissons_index, roles,
cidr_permissions, role_members], dataCenters: [], hosts: [], previewKind: NONE,
# of ranges: 1, pull repair: false, force repair: false, optimise streams:
false, ignore unreplicated keyspaces: false, repairPaxos: false, paxosOnly:
false)
2026-06-02 07:07:34,912 [INFO ] [RequestResponseStage-3] cluster_id=xxx
ip_address=10.0.0.1 AbstractRepairTask.java:68 - Starting RepairSession for
CommonRange\{endpoints=[/100.66.124.212:7000, /169.73.224.14:7000,
/100.66.124.5:7000, /169.80.166.25:7000, /169.19.190.26:7000,
/10.49.227.96:7000, /169.19.188.28:7000, /100.91.229.42:7000,
/169.75.134.52:7000, /10.49.229.9:7000, /100.66.125.114:7000,
/169.93.220.172:7000, /10.49.226.162:7000, /100.91.228.216:7000,
/169.126.118.203:7000, /169.93.225.173:7000, /169.109.78.93:7000,
/169.73.203.45:7000, /100.91.228.95:7000, /169.93.159.63:7000,
/169.75.132.17:7000, /10.49.230.74:7000, /169.109.65.172:7000,
/169.19.188.25:7000, /100.91.228.46:7000, /169.19.190.28:7000,
/169.73.224.13:7000, /169.93.225.175:7000, /169.93.237.13:7000,
/100.66.124.169:7000, /100.91.229.50:7000, /10.49.225.27:7000,
/169.93.220.174:7000, /100.66.124.120:7000, /100.91.228.247:7000,
/100.66.125.116:7000, /10.49.228.151:7000, /169.126.118.202:7000,
/169.109.65.176:7000, /169.73.219.40:7000, /169.93.225.172:7000,
/10.49.227.148:7000, /169.93.159.67:7000, /169.109.78.91:7000,
/100.91.228.18:7000, /169.19.190.23:7000, /169.75.134.211:7000,
/169.80.166.24:7000], transEndpoints=[],
ranges=[(-7344187443085297674,-7341565925883857683]], hasSkippedReplicas=false}
> TrieMemtable MemtableReclaimMemory AssertionError: Negative released in
> MemtablePool$SubPool
> --------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21390
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21390
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Memtable
> Reporter: Praveen Reddy Arra
> Priority: Normal
> Attachments: image-2026-05-21-09-17-49-716.png
>
>
> We have started seeing this fatal exception in Apache Cassandra 5.0.6 on one
> of our clusters.
> {code:java}
> [ERROR] [MemtableReclaimMemory:1] cluster_id=xxx ip_address=xxx.xxx.xxx.xxx
> JVMStabilityInspector.java:70 - Exception in thread
> Thread[MemtableReclaimMemory:1,5,MemtableReclaimMemory]
> java.lang.AssertionError: Negative released: -4332
> at
> org.apache.cassandra.utils.memory.MemtablePool$SubPool.released(MemtablePool.java:194)
> at
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.releaseAll(MemtableAllocator.java:153)
> at
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.setDiscarded(MemtableAllocator.java:144)
> at
> org.apache.cassandra.utils.memory.MemtableAllocator.setDiscarded(MemtableAllocator.java:95)
> at
> org.apache.cassandra.utils.memory.NativeAllocator.setDiscarded(NativeAllocator.java:205)
> at
> org.apache.cassandra.db.memtable.AbstractAllocatorMemtable.discard(AbstractAllocatorMemtable.java:171)
> at
> org.apache.cassandra.db.memtable.TrieMemtable.discard(TrieMemtable.java:163)
> at
> org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1398)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:26)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:842)
> {code}
> {code:yaml}
> memtable_allocation_type - heap_buffers
> file_cache_enabled: true
> file_cache_size: 2048MiB
> memtable:
> configurations:
> skiplist:
> class_name: SkipListMemtable
> trie:
> class_name: TrieMemtable
> default:
> inherits: trie
> {code}
> it looks similar to the open MemtableReclaimMemory assert issue in
> [CASSANDRA-18159|https://issues.apache.org/jira/browse/CASSANDRA-18159]
> Environment is RHEL 8.10 with OpenJDK 17 and 16GB heap, -ea enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
