[
https://issues.apache.org/jira/browse/CASSANDRA-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17925017#comment-17925017
]
Dmitry Konstantinov commented on CASSANDRA-20298:
-------------------------------------------------
So, the heap is occupied with memtables actually, we have 849 of them:
{code:java}
[junit-timeout] INFO [7] 2025-02-07 18:15:03,590 CommitLogCQLTest.java:164 -
=== Histogram ===
[junit-timeout] num #instances #bytes class name (module)
[junit-timeout] -------------------------------------------------------
[junit-timeout] 1: 71277 904901920 [B ([email protected])
[junit-timeout] 2: 5030 7141800 [J ([email protected])
[junit-timeout] 3: 50749 2841944 java.nio.HeapByteBuffer
([email protected])
[junit-timeout] 4: 57510 2010088 [Ljava.lang.Object;
([email protected])
[junit-timeout] 5: 68356 1640544 java.lang.String
([email protected])
[junit-timeout] 6: 3937 1273808 [I ([email protected])
[junit-timeout] 7: 9204 1111584 java.lang.Class
([email protected])
[junit-timeout] 8: 24877 995080
org.apache.cassandra.db.rows.BufferCell
[junit-timeout] 9: 24852 994080
org.apache.cassandra.db.rows.BTreeRow
[junit-timeout] 10: 28336 906752
java.util.concurrent.ConcurrentHashMap$Node ([email protected])
[junit-timeout] 11: 25673 821536
org.apache.cassandra.db.rows.EncodingStats
[junit-timeout] 12: 24850 795200
org.apache.cassandra.db.partitions.BTreePartitionData
[junit-timeout] 13: 24842 794944
org.apache.cassandra.db.partitions.AtomicBTreePartition
[junit-timeout] 14: 32259 774216
javax.management.ObjectName$Property ([email protected])
[junit-timeout] 15: 22531 720992 java.util.HashMap$Node
([email protected])
[junit-timeout] 16: 25663 615912
java.util.concurrent.ConcurrentSkipListMap$Node ([email protected])
[junit-timeout] 17: 24851 596424
org.apache.cassandra.dht.Murmur3Partitioner$LongToken
[junit-timeout] 18: 24848 596352
org.apache.cassandra.db.LivenessInfo
[junit-timeout] 19: 24846 596304
org.apache.cassandra.db.BufferDecoratedKey
[junit-timeout] 20: 16925 535040
[Ljavax.management.ObjectName$Property; ([email protected])
[junit-timeout] 21: 2091 396976 [S ([email protected])
[junit-timeout] 22: 2858 375776 [Ljava.util.HashMap$Node;
([email protected])
[junit-timeout] 23: 13084 314016
java.util.concurrent.ConcurrentSkipListMap$Index ([email protected])
[junit-timeout] 24: 9453 302496
java.util.concurrent.atomic.LongAdder ([email protected])
[junit-timeout] 25: 8463 270816 javax.management.ObjectName
([email protected])
[junit-timeout] 26: 369 232112
[Ljava.util.concurrent.ConcurrentHashMap$Node; ([email protected])
[junit-timeout] 27: 4656 223488 com.codahale.metrics.EWMA
[junit-timeout] 28: 8458 202992
com.sun.jmx.mbeanserver.NamedObject ([email protected])
[junit-timeout] 29: 8418 202032
com.sun.jmx.mbeanserver.StandardMBeanSupport ([email protected])
[junit-timeout] 30: 355 168064 [C ([email protected])
[junit-timeout] 31: 9870 157920 java.lang.Object
([email protected])
[junit-timeout] 32: 1703 149864 java.lang.reflect.Method
([email protected])
[junit-timeout] 33: 5840 140160
java.util.concurrent.atomic.AtomicLong ([email protected])
[junit-timeout] 34: 2633 126384 java.lang.invoke.MemberName
([email protected])
[junit-timeout] 35: 2928 117120 java.lang.ref.SoftReference
([email protected])
[junit-timeout] 36: 3465 110880
org.apache.cassandra.metrics.TableMetrics$$Lambda$870/0x0000000801459e48
[junit-timeout] 37: 366 102480
java.util.concurrent.atomic.Striped64$Cell ([email protected])
[junit-timeout] 38: 2124 101952 java.util.HashMap
([email protected])
[junit-timeout] 39: 1701 81648
java.util.concurrent.ConcurrentSkipListMap ([email protected])
[junit-timeout] 40: 1986 79440
sun.util.locale.LocaleObjectCache$CacheEntry ([email protected])
[junit-timeout] 41: 3213 77112
org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge
[junit-timeout] 42: 849 74712
org.apache.cassandra.db.memtable.SkipListMemtable
[junit-timeout] 43: 1865 74600 java.lang.invoke.MethodType
([email protected])
[junit-timeout] 44: 1696 67840
org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator {code}
We have OOM thrown from MemtableFlushWriter thread pool:
{code}
[junit-timeout] ERROR [MemtableFlushWriter:1] 2025-02-07 18:15:03,445
JVMStabilityInspector.java:70 - Exception in thread
Thread[MemtableFlushWriter:1,5,MemtableFlushWriter]
[junit-timeout] java.lang.OutOfMemoryError: Java heap space
[junit-timeout] at java.base/java.util.Arrays.copyOf(Arrays.java:3481)
[junit-timeout] at
com.google.common.collect.ImmutableList$Builder.getReadyToExpandTo(ImmutableList.java:797)
[junit-timeout] at
com.google.common.collect.ImmutableList$Builder.add(ImmutableList.java:816)
[junit-timeout] at
com.google.common.collect.ImmutableList$Builder.add(ImmutableList.java:776)
[junit-timeout] at
com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:493)
[junit-timeout] at
com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:884)
[junit-timeout] at
com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:282)
[junit-timeout] at
com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
[junit-timeout] at
org.apache.cassandra.db.lifecycle.View$6.apply(View.java:336)
[junit-timeout] at
org.apache.cassandra.db.lifecycle.View$6.apply(View.java:332)
[junit-timeout] at
org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:162)
[junit-timeout] at
org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:135)
[junit-timeout] at
org.apache.cassandra.db.lifecycle.Tracker.markFlushing(Tracker.java:391)
[junit-timeout] at
org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1243)
[junit-timeout] at
org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
[junit-timeout] at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[junit-timeout] at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[junit-timeout] at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout] at java.base/java.lang.Thread.run(Thread.java:833)
{code}
The mutation threads are mostly in two places: looking for a memtable and
writing a commit log:
{code}
[junit-timeout] INFO [7] 2025-02-07 18:15:03,600 CommitLogCQLTest.java:212 -
"MemtableFlushWriter:1"
[junit-timeout] java.lang.Thread.State: RUNNABLE, waiting time: -1
[junit-timeout] at
app//com.google.common.collect.Iterators.forArray(Iterators.java:1074)
[junit-timeout] at
app//com.google.common.collect.RegularImmutableList.listIterator(RegularImmutableList.java:85)
[junit-timeout] at
app//com.google.common.collect.ImmutableList.listIterator(ImmutableList.java:404)
[junit-timeout] at
app//com.google.common.collect.ImmutableList.iterator(ImmutableList.java:399)
[junit-timeout] at
app//com.google.common.collect.ImmutableList.iterator(ImmutableList.java:63)
[junit-timeout] at
app//com.google.common.collect.Iterables$4.iterator(Iterables.java:583)
[junit-timeout] at
app//com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.View$6.apply(View.java:336)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.View$6.apply(View.java:332)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:162)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:135)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.Tracker.markFlushing(Tracker.java:391)
[junit-timeout] at
app//org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1243)
[junit-timeout] at
app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
[junit-timeout] at
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[junit-timeout] at
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[junit-timeout] at
app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout] at
[email protected]/java.lang.Thread.run(Thread.java:833)
[junit-timeout] INFO [7] 2025-02-07 18:15:03,600 CommitLogCQLTest.java:212 -
"MemtableFlushWriter:2"
[junit-timeout] java.lang.Thread.State: RUNNABLE, waiting time: -1
[junit-timeout] at
app//com.google.common.base.Predicates$IsEqualToPredicate.apply(Predicates.java:461)
[junit-timeout] at
app//com.google.common.base.Predicates$NotPredicate.apply(Predicates.java:327)
[junit-timeout] at
app//com.google.common.collect.Iterators$5.computeNext(Iterators.java:672)
[junit-timeout] at
app//com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
[junit-timeout] at
app//com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
[junit-timeout] at
app//com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:492)
[junit-timeout] at
app//com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:884)
[junit-timeout] at
app//com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:282)
[junit-timeout] at
app//com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:239)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.View$6.apply(View.java:336)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.View$6.apply(View.java:332)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:162)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:135)
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.Tracker.markFlushing(Tracker.java:391)
[junit-timeout] at
app//org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1243)
[junit-timeout] at
app//org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
[junit-timeout] at
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[junit-timeout] at
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[junit-timeout] at
app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout] at
[email protected]/java.lang.Thread.run(Thread.java:833)
[junit-timeout] INFO [7] 2025-02-07 18:15:03,600 CommitLogCQLTest.java:212 -
"0"
[junit-timeout] java.lang.Thread.State: RUNNABLE, waiting time: -1
[junit-timeout] at
app//org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:318)
[junit-timeout] at
app//org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99)
[junit-timeout] at
app//org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:625)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.apply(Keyspace.java:510)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:249)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:269)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:692)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:683)
[junit-timeout] at
app//org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:452)
[junit-timeout] at
app//org.apache.cassandra.db.commitlog.CommitLogCQLTest$1.run(CommitLogCQLTest.java:99)
[junit-timeout] INFO [7] 2025-02-07 18:15:03,601 CommitLogCQLTest.java:212 -
"2"
[junit-timeout] java.lang.Thread.State: RUNNABLE, waiting time: -1
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.Tracker.getMemtableFor(Tracker.java:366)
[junit-timeout] at
app//org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1472)
[junit-timeout] at
app//org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:38)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:653)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.apply(Keyspace.java:510)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:249)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:269)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:692)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:683)
[junit-timeout] at
app//org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:452)
[junit-timeout] at
app//org.apache.cassandra.db.commitlog.CommitLogCQLTest$1.run(CommitLogCQLTest.java:99)
[junit-timeout] INFO [7] 2025-02-07 18:15:03,601 CommitLogCQLTest.java:212 -
"3"
[junit-timeout] java.lang.Thread.State: RUNNABLE, waiting time: -1
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.Tracker.getMemtableFor(Tracker.java:366)
[junit-timeout] at
app//org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1472)
[junit-timeout] at
app//org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:38)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:653)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.apply(Keyspace.java:510)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:249)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:269)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:692)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:683)
[junit-timeout] at
app//org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:452)
[junit-timeout] at
app//org.apache.cassandra.db.commitlog.CommitLogCQLTest$1.run(CommitLogCQLTest.java:99)
[junit-timeout] INFO [7] 2025-02-07 18:15:03,601 CommitLogCQLTest.java:212 -
"4"
[junit-timeout] java.lang.Thread.State: RUNNABLE, waiting time: -1
[junit-timeout] at
app//org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:318)
[junit-timeout] at
app//org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99)
[junit-timeout] at
app//org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:625)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.apply(Keyspace.java:510)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:249)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:269)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:692)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:683)
[junit-timeout] at
app//org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:452)
[junit-timeout] at
app//org.apache.cassandra.db.commitlog.CommitLogCQLTest$1.run(CommitLogCQLTest.java:99)
[junit-timeout] INFO [7] 2025-02-07 18:15:03,601 CommitLogCQLTest.java:212 -
"5"
[junit-timeout] java.lang.Thread.State: RUNNABLE, waiting time: -1
[junit-timeout] at
app//org.apache.cassandra.db.lifecycle.Tracker.getMemtableFor(Tracker.java:366)
[junit-timeout] at
app//org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1472)
[junit-timeout] at
app//org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:38)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:653)
[junit-timeout] at
app//org.apache.cassandra.db.Keyspace.apply(Keyspace.java:510)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:249)
[junit-timeout] at
app//org.apache.cassandra.db.Mutation.apply(Mutation.java:269)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:692)
[junit-timeout] at
app//org.apache.cassandra.cql3.statements.ModificationStatement.executeLocally(ModificationStatement.java:683)
[junit-timeout] at
app//org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:452)
[junit-timeout] at
app//org.apache.cassandra.db.commitlog.CommitLogCQLTest$1.run(CommitLogCQLTest.java:99)
[junit-timeout]
{code}
> Test failure CommitLogCQLTest.testSwitchMemtable
> ------------------------------------------------
>
> Key: CASSANDRA-20298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20298
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
> Reporter: Brandon Williams
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Fix For: 5.0.x
>
> Attachments:
> TEST-org.apache.cassandra.db.commitlog.CommitLogCQLTest.log,
> TEST-org.apache.cassandra.db.commitlog.CommitLogCQLTest.xml, steps.log,
> test_log_with_heap_histo_and_thread_dump.txt
>
>
> Seen here:
> https://app.circleci.com/pipelines/github/driftx/cassandra/1831/workflows/0de1611d-d409-4d15-8171-dcf7183a8c61/jobs/112290/tests
> {noformat}
> unit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please
> note the time in the report does not reflect the time until the VM exit.
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.util.Vector.forEach(Vector.java:1365)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.util.Vector.forEach(Vector.java:1365)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
