[
https://issues.apache.org/jira/browse/CASSANDRA-20298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17925018#comment-17925018
]
Dmitry Konstantinov commented on CASSANDRA-20298:
-------------------------------------------------
It looks like we have a kind of livelock here:
org.apache.cassandra.db.lifecycle.Tracker#markFlushing
We have an optimistic locking logic when we take the current set of memtables
and evaluate new one. If the current view state is still the same we replace it
with a new one. In our case we flush memtables very frequently in several
threads + async -> the current view is changing very frequently, so we fail a
lot in this loop.
While we are failing in optimistic locking - more and more memtables are
flushed and added into the view, so the iteration over them in Tracker.apply
becomes longer and longer => we spend more time to calculate a new state and
our chances to fail with replacing a view because the current is changed are
increasing. So, we have a negative feedback loop when we degrade more and more
until we do not consume all space with memtables.
> Test failure CommitLogCQLTest.testSwitchMemtable
> ------------------------------------------------
>
> Key: CASSANDRA-20298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20298
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
> Reporter: Brandon Williams
> Assignee: Dmitry Konstantinov
> Priority: Normal
> Fix For: 5.0.x
>
> Attachments:
> TEST-org.apache.cassandra.db.commitlog.CommitLogCQLTest.log,
> TEST-org.apache.cassandra.db.commitlog.CommitLogCQLTest.xml, steps.log,
> test_log_with_heap_histo_and_thread_dump.txt
>
>
> Seen here:
> https://app.circleci.com/pipelines/github/driftx/cassandra/1831/workflows/0de1611d-d409-4d15-8171-dcf7183a8c61/jobs/112290/tests
> {noformat}
> unit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please
> note the time in the report does not reflect the time until the VM exit.
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.util.Vector.forEach(Vector.java:1365)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.util.Vector.forEach(Vector.java:1365)
> at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
