[
https://issues.apache.org/jira/browse/CASSANDRA-20610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094935#comment-18094935
]
Michael Semb Wever edited comment on CASSANDRA-20610 at 7/9/26 9:07 AM:
------------------------------------------------------------------------
[~chrisjmiller] , the 20251014 thread dump (looks to me) matches
[[email protected]]'s observations: 31/35 MutationStage threads
parked in {{{}SubAllocator.allocate(MemtableAllocator.java:195){}}}, both
MemtableFlushWriter threads at {{writeBarrier.await()}}
({{{}Flush.run:1193{}}}), MemtablePostFlush stuck.
But [[email protected]]… IIUC, shouldn't
{{writeBarrier.markBlocking()}} (called immediately before {{await()}} in
{{{}Flush.run{}}}) release your {_}Thread-1{_}? (i.e. what [~benedict] writes
here)
If so, then the pre-barrier {{Group}} parked at line 195 should be woken via
{{isBlockingSignal}} and permitted to exceed the memory limit, and a
post-barrier {{Group}} isn't waited on by the barrier. So maybe we would still
require one of (a) a lost wakeup in the markBlocking path, (b) a leaked
{{Group}} with no thread behind it, or (c) a pre-barrier op parked somewhere
{{markBlocking()}} that can't be reached ??
Possibly relevant to (c): the only write-path thread in the 20251014 dump
outside the allocator is {{{}read-hotness-tracker:1{}}}, holding an open write
{{Group}} inside {{beginWrite}} → {{CommitLog.add}} →
{{PeriodicCommitLogService.maybeWaitForSync}} → {{awaitSyncAt}} . (We've
witnessed elsewhere something similar with {{BatchCommitLogService}} too.)
[~chrisjmiller] , [[email protected]] , could it help with a heap
dump, to walk the {{Keyspace.writeOrder}} {{Group}} chain to see which
{{Group}} the barrier is waiting on ?
was (Author: michaelsembwever):
[~chrisjmiller] , the 20251014 dump (looks to me) matches
[[email protected]]'s observations: 31/35 MutationStage threads
parked in {{{}SubAllocator.allocate(MemtableAllocator.java:195){}}}, both
MemtableFlushWriter threads at {{writeBarrier.await()}}
({{{}Flush.run:1193{}}}), MemtablePostFlush stuck.
But [[email protected]]… IIUC, shouldn't
{{writeBarrier.markBlocking()}} (called immediately before {{await()}} in
{{{}Flush.run{}}}) release your {_}Thread-1{_}? (i.e. what [~benedict] writes
here)
If so, then the pre-barrier {{Group}} parked at line 195 should be woken via
{{isBlockingSignal}} and permitted to exceed the memory limit, and a
post-barrier {{Group}} isn't waited on by the barrier. So maybe we would still
require one of (a) a lost wakeup in the markBlocking path, (b) a leaked
{{Group}} with no thread behind it, or (c) a pre-barrier op parked somewhere
{{markBlocking()}} that can't be reached ??
Possibly relevant to (c): the only write-path thread in this dump outside the
allocator is {{{}read-hotness-tracker:1{}}}, holding an open write {{Group}}
inside {{beginWrite}} → {{CommitLog.add}} →
{{PeriodicCommitLogService.maybeWaitForSync}} → {{awaitSyncAt}} . (We've
witnessed elsewhere something similar with {{BatchCommitLogService}} too.)
[~chrisjmiller] , [[email protected]] , would it help with a heap
dump, to walk the {{Keyspace.writeOrder}} {{Group}} chain to see which
{{Group}} the barrier is waiting on ?
> nodetool unreachable shows all other instances down
> ---------------------------------------------------
>
> Key: CASSANDRA-20610
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20610
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Chris Miller
> Priority: Normal
> Attachments: CASSANDRA-20610_cassandra.yaml,
> cassandra_thread_dump_20250530.txt, cassandra_thread_dump_20251014.txt
>
>
> see first comment
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
