[
https://issues.apache.org/jira/browse/CASSANDRA-19576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114515#comment-18114515
]
Mykyta Bozhenko commented on CASSANDRA-19576:
---------------------------------------------
I reproduced the stall and then withdrew my patch (PR #5141, closed); posting
the analysis so it is not lost. No patch proposed.
Reproduction: memtable pool squeezed to 1 MiB, flush blocked with Byteman so it
can never be reclaimed, a filler thread exhausts the pool, then a major
compaction. On current trunk the compaction does not finish within 45 s and the
compaction thread is parked in the history write:
BLOCKED thread=CompactionExecutor:4 state=WAITING in_memtable_allocation=true
at Awaitable$Defaults.awaitThrowUncheckedOnInterrupt
at MemtableAllocator$SubAllocator.allocate(MemtableAllocator.java:196)
at SlabAllocator.allocate
Why "make it non-blocking" was not enough. I moved the write to a dedicated
single-threaded executor with a bounded queue. The compaction is freed (about
200 ms instead of not finishing), but the history is still not written: that
writer thread blocks on the same pool, so deferring moves the dependency rather
than removing it. It also leaves teardown to decide what to do with an
unfinishable diagnostic write, and under sustained pressure the bounded queue
fills and history stops being recorded entirely, which is exactly when an
operator would look at it.
Compaction frees resources, yet its last step allocates from the pool it is
relieving, exhausted mostly by user tables. A fix that holds up needs the
record to not depend on that pool: recorded outside the CQL write path, or
system-keyspace writes given a reserve independent of user-table pressure.
ColumnFamilyStore.Flush.run already has a narrow reserve for writes the flush
barrier is waiting on, but extending it to diagnostics would give a reserve to
the least critical work in the system. That is a design call for the owners of
this area, which is why I closed the PR.
> blocking on memtable allocations blocks compactions from completing
> -------------------------------------------------------------------
>
> Key: CASSANDRA-19576
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19576
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Reporter: Jon Haddad
> Priority: Normal
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {{updateCompactionHistory}} writes to compaction_history in a blocking
> manner. If we've also run out of memtable memory, this call will block and
> compaction falls behind.
> It might make sense to have this be non blocking. We'd risk having a missing
> entry in nodetool getcompactionhistory, but I think that's preferable to
> having a dead node.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
