[
https://issues.apache.org/jira/browse/CASSANDRA-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson reassigned CASSANDRA-13923:
-------------------------------------------
Assignee: Marcus Eriksson
> Flushers blocked due to many SSTables
> -------------------------------------
>
> Key: CASSANDRA-13923
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13923
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction, Local Write-Read Paths
> Environment: Cassandra 3.11.0
> Centos 6 (downgraded JNA)
> 64GB RAM
> 12-disk JBOD
> Reporter: Dan Kinder
> Assignee: Marcus Eriksson
> Attachments: cassandra-jstack-readstage.txt, cassandra-jstack.txt
>
>
> This started on the mailing list and I'm not 100% sure of the root cause,
> feel free to re-title if needed.
> I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving
> traffic, thread pools begin to back up and grow pending tasks indefinitely.
> This happens to multiple different stages (Read, Mutation) and consistently
> builds pending tasks for MemtablePostFlush and MemtableFlushWriter.
> Using jstack shows that there is blocking going on when trying to call
> getCompactionCandidates, which seems to happen on flush. We have fairly large
> nodes that have ~15,000 SSTables per node, all LCS.
> I seems like this can cause reads to get blocked because they try to acquire
> a read lock when calling shouldDefragment.
> And writes, of course, block once we can't allocate anymore memtables,
> because flushes are backed up.
> We did not have this problem in 2.2.6, so it seems like there is some
> regression causing it to be incredibly slow trying to do calls like
> getCompactionCandidates that list out the SSTables.
> In our case this causes nodes to build up pending tasks and simply stop
> responding to requests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
