[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202237#comment-16202237
]
Dan Kinder edited comment on CASSANDRA-13943 at 10/12/17 5:24 PM:
------------------------------------------------------------------
[~krummas] after a night I am seeing most nodes start to build up
MemtablePostFlush Pending tasks (up to several hundred) and MemtableFlushWriter
Pending tasks (less so than MemtablePostFlush). Several nodes have blocked on
MutationStage and grown lots of pending tasks in that, I assume because
flushing is blocked and there's nowhere else to write.
Attached a stack dump from one of these heavily blocked nodes.
Also notable, since it seems like the blocking is in
{{LeveledManifest.getCompactionCandidates()}}, here is what the SSTable levels
look like for the biggest table:
52 SSTable Level: 0
75 SSTable Level: 1
122 SSTable Level: 2
360 SSTable Level: 3
2724 SSTable Level: 4
10647 SSTable Level: 5
Several other tables have comparable number of SSTables in L0 (20-40 or so),
but those only have a few hundred or a few thousand SSTables.
was (Author: dkinder):
[~krummas] after a night I am seeing most nodes start to build up
MemtablePostFlush Pending tasks (up to several hundred) and MemtableFlushWriter
Pending tasks (less so than MemtablePostFlush). Several nodes have blocked on
MutationStage and grown lots of pending tasks in that, I assume because
flushing is blocked and there's nowhere else to write.
Attached a stack dump from one of these heavily blocked nodes.
> Infinite compaction of L0 SSTables in JBOD
> ------------------------------------------
>
> Key: CASSANDRA-13943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.11.0 / Centos 6
> Reporter: Dan Kinder
> Assignee: Marcus Eriksson
> Attachments: cassandra-jstack-2017-10-12.txt, debug.log,
> debug.log-with-commit-d8f3f2780
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over.
> Attaching logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It
> does create new SSTables but immediately recompacts again. Note that I am not
> inserting anything at the moment, there is no flushing happening on this
> table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates.
> But they shouldn't be, they are on different disks and I ran nodetool
> relocatesstables as well as nodetool compact. So, it tries to compact them
> together, but the compaction results in the exact same 2 SSTables on the 2
> disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU
> doing this for multiple tables, it seems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
