[jira] [Updated] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hanna updated CASSANDRA-13943: - Labels: jbod-aware-compaction (was: ) > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson >Priority: Major > Labels: jbod-aware-compaction > Attachments: cassandra-jstack-2017-10-12-infinite-sstable-adding.txt, > cassandra-jstack-2017-10-12.txt, cassandra.yaml, debug.log, > debug.log-with-commit-d8f3f2780, debug.log.1.zip, debug.log.zip, jvm.options > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Kinder updated CASSANDRA-13943:
---
Attachment: cassandra.yaml
jvm.options
debug.log.zip
debug.log.1.zip
cassandra-jstack-2017-10-12-infinite-sstable-adding.txt
No, as far as I can tell no infinite compactions.
I'm doing several hundred mutation per second for one of the tables.
Attached debug logs and configuration. debug.log.1.zip contains node startup.
I also just started seeing some crazy behavior on a node I killed and
restarted... it was adding the same SSTables over and over endlessly:
{noformat}
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,109 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,110 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,110 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,110 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,111 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,111 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
{noformat}
I took a quick jstack before killing it, attached as
cassandra-jstack-2017-10-12-infinite-sstable-adding.txt
> Infinite compaction of L0 SSTables in JBOD
> --
>
> Key: CASSANDRA-13943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.11.0 / Centos 6
>Reporter: Dan Kinder
>Assignee: Marcus Eriksson
> Attachments: cassandra-jstack-2017-10-12-infinite-sstable-adding.txt,
> cassandra-jstack-2017-10-12.txt, cassandra.yaml, debug.log,
> debug.log-with-commit-d8f3f2780, debug.log.1.zip, debug.log.zip, jvm.options
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over.
> Attaching logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It
> does create new SSTables but immediately recompacts again. Note that I am not
> inserting anything at the moment, there is no flushing happening on this
> table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates.
> But they shouldn't be, they are on different disks and I ran nodetool
> relocatesstables as well as nodetool compact. So, it tries to compact them
> together, but the compaction results in the exact same 2 SSTables on the 2
> disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU
> doing this for multiple tables, it seems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Kinder updated CASSANDRA-13943: --- Attachment: cassandra-jstack-2017-10-12.txt [~krummas] after a night I am seeing most nodes start to build up MemtablePostFlush Pending tasks (up to several hundred) and MemtableFlushWriter Pending tasks (less so than MemtablePostFlush). Several nodes have blocked on MutationStage and grown lots of pending tasks in that, I assume because flushing is blocked and there's nowhere else to write. Attached a stack dump from one of these heavily blocked nodes. > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: cassandra-jstack-2017-10-12.txt, debug.log, > debug.log-with-commit-d8f3f2780 > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Kinder updated CASSANDRA-13943: --- Attachment: debug.log-with-commit-d8f3f2780 Okay, stood up a node on this commit: https://github.com/krummas/cassandra/commit/d8f3f2780fbb9e789f90b095d1f109ebb16f46ff and attached the log. > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder >Assignee: Marcus Eriksson > Attachments: debug.log, debug.log-with-commit-d8f3f2780 > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13943) Infinite compaction of L0 SSTables in JBOD
[ https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Kinder updated CASSANDRA-13943: --- Attachment: debug.log > Infinite compaction of L0 SSTables in JBOD > -- > > Key: CASSANDRA-13943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13943 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.11.0 / Centos 6 >Reporter: Dan Kinder > Attachments: debug.log > > > I recently upgraded from 2.2.6 to 3.11.0. > I am seeing Cassandra loop infinitely compacting the same data over and over. > Attaching logs. > It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It > does create new SSTables but immediately recompacts again. Note that I am not > inserting anything at the moment, there is no flushing happening on this > table (Memtable switch count has not changed). > My theory is that it somehow thinks those should be compaction candidates. > But they shouldn't be, they are on different disks and I ran nodetool > relocatesstables as well as nodetool compact. So, it tries to compact them > together, but the compaction results in the exact same 2 SSTables on the 2 > disks, because the keys are split by data disk. > This is pretty serious, because all our nodes right now are consuming CPU > doing this for multiple tables, it seems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
