[
https://issues.apache.org/jira/browse/CASSANDRA-13943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Kinder updated CASSANDRA-13943:
-----------------------------------
Attachment: cassandra.yaml
jvm.options
debug.log.zip
debug.log.1.zip
cassandra-jstack-2017-10-12-infinite-sstable-adding.txt
No, as far as I can tell no infinite compactions.
I'm doing several hundred mutation per second for one of the tables.
Attached debug logs and configuration. debug.log.1.zip contains node startup.
I also just started seeing some crazy behavior on a node I killed and
restarted... it was adding the same SSTables over and over endlessly:
{noformat}
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,109 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,110 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,110 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,110 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,111 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
INFO [CompactionExecutor:49] 2017-10-12 11:26:03,111 LeveledManifest.java:474
- Adding high-level (L5)
BigTableReader(path='/srv/disk11/cassandra-data/walker/links-a3d582c0a6683450a030de61dc4b9823/mc-1237377-big-Data.db')
to candidates
{noformat}
I took a quick jstack before killing it, attached as
cassandra-jstack-2017-10-12-infinite-sstable-adding.txt
> Infinite compaction of L0 SSTables in JBOD
> ------------------------------------------
>
> Key: CASSANDRA-13943
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13943
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Cassandra 3.11.0 / Centos 6
> Reporter: Dan Kinder
> Assignee: Marcus Eriksson
> Attachments: cassandra-jstack-2017-10-12-infinite-sstable-adding.txt,
> cassandra-jstack-2017-10-12.txt, cassandra.yaml, debug.log,
> debug.log-with-commit-d8f3f2780, debug.log.1.zip, debug.log.zip, jvm.options
>
>
> I recently upgraded from 2.2.6 to 3.11.0.
> I am seeing Cassandra loop infinitely compacting the same data over and over.
> Attaching logs.
> It is compacting two tables, one on /srv/disk10, the other on /srv/disk1. It
> does create new SSTables but immediately recompacts again. Note that I am not
> inserting anything at the moment, there is no flushing happening on this
> table (Memtable switch count has not changed).
> My theory is that it somehow thinks those should be compaction candidates.
> But they shouldn't be, they are on different disks and I ran nodetool
> relocatesstables as well as nodetool compact. So, it tries to compact them
> together, but the compaction results in the exact same 2 SSTables on the 2
> disks, because the keys are split by data disk.
> This is pretty serious, because all our nodes right now are consuming CPU
> doing this for multiple tables, it seems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
