[
https://issues.apache.org/jira/browse/CASSANDRA-10829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062136#comment-15062136
]
Marcus Eriksson commented on CASSANDRA-10829:
---------------------------------------------
Problem is that we use the sstables marked as 'compacting' to create the
canonical view (to get the original sstable instances) and when running
cleanup, scrub, upgradesstables and anticompaction we mark each sstable as
compacted as we finish it, but we keep it as compacting until the entire
operation is done. This means that we have sstables which are marked as
compacted in the CANONICAL_SSTABLES and we cannot reference them and it makes
us log the 'spinning ... ' message.
There is also an issue with the log rate limiting, it was meant to only log
every 100ms, but it logs every iteration.
Patch that unmarks compacting once the sstable finished and fixes the rate
limiting:
https://github.com/krummas/cassandra/commits/marcuse/10829
and tests:
https://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-10829-dtest/
https://cassci.datastax.com/view/Dev/view/krummas/job/krummas-marcuse-10829-testall/
seems only the rate limiting fix applies to 2.2+
> cleanup + repair generates a lot of logs
> ----------------------------------------
>
> Key: CASSANDRA-10829
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10829
> Project: Cassandra
> Issue Type: Bug
> Environment: 5 nodes on Cassandra 2.1.11 (on Debian)
> Reporter: Fabien Rousseau
> Assignee: Marcus Eriksson
>
> One of our node generates a lot of cassandra logs (int the 10 MB/s) and CPU
> usage has increased (by a factor 2-3).
> This was most probably triggered by a "nodetool snapshot" while a cleanup was
> already running on this node.
> An example of those logs:
> 2015-12-08 09:15:17,794 INFO
> [ValidationExecutor:689]ColumnFamilyStore.java:1923 Spinning trying to
> capture released readers [...]
> 2015-12-08 09:15:17,794 INFO
> [ValidationExecutor:689]ColumnFamilyStore.java:1924 Spinning trying to
> capture all readers [...]
> 2015-12-08 09:15:17,795 INFO
> [ValidationExecutor:689]ColumnFamilyStore.java:1923 Spinning trying to
> capture released readers [...]
> 2015-12-08 09:15:17,795 INFO
> [ValidationExecutor:689]ColumnFamilyStore.java:1924 Spinning trying to
> capture all readers [...]
> (I removed SSTableReader information because it's rather long... I can share
> it privately if needed)
> Note that the date has not been changed (only 1ms between logs)
> It should not generate that gigantic amount of logs :)
> This is probably linked to:
> https://issues.apache.org/jira/browse/CASSANDRA-9637
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
