[
https://issues.apache.org/jira/browse/CASSANDRA-10270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731998#comment-14731998
]
Adam Bliss edited comment on CASSANDRA-10270 at 9/5/15 5:29 PM:
----------------------------------------------------------------
Here are some more observations.
1. I have disabled all writes to the cluster while I investigate this issue.
2. After I restart cassandra on a node, I see that it begins compacting. Just to
be sure, I run `nodetool enableautocompaction`, and I set the compaction
throughput to 0 (unlimited).
3. At this point, `nodetool compactionstats` shows:
pending tasks: 19
id compaction type keyspace
table completed total unit progress
35b1c1d0-534c-11e5-8ad9-0b1e51b9a459 Compaction megacrawl2
ranks_by_domain 2447918115 7453147913 bytes 32.84%
Active compaction remaining time : n/a
4. Half an hour later, compaction seems to have been progressing, and I see
these compactionstats:
pending tasks: 15
id compaction type keyspace
table completed total unit progress
4e5ee880-5350-11e5-8ad9-0b1e51b9a459 Compaction megacrawl2
ranks_by_domain 7424801306 7448824286 bytes 99.68%
Active compaction remaining time : n/a
5. But after this point, compaction ceases. From this point on, `nodetool
compactionstats` always shows `pending tasks: 0`. I ran it every 5 seconds for a
day. I even manually ran `nodetool compact` with no effect.
6. On my most recent run, it stopped with 368 sstables (down from about 1000
when I stopped writing to the cluster). It seems to be able to compact a hundred
or so on each restart. Here's a histogram of table sizes:
$ ls -SsHh *Data.db | cut -c 1-4 |uniq -c
4 2.5G 7 2.4G 10 2.3G 4 2.2G 1 2.0G 1 1.9G
1 884M 1 331M 1 329M 3 325M 3 323M 1 322M
2 321M 2 319M 1 318M 2 317M 2 315M 3 314M
2 313M 4 312M 6 311M 3 310M 2 309M 4 308M
3 307M 2 306M 2 305M 1 300M 15 88M 94 87M
59 86M 4 85M 8 84M 15 83M 25 82M 37 81M
23 80M 9 79M 1 51M
7. I am attching system.txt.gz, the logfile snippet from this period, where it
goes from
functionally compacting (with 15 pending tasks) to suddenly deciding there's
nothing more to do.
was (Author: abliss):
Here are some more observations.
1. I have disabled all writes to the cluster while I investigate this issue.
2. After I restart cassandra on a node, I see that it begins compacting. Just to
be sure, I run `nodetool enableautocompaction`, and I set the compaction
throughput to 0 (unlimited).
3. At this point, `nodetool compactionstats` shows:
pending tasks: 19
id compaction type keyspace
table completed total unit progress
35b1c1d0-534c-11e5-8ad9-0b1e51b9a459 Compaction megacrawl2
ranks_by_domain 2447918115 7453147913 bytes 32.84%
Active compaction remaining time : n/a
4. Half an hour later, compaction seems to have been progressing, and I see
these compactionstats:
pending tasks: 15
id compaction type keyspace
table completed total unit progress
4e5ee880-5350-11e5-8ad9-0b1e51b9a459 Compaction megacrawl2
ranks_by_domain 7424801306 7448824286 bytes 99.68%
Active compaction remaining time : n/a
5. But after this point, compaction ceases. From this point on, `nodetool
compactionstats` always shows `pending tasks: 0`. I ran it every 5 seconds for a
day. I even manually ran `nodetool compact` with no effect.
6. On my most recent run, it stopped with 368 sstables (down from about 1000
when I stopped writing to the cluster). It seems to be able to compact a hundred
or so on each restart. Here's a histogram of table sizes:
$ ls -SsHh *Data.db | cut -c 1-4 |uniq -c
4 2.5G 7 2.4G 10 2.3G 4 2.2G 1 2.0G 1 1.9G
1 884M 1 331M 1 329M 3 325M 3 323M 1 322M
2 321M 2 319M 1 318M 2 317M 2 315M 3 314M
2 313M 4 312M 6 311M 3 310M 2 309M 4 308M
3 307M 2 306M 2 305M 1 300M 15 88M 94 87M
59 86M 4 85M 8 84M 15 83M 25 82M 37 81M
23 80M 9 79M 1 51M
7. I am attching the logfile snippet from this period, where it goes from
functionally compacting (with 15 pending tasks) to suddenly deciding there's
nothing more to do.
> Cassandra stops compacting
> --------------------------
>
> Key: CASSANDRA-10270
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10270
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux (google cloud click-to-deploy, default settings, 3
> nodes)
> Reporter: Adam Bliss
> Fix For: 2.2.x
>
> Attachments: system.log.gz, system.txt.gz
>
>
> My cassandra cluster won't keep compacting. I notice that if I restart, it
> does compact for a while, but after a time it stops. As a result, after
> adding a bunch of rows, I ended up with about 1000 sstables per node.
> I'll attach more logs in a minute, but it seems like this might be the most
> relevant part:
> {noformat}
> INFO [CompactionExecutor:1] 2015-09-04 14:22:55,796
> CompactionManager.java:1433 - Compaction interrupted:
> Compaction@fff9bcd0-3b1f-11e5-8df6-33158d7bf3bf(megacrawl2, ranks_by_domain,
> 812501702/7543091905)bytes
> DEBUG [CompactionExecutor:1] 2015-09-04 14:22:55,797
> CompactionManager.java:1437 - Full interruption stack trace:
> org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction
> interrupted: Compaction@fff9bcd0-3b1f-11e5-8df6-33158d7bf3bf(megacrawl2,
> ranks_by_
> domain, 812501702/7543091905)bytes
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:180)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> ~[apache-cassandra-2.2.1.jar:2.2.1] at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:236)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_79]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_79]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_79]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
> DEBUG [CompactionExecutor:1] 2015-09-04 14:22:55,797
> CompactionManager.java:222 - Checking system.local
> DEBUG [CompactionExecutor:1] 2015-09-04
> 14:22:55,797 SizeTieredCompactionStrategy.java:85 - Compaction buckets are
> [[BigTableReader(path='/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/la-83-big-Data.db'),
>
> BigTableReader(path='/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/la-81-big-Data.db'),
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
