[ https://issues.apache.org/jira/browse/CASSANDRA-11025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Yeksigian resolved CASSANDRA-11025. ---------------------------------------- Resolution: Duplicate Fix Version/s: 2.0.17 This appears to be a dupe of CASSANDRA-9662, which was fixed in 2.0.17. Note that 2.0 is EOL, so no new patches or releases will be made. If you try with 2.1 and find the same issue, please reopen. > Too many compactions on certain node when too many empty tables are created > --------------------------------------------------------------------------- > > Key: CASSANDRA-11025 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11025 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 4 nodes with 24 core cpu and 32G ram, CentOS 6.5. Each > node is configured with 8G heap size. > Reporter: Shuo Chen > Fix For: 2.0.17 > > > I have configured 4 nodes of cassandra cluster version 2.0.16. Each node has > about 10G load. One of the node has too many pending compactions not too long > and too many full gc after it restarts. > Here is part of gc histogram: > num #instances #bytes class name > ---------------------------------------------- > 1: 67758530 2168272960 java.util.concurrent.FutureTask > 2: 67759745 1626233880 > java.util.concurrent.Executors$RunnableAdapter > 3: 67758576 1626205824 > java.util.concurrent.LinkedBlockingQueue$Node > 4: 67758529 1626204696 > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask > 5: 16935 72995576 [B > 6: 240534 11545632 java.nio.HeapByteBuffer > 7: 37416 5969800 [C > 8: 41447 5624856 <constMethodKlass> > 9: 41447 5315504 <methodKlass> > 10: 104850 5032800 > edu.stanford.ppl.concurrent.SnapTreeMap$Node > 11: 4110 4564144 <constantPoolKlass> > 12: 104781 3352992 org.apache.cassandra.db.Column > 13: 4110 2824016 <instanceKlassKlass> > Here is the nodetool stats: > [cassandra@whaty181 apache-cassandra-2.0.16]$ bin/nodetool compactionstats > pending tasks: 64642341 > Active compaction remaining time : n/a > However in system.log, there are not too much log of compaction. I used > inotify to monitor the data directory events. There are few events when > pending tasks accumulate. > I used jmap to dump the heap and analyzed the > java.util.concurrent.FutureTask. It contains many task of CompactionExecutor. > I checked the cf of CompactionExecutor. However most of the cfs are never > inserted any data since created. > I have created 6 keyspaces and 5 keyspaces never inserted any data. In these > 5 keyspaces, 2 keyspaces contain 42 cfs each and 3 keyspaces contains 6 cfs > in total. So there are about 90 empty cfs. > All of these cfs are super column families created using casandra-cli > After I dropped these 5 keyspaces and restarted this node, the compaction > status of this node is normal again. So I suggests there is many some bug > concerning compaction on empty columnfamily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)