[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338466#comment-14338466 ]
Benedict commented on CASSANDRA-8860: ------------------------------------- {quote} "CompactionExecutor:14" daemon prio=10 tid=0x00002aaace7a7800 nid=0x7e31 runnable [0x0000000042a79000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.transfer(HashMap.java:605) at java.util.HashMap.resize(HashMap.java:585) at java.util.HashMap.addEntry(HashMap.java:883) at java.util.HashMap.put(HashMap.java:509) at java.util.HashSet.add(HashSet.java:217) at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.createOverlapChain(SizeTieredCompactionStrategy.java:235) at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.filterColdSSTables(SizeTieredCompactionStrategy.java:202) at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundSSTables(SizeTieredCompactionStrategy.java:85) at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundTask(SizeTieredCompactionStrategy.java:320) - locked <0x000000064f254560> (a org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy) at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:72) - locked <0x000000064f254500> (a org.apache.cassandra.db.compaction.WrappingCompactionStrategy) at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:234) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) {quote} Looks like a viable candidate, and was introduced recently. I can't see anything fundamentally broken about it, although it does have O(n^2) memory utilisation in the number of sstables. This would require around 8500 sstables to be involved in the compaction to create the current heap pressure, which is both: 1) possible for some users (so probably we need to rethink); and 2) also pretty uncommon? so perhaps not the issue here? Could we get a list of the data files present on disk for this table, your log file, and perhaps a heap dump? > Too many java.util.HashMap$Entry objects in heap > ------------------------------------------------ > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 > Reporter: Phil Yang > Attachments: cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)