[
https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338466#comment-14338466
]
Benedict edited comment on CASSANDRA-8860 at 2/26/15 2:37 PM:
--------------------------------------------------------------
bq. it sounds like the cache code
It's a regular HashMap, not a CLHM, causing this heap pressure.
{quote}
"CompactionExecutor:14" daemon prio=10 tid=0x00002aaace7a7800 nid=0x7e31
runnable [0x0000000042a79000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.transfer(HashMap.java:605)
at java.util.HashMap.resize(HashMap.java:585)
at java.util.HashMap.addEntry(HashMap.java:883)
at java.util.HashMap.put(HashMap.java:509)
at java.util.HashSet.add(HashSet.java:217)
at
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.createOverlapChain(SizeTieredCompactionStrategy.java:235)
at
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.filterColdSSTables(SizeTieredCompactionStrategy.java:202)
at
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundSSTables(SizeTieredCompactionStrategy.java:85)
at
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundTask(SizeTieredCompactionStrategy.java:320)
- locked <0x000000064f254560> (a
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy)
at
org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:72)
- locked <0x000000064f254500> (a
org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:234)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{quote}
Looks like a viable candidate, and was introduced recently. I can't see
anything fundamentally broken about it, although it does have O(n^2) memory
utilisation in the number of sstables. This would require around 8500 sstables
to be considered for compaction to create the current heap pressure, which is
both: 1) possible for some users (so probably we need to rethink); and 2) also
pretty uncommon? so perhaps not the issue here?
Could we get a list of the data files present on disk for this table, your log
file, and perhaps a heap dump?
was (Author: benedict):
{quote}
"CompactionExecutor:14" daemon prio=10 tid=0x00002aaace7a7800 nid=0x7e31
runnable [0x0000000042a79000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.transfer(HashMap.java:605)
at java.util.HashMap.resize(HashMap.java:585)
at java.util.HashMap.addEntry(HashMap.java:883)
at java.util.HashMap.put(HashMap.java:509)
at java.util.HashSet.add(HashSet.java:217)
at
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.createOverlapChain(SizeTieredCompactionStrategy.java:235)
at
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.filterColdSSTables(SizeTieredCompactionStrategy.java:202)
at
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundSSTables(SizeTieredCompactionStrategy.java:85)
at
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundTask(SizeTieredCompactionStrategy.java:320)
- locked <0x000000064f254560> (a
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy)
at
org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:72)
- locked <0x000000064f254500> (a
org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:234)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{quote}
Looks like a viable candidate, and was introduced recently. I can't see
anything fundamentally broken about it, although it does have O(n^2) memory
utilisation in the number of sstables. This would require around 8500 sstables
to be involved in the compaction to create the current heap pressure, which is
both: 1) possible for some users (so probably we need to rethink); and 2) also
pretty uncommon? so perhaps not the issue here?
Could we get a list of the data files present on disk for this table, your log
file, and perhaps a heap dump?
> Too many java.util.HashMap$Entry objects in heap
> ------------------------------------------------
>
> Key: CASSANDRA-8860
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8860
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.3, jdk 1.7u51
> Reporter: Phil Yang
> Attachments: cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt
>
>
> While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have
> GC issue after the node restarting successfully. Old gen grows very fast and
> most of the space can not be recycled after setting its status to normal
> immediately. The qps of both reading and writing are very low and there is no
> heavy compaction.
> Jmap result seems strange that there are too many java.util.HashMap$Entry
> objects in heap, where in my experience the "[B" is usually the No1.
> If I downgrade it to 2.1.1, this issue will not appear.
> I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if
> someone need it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
