[jira] [Updated] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-8860: --- Reviewer: Tyler Hobbs (was: Benedict) > Too many java.util.HashMap$Entry objects in heap > > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 >Reporter: Phil Yang >Assignee: Marcus Eriksson > Fix For: 2.1.4 > > Attachments: 0001-remove-cold_reads_to_omit.patch, 8860-v2.txt, > 8860.txt, cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt, > jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-8860: --- Attachment: 0001-remove-cold_reads_to_omit.patch patch to remove option attached keeping the option in 2.1 if anyone has automated creating tables etc, but will remove entirely in 3.0 > Too many java.util.HashMap$Entry objects in heap > > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.1.4 > > Attachments: 0001-remove-cold_reads_to_omit.patch, 8860-v2.txt, > 8860.txt, cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt, > jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated CASSANDRA-8860: - Attachment: jstat-before.txt jstat-afterv1.txt jstat-afterv2.txt 8860-v2.txt I upload a new patch according to my new algorithm. We can compare three jstat outputs with different versions' code in my cluster and see that v1 will not exhaust the heap but still have high gc pressure, while v2's gc pressure is much more lower. > Too many java.util.HashMap$Entry objects in heap > > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 >Reporter: Phil Yang >Assignee: Marcus Eriksson > Fix For: 2.1.4 > > Attachments: 8860-v2.txt, 8860.txt, cassandra-env.sh, cassandra.yaml, > jmap.txt, jstack.txt, jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated CASSANDRA-8860: - Attachment: 8860.txt I submit a patch to fix this issue. The idea is very simple: keep only one overlapChain in heap whose estimateCompactionGain < 0.7 and has the max size from any other overlapChains. > Too many java.util.HashMap$Entry objects in heap > > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 >Reporter: Phil Yang >Assignee: Marcus Eriksson > Fix For: 2.1.4 > > Attachments: 8860.txt, cassandra-env.sh, cassandra.yaml, jmap.txt, > jstack.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-8860: --- Fix Version/s: 2.1.4 > Too many java.util.HashMap$Entry objects in heap > > > Key: CASSANDRA-8860 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.1.3, jdk 1.7u51 >Reporter: Phil Yang >Assignee: Marcus Eriksson > Fix For: 2.1.4 > > Attachments: cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt > > > While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have > GC issue after the node restarting successfully. Old gen grows very fast and > most of the space can not be recycled after setting its status to normal > immediately. The qps of both reading and writing are very low and there is no > heavy compaction. > Jmap result seems strange that there are too many java.util.HashMap$Entry > objects in heap, where in my experience the "[B" is usually the No1. > If I downgrade it to 2.1.1, this issue will not appear. > I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if > someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
