[ https://issues.apache.org/jira/browse/CASSANDRA-9681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606057#comment-14606057 ]
mlowicki commented on CASSANDRA-9681: ------------------------------------- It started ~ 04.06 (at about the same time on all affected boxes - https://www.dropbox.com/s/9c6p2xdmncktbnu/Screenshot%202015-06-29%2020.16.02.png?dl=0, https://www.dropbox.com/s/gs8bztzr394icz0/Screenshot%202015-06-29%2020.16.24.png?dl=0). Will attach logs soon. > Memtable heap size grows and many long GC pauses are triggered > -------------------------------------------------------------- > > Key: CASSANDRA-9681 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9681 > Project: Cassandra > Issue Type: Bug > Environment: C* 2.1.7, Debian Wheezy > Reporter: mlowicki > Assignee: Benedict > Priority: Critical > Fix For: 2.1.x > > Attachments: cassandra.yaml > > > C* 2.1.7 cluster is behaving really bad after 1-2 days. > {{gauges.cassandra.jmx.org.apache.cassandra.metrics.ColumnFamily.AllMemtablesHeapSize.Value}} > jumps to 7 GB > (https://www.dropbox.com/s/vraggy292erkzd2/Screenshot%202015-06-29%2019.12.53.png?dl=0) > on 3/6 nodes in each data center and then there are many long GC pauses. > Cluster is using default heap size values ({{-Xms8192M -Xmx8192M -Xmn2048M}}) > Before C* 2.1.5 memtables heap size was basically constant ~500MB > (https://www.dropbox.com/s/fjdywik5lojstvn/Screenshot%202015-06-29%2019.30.00.png?dl=0) > After restarting all nodes is behaves stable for 1-2days. Today I've done > that and long GC pauses are gone (~18:00 > https://www.dropbox.com/s/7vo3ynz505rsfq3/Screenshot%202015-06-29%2019.28.37.png?dl=0). > The only pattern we've found so far is that long GC pauses are happening > basically at the same time on all nodes in the same data center - even on the > ones where memtables heap size is not growing. > Cliffs on the graphs are nodes restarts. > Used memory on boxes where {{AllMemtabelesHeapSize}} grows, stays at the same > level - > https://www.dropbox.com/s/tes9abykixs86rf/Screenshot%202015-06-29%2019.37.52.png?dl=0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)