Also it might be: https://issues.apache.org/jira/browse/CASSANDRA-6541
That is causing the high heap. -Jeremiah On Feb 18, 2014, at 5:01 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > Sounds like you have CMSInitiatingOccupancyFraction set close to 60. > You can raise that and/or figure out how to use less heap. > > On Mon, Feb 17, 2014 at 5:06 PM, John Pyeatt <john.pye...@singlewire.com> > wrote: >> I have a 6 node cluster running on AWS. We are using m1.large instances with >> heap size set to 3G. >> >> 5 of the 6 nodes seem quite healthy. The 6th one however is running >> GCInspector GC for ConcurrentMarkSweep every 15 seconds or so. There is >> nothing going on on this box. No repairs and almost not user activity. But >> the CPU is almost continuously at 50% or more. >> >> The only message in the log at all is the >> INFO 2014-02-17 22:58:53,429 [ScheduledTasks:1] GCInspector GC for >> ConcurrentMarkSweep: 213 ms for 1 collections, 1964940024 used; max is >> 3200253952 >> INFO 2014-02-17 22:59:07,431 [ScheduledTasks:1] GCInspector GC for >> ConcurrentMarkSweep: 250 ms for 1 collections, 1983269488 used; max is >> 3200253952 >> INFO 2014-02-17 22:59:21,522 [ScheduledTasks:1] GCInspector GC for >> ConcurrentMarkSweep: 280 ms for 1 collections, 1998214480 used; max is >> 3200253952 >> INFO 2014-02-17 22:59:36,527 [ScheduledTasks:1] GCInspector GC for >> ConcurrentMarkSweep: 305 ms for 1 collections, 2013065592 used; max is >> 3200253952 >> INFO 2014-02-17 22:59:50,529 [ScheduledTasks:1] GCInspector GC for >> ConcurrentMarkSweep: 334 ms for 1 collections, 2028069232 used; max is >> 3200253952 >> >> We don't see any of these messages on the other nodes in the cluster. >> >> We are seeing similar behaviour for both our production and QA clusters. >> Production is running cassandra 1.2.9 and QA is running 1.2.13. >> >> Here are some of the cassandra settings that I would think might be >> relevant. >> >> flush_largest_memtables_at: 0.75 >> reduce_cache_sizes_at: 0.85 >> reduce_cache_capacity_to: 0.6 >> in_memory_compaction_limit_in_mb: 64 >> >> Does anyone have any ideas why we are seeing this so selectively on one box? >> >> Any cures??? >> -- >> John Pyeatt >> Singlewire Software, LLC >> www.singlewire.com >> ------------------ >> 608.661.1184 >> john.pye...@singlewire.com > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder, http://www.datastax.com > @spyced
