> -Xms8049M > -Xmx8049M > -Xmn800M That's a healthy amount of memory for the JVM.
If you are using Row Caches, reduce their size and/or ensure you are using Serializing (off heap) caches. Also consider changing the yaml conf flush_largest_memtables_at from 0.75 to 0.80 so it is different to the CMS occupancy setting. If you have a lot of rows, 100's of millions, consider reducing the bloom filter false positive ratio. Or just upgrade to 1.2 which uses less JVM memory. Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 9/02/2013, at 7:46 AM, André Cruz <andre.c...@co.sapo.pt> wrote: > Hello. > > I've noticed I get the frequent JVM warning in the logs about the heap being > full: > > WARN [ScheduledTasks:1] 2013-02-08 18:14:20,410 GCInspector.java (line 145) > Heap is 0.731554347747841 full. You may need to reduce memtable and/or cache > sizes. Cassandra will now flush up to the two largest memtables to free up > memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you > don't want Cassandra to do this automatically > WARN [ScheduledTasks:1] 2013-02-08 18:14:20,418 StorageService.java (line > 2855) Flushing CFS(Keyspace='Disco', ColumnFamily='FilesPerBlock') to relieve > memory pressure > INFO [ScheduledTasks:1] 2013-02-08 18:14:20,418 ColumnFamilyStore.java (line > 659) Enqueuing flush of Memtable-FilesPerBlock@1804403938(6275300/63189158 > serialized/live bytes, 52227 ops) > INFO [FlushWriter:4500] 2013-02-08 18:14:20,419 Memtable.java (line 264) > Writing Memtable-FilesPerBlock@1804403938(6275300/63189158 serialized/live > bytes, 52227 ops) > INFO [FlushWriter:4500] 2013-02-08 18:14:21,059 Memtable.java (line 305) > Completed flushing > /servers/storage/cassandra-data/Disco/FilesPerBlock/Disco-FilesPerBlock-he-6154-Data.db > (6332375 bytes) for commitlog position > ReplayPosition(segmentId=1357730625412, position=10756636) > WARN [ScheduledTasks:1] 2013-02-08 18:23:31,970 GCInspector.java (line 145) > Heap is 0.6835904101057064 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > WARN [ScheduledTasks:1] 2013-02-08 18:23:31,971 StorageService.java (line > 2855) Flushing CFS(Keyspace='Disco', ColumnFamily='BlocksKnownPerUser') to > relieve memory pressure > INFO [ScheduledTasks:1] 2013-02-08 18:23:31,972 ColumnFamilyStore.java (line > 659) Enqueuing flush of > Memtable-BlocksKnownPerUser@2072550435(1834642/60143054 serialized/live > bytes, 67010 ops) > INFO [FlushWriter:4501] 2013-02-08 18:23:31,972 Memtable.java (line 264) > Writing Memtable-BlocksKnownPerUser@2072550435(1834642/60143054 > serialized/live bytes, 67010 ops) > INFO [FlushWriter:4501] 2013-02-08 18:23:32,827 Memtable.java (line 305) > Completed flushing > /servers/storage/cassandra-data/Disco/BlocksKnownPerUser/Disco-BlocksKnownPerUser-he-484930-Data.db > (7404407 bytes) for commitlog position > ReplayPosition(segmentId=1357730625413, position=6093472) > WARN [ScheduledTasks:1] 2013-02-08 18:29:46,198 GCInspector.java (line 145) > Heap is 0.6871977390878024 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > WARN [ScheduledTasks:1] 2013-02-08 18:29:46,199 StorageService.java (line > 2855) Flushing CFS(Keyspace='Disco', ColumnFamily='FileRevision') to relieve > memory pressure > INFO [ScheduledTasks:1] 2013-02-08 18:29:46,200 ColumnFamilyStore.java (line > 659) Enqueuing flush of Memtable-FileRevision@1526026442(7245147/63711465 > serialized/live bytes, 23779 ops) > INFO [FlushWriter:4502] 2013-02-08 18:29:46,201 Memtable.java (line 264) > Writing Memtable-FileRevision@1526026442(7245147/63711465 serialized/live > bytes, 23779 ops) > INFO [FlushWriter:4502] 2013-02-08 18:29:46,769 Memtable.java (line 305) > Completed flushing > /servers/storage/cassandra-data/Disco/FileRevision/Disco-FileRevision-he-5438-Data.db > (5480642 bytes) for commitlog position > ReplayPosition(segmentId=1357730625413, position=29816878) > INFO [ScheduledTasks:1] 2013-02-08 18:34:13,442 GCInspector.java (line 122) > GC for ConcurrentMarkSweep: 352 ms for 1 collections, 5902597760 used; max is > 8357150720 > WARN [ScheduledTasks:1] 2013-02-08 18:34:13,442 GCInspector.java (line 145) > Heap is 0.7062930845406603 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > WARN [ScheduledTasks:1] 2013-02-08 18:34:13,443 StorageService.java (line > 2855) Flushing CFS(Keyspace='Disco', ColumnFamily='NamespaceFile') to relieve > memory pressure > INFO [ScheduledTasks:1] 2013-02-08 18:34:13,443 ColumnFamilyStore.java (line > 659) Enqueuing flush of Memtable-NamespaceFile@1719163849(14395839/64692951 > serialized/live bytes, 51765 ops) > INFO [FlushWriter:4503] 2013-02-08 18:34:13,446 Memtable.java (line 264) > Writing Memtable-NamespaceFile@1719163849(14395839/64692951 serialized/live > bytes, 51765 ops) > INFO [FlushWriter:4503] 2013-02-08 18:34:14,173 Memtable.java (line 305) > Completed flushing > /servers/storage/cassandra-data/Disco/NamespaceFile/Disco-NamespaceFile-he-5516-Data.db > (7747535 bytes) for commitlog position > ReplayPosition(segmentId=1357730625414, position=14940209) > WARN [ScheduledTasks:1] 2013-02-08 18:37:00,593 GCInspector.java (line 145) > Heap is 0.6828223856659127 full. You may need to reduce memtable and/or > cache sizes. Cassandra will now flush up to the two largest memtables to > free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > WARN [ScheduledTasks:1] 2013-02-08 18:37:00,593 StorageService.java (line > 2855) Flushing CFS(Keyspace='Disco', ColumnFamily='RevisionLog') to relieve > memory pressure > INFO [ScheduledTasks:1] 2013-02-08 18:37:00,594 ColumnFamilyStore.java (line > 659) Enqueuing flush of Memtable-RevisionLog@726433503(12434137/65978509 > serialized/live bytes, 59638 ops) > INFO [FlushWriter:4504] 2013-02-08 18:37:00,596 Memtable.java (line 264) > Writing Memtable-RevisionLog@726433503(12434137/65978509 serialized/live > bytes, 59638 ops) > INFO [FlushWriter:4504] 2013-02-08 18:37:01,523 Memtable.java (line 305) > Completed flushing > /servers/storage/cassandra-data/Disco/RevisionLog/Disco-RevisionLog-he-1664-Data.db > (2366787 bytes) for commitlog position > ReplayPosition(segmentId=1357730625414, position=27626400) > > > Is this healthy GC behaviour? Or should I adjust the memory/gc settings? This > machine has 32GB RAM and my current JVM settings are: > > -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 > -Xms8049M > -Xmx8049M > -Xmn800M > -XX:+HeapDumpOnOutOfMemoryError > -Xss196k > -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC > -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly > > Thanks, > André Cruz > >
