To amplify on this, the collections before tenuring are copying collections and thus immune to fragmentation. CMS, however, stands for Concurrent Mark Sweep and mark-sweep collectors are subject to fragmentation. When fragmentation causes a failure to allocate, then you get a full copying GC.
If you can possibly avoid non-permanent allocations from getting to the CMS, then you can completely prevent fragmentation. This would be a huge win. If you can radically decrease number of objects being tenured as a fraction of the total tenured heap, you won't prevent CMS from having anything to do, but there is likely to be a threshold effect where fragmentation becomes a non-issue because freed blocks get glued back together quickly enough to keep the fragmentation at a workable level. You can tell how well you are doing at this by printing the tenuring distribution. It is quite informative. On Thu, Jan 27, 2011 at 4:44 PM, Stack <[email protected]> wrote: > > export HBASE_HEAPSIZE=8192 > > export HBASE_OPTS="-XX:+UseCMSInitiatingOccupancyOnly > > -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled > > -XX:SurvivorRatio=8 -XX:NewRatio=3 -XX:MaxTenuringThreshold=1 > > -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC > > -XX:+CMSIncrementalMode" > > export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails > > -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log" > > > > You don't need this on your class of hardware: CMSIncrementalMode > > But the one that looks off is XX:MaxTenuringThreshold. It struck me > as odd (and Ted Dunning as frightening) as it means all gets promoted > to old gen after one young gen GC. Let the young gen churn some more > before items earn a promotion. Make it 5 or 7 or something. See if > you go longer between full GCs? Initiating fraction of 60 is low. > Its probably costing you a bunch of CPU. Try the > XX:MaxTenuringThreshold first. >
