This sounds similar to something I've seen before, but in that case I found the winning GC arguments to be something like
-XX:+UseParallelGC -XX:+UseParallelOldGC -XX:MaxDirectMemorySize=128m (note the old gen parallel compacting collector rather than the ParNew collector which IIRC is used with concurrent GC by default) I don't recall the MaxDirectMemorySize on its own preventing massive off-heap memory allocations from piling up. Just my 2 cents, YMMV. Sandy -----Original Message----- From: Laxman [mailto:[email protected]] Sent: Wednesday, July 11, 2012 9:22 PM To: 'Pablo Musa'; [email protected] Subject: RE: Hmaster and HRegionServer disappearance reason to ask > > 1) Fix the direct memory usage to a fixed value - > XX:MaxDirectMemorySize=1G > > This flag should be in RS ou DN? We need to apply for both but limit can be increased based on your load (May be 2G). Also we can to apply to all processes which are having following symptoms. 1) Allocated heap is few GB (4 to 8) 2) VIRT/RES will occupy double the heap (like 15GB) or even more 3) Long pauses in GC log (allocated heap is just <=8GB) 4) Your application uses lot of NIO/RMI calls(Ex: DataNode, RegionServer) In our cluster we apply for all server processes (NN, DN, HM, RS, JT, TT, ZooKeeper). Long pauses are disappeared after we set this flag (esp. for DN and RS). -- Regards, Laxman
