Java 6 update 34 can rotate GC Logs. -XX:+UseGCLogFileRotation http://stackoverflow.com/questions/3822097/rolling-garbage-collector-logs-in-java
As for profiling memory dumps, jprofiler7, yourrkit, etc.. YMMV. On Thu, Nov 1, 2012 at 10:01 AM, Jeff Whiting <[email protected]> wrote: > We don't have GC logging enabled (we did but gc.log would begin filling up > the hdd and there was no way to clear it out without restarting the region > server). Anyway to en gc.log and keep it to a reasonable size? > > I have two separate jmap dumps of the a region server before it dies. I > haven't really looked into those yet. I'll try to do that today. I've > typically used eclipse memory analyzer tool or netbeans. Is there a > profiler you'd recommend? > > ~Jeff > > On 10/30/2012 11:39 PM, Stack wrote: > >> On Mon, Oct 29, 2012 at 3:55 PM, Jeff Whiting <[email protected]> >> wrote: >> >>> However what we are seeing is that our memory usage goes up slowly until >>> the >>> region server starts sputtering due to gc collection issues and it will >>> eventually get timed out by zookeeper and be killed. >>> >>> Hey Jeff. You have GC logging enabled? Might not tell you more than >> you already know, that something is retaining more and more objects >> over time. You have a dumped heap? What have you used to poke at >> it? You generally want to find the objects that have the deepest size >> (Not all profilers let you do this though). This is usually enough to >> give you a clue. >> >> Anything particular about the character of your load? Ram asks if any >> big cells in the mix? >> >> St.Ack >> >> >> >> >> At this point I feel somewhat lost as to how to debug the problem. I'm >>> not >>> sure what to do next to figure out what is going on. Any suggestions as >>> to >>> what to look for or debug where the memory is being used? I can generate >>> heap dumps via jmap (although it effectively kills the region server) >>> but I >>> don't really know what to look for to see where the memory is going. I >>> also >>> have jmx setup on each region server and can connect to it that way. >>> >>> Thanks, >>> ~Jeff >>> >>> -- >>> Jeff Whiting >>> Qualtrics Senior Software Engineer >>> [email protected] >>> >>> > -- > Jeff Whiting > Qualtrics Senior Software Engineer > [email protected] > >
