to keep this updated The Issue was cache - SoftReference. http://bugs.sun.com/bugdatabase/view_bug.do;jsessionid=cfd518f51afc7780e5188276b5f9?bug_id=6912889
The monitoring ran for month now and the application is stable. we changed the SoftRefLRUPolicyMSPerMB. Sharad ________________________________ From: Adam Heath <[email protected]> To: [email protected] Sent: Fri, 7 May, 2010 1:28:08 AM Subject: Re: Is there a possible memory issue causing objects to stay in old generation for long - ofbiz and javolution sharad bhushan wrote: > Hello Everyone, > > I have been trying to get the understanding of memory allocation when > debugging the load and memory issue in our ofbiz instance > > Here is the snap shot of our gc log after three days of running and this had > been continuing for next two days and went low > eventually not responding. This Full GC run as below was continuous with > tenured memory not clearing and Full GC kept running all > together for 2 days with breaks for few hours, it still survived "out of > memory" as young generation was releasing on Full GC. > > 328821.354: [Full GC 328821.354: [Tenured: 1048576K->1048575K(1048576K), > 2.9016030 secs] 1520444K->1087815K(1520448K), [Perm : > 49067K->49050K(49152K)], 2.9017140 secs] [Times: user=2.90 sys=0.00, > real=2.90 secs] > 328825.334: [Full GC 328825.334: [Tenured: 1048575K->1048575K(1048576K), > 2.4392020 secs] 1520447K->1097197K(1520448K), [Perm : > 49054K->49054K(49152K)], 2.4393150 secs] [Times: user=2.43 sys=0.00, > real=2.44 secs] > 328828.673: [Full GC 328828.673: [Tenured: 1048575K->1048575K(1048576K), > 2.4696630 secs] 1520447K->1107798K(1520448K), [Perm : > 49063K->49063K(49152K)], 2.4697760 secs] [Times: user=2.46 sys=0.00, > real=2.46 secs] > > 403173.352: [Full GC 403173.352: [Tenured: 1048576K->1048576K(1048576K), > 2.7718690 secs] 1520447K->1252656K(1520448K), [Perm : > 49417K->49417K(49664K)], 2.7719880 secs] [Times: user=2.77 sys=0.00, > real=2.77 secs] > 403176.958: [Full GC 403176.958: [Tenured: 1048576K->1048576K(1048576K), > 2.7772930 secs] 1520448K->1264672K(1520448K), [Perm : > 49421K->49421K(49664K)], 2.7774020 secs] [Times: user=2.78 sys=0.00, > real=2.77 secs] > 403180.713: [Full GC 403180.713: [Tenured: 1048576K->1048576K(1048576K), > 2.7374900 secs] 1520447K->1267832K(1520448K), [Perm : > 49426K->49426K(49664K)], 2.7376110 secs] [Times: user=2.73 sys=0.00, > real=2.74 secs] > > With above said i would like to give the top classes on doing jmap -histo > javolution FastMap$Entry - ~240MB - nearly 4,900,000 objects > javolution FastMap$Entry[] - ~120MB > javolution HashMap$Entry - ~90MB You have to be very very careful. All FastMap instances, and all their keys/values, and everything else they point to, including various chained classloaders, then the classes loaded by those classloaders, and all the static references in *those* classes, etc. I've found that the various memory graph walking tools don't really handle finding the *correct* top-level items to show. > > The java max heap (-Xmx) is 1.5GB. > > the tenured - old gen is ~1000MB and the above three elements are 50% > occupied, certainly after drilling through few heap dumps it describes of > above three classes. > > The tenured space does not get cleared, I am looking to suspect the FastMap, > specifically, we have been using ofbiz cache too. > > Can there be any inputs or observation about the possible issue in memory or > javolution, any directions for the above description would certainly help me. > > Regards > Sharad > > >
