Can you try Flight Recorder with allocation profile on?
If Full GC can not clean those objects, the application probably is
holding those. Might be a memory leak.
Thanks
Jenny
On 11/09/2016 11:38 PM, Amit Mishra wrote:
Hello Charlie/Poonam/team,
Need your help/suggestions on how to troubleshoot memory leak without
taking any heap dump.
We are facing random Promotion failure followed by Continuous
concurrent mode failures/Full GC events that impacts our Standalone
application for long time until restart.
Application GC remain stable for more than a week with smooth saw
tooth pattern and suddenly something happened within 1 hour or so that
results in severe GC failure and ultimately application failure.
We have verified traffic pattern/application logs and other dependent
application logs but there is no indication on why suddenly at one
point of time heap usage kept on increasing which results in CMS
failures.(Traffic pattern is fairly stable and there are no scheduled
or cron jobs during time of issue)
We cannot take heap dump as this is standalone application having big
heap size.(32G)
We have collected histogram during issue time and of non- issue time
and found that instances of 2-3 classes have been suddenly increased
from 200-300 MB to 5G+ but not sure how we can dig into code to find
out what cause those classes instances to surge.
Please guide me how to troubleshoot this issue in terms of any light
weight tool that would exactly pin point methods or calls that can
lead to this memory leak as we can’t take heap dump which is very
heavy impacting tool.
One more question is why Full GC not able to clean generations even
after multiple attempts and a continuous loop of GC failures being
created which got resolved only after application restart, does it
indicates that no new objects was creating & it was only GC algorithm
which started failing and increased heap usage.
Many thanks in advance for your kind support and guidance.
This is GC graph and attached is GC file.
cid:image002.jpg@01D23948.747997C0
Histogram snapshots:
java.util.HashMap$Entry was only 400 MB before issue and then 5.5G
during issue same thing true for AcctSessionInfo and java.lang.String
class instances.
Non issue time:
num #instances #bytes class name
----------------------------------------------
1: 13613915 2219936904 [Ljava.lang.Object;
2: 10065566 1569906056 [Ljava.util.HashMap$Entry;
3: 2671564 1175488160
com.redknee.product.s5600.ipc.xgen.PdpContextID
4: 17247420 903565648 [C
5: 10055084 723966048 java.util.HashMap
6: 17208464 688338560 java.lang.String
7: 7843562 439239472 java.util.HashMap$Entry
8: 10065566 402622640 java.util.HashMap$FrontCache
Issue time :Heap usage around 28G
num #instances #bytes class name
----------------------------------------------
1: 118037170 6600874168 [C
2: 103071116 5771982496 java.util.HashMap$Entry
3: 101560457 5687385592
com.redknee.product.s5600.ipc.xgen.AcctSessionInfo
4: 118042761 4721710440 java.lang.String
5: 9942863 3020272632 [Ljava.lang.Object;
6: 7537560 2737186632 [Ljava.util.HashMap$Entry;
7: 1453865 639700600
com.redknee.product.s5600.ipc.xgen.PdpContextID
8: 7537148 542674656 java.util.HashMap
Thanks,
Amit Mishra
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
