For what it's worth, when possible, I think it's easier to use the heap samples to understand leaks (it's often what our users end up doing; ie: hmm why is this heap allocation still remaining live?).
You could also dump the heap samples at OOM and/or VM exit. I know it is not always comparable but it might be able to answer the questions your users are asking. Thanks, Jc On Wed, Aug 22, 2018 at 2:14 PM Bernd Eckenfels <e...@zusammenkunft.net> wrote: > Hello, > > > > I could imagine a fork-and-coredump Approach would be possible. This > Limits the application pause to cloning the pagetables (not sure if there > is a mode to avoid this). > > > > This does however put stress on the virtual Memory (and requires > additional ram for the cloned structures). After the clone the core Dump > wont be faster, but it wont affect the app so much anymore. > > > > Gruss > > Bernd > > -- > http://bernd.eckenfels.net > > > > *Von: *David Holmes <david.hol...@oracle.com> > *Gesendet: *Mittwoch, 22. August 2018 22:56 > *An: *Ying Su <yin...@fb.com> > *Cc: *serviceability-dev <serviceability-dev@openjdk.java.net>; > hotspot-...@openjdk.java.net > *Betreff: *Re: custom/fast heap dumper > > > > Adding serviceability-dev to get more input. > > > > Cheers, > > David > > > > On 23/08/2018 5:58 AM, Ying Su wrote: > > > Hi Krystal, > > > > > > Thank you very much for your valuable input! The online analysis would > definitely help in a lot of cases. However there are still some cases > getting a heap dump would help or ease the way of investigation: > > > > > > 1. When there is slow memory leak; > > > 2. When there is OOM on heap. Getting a full dump at OOM greatly > helped us in finding the problem; > > > 3. When full GCs happen. > > > The occurrences of 2) and 3) are hard to predict and online analysis > usually requires to be taken before these events happen for a period of > time and could create lots of false positive data collections. So taking a > heap dump still helps us in a lot of ways. > > > > > > That being said, we are still looking for expert opinions on the best > way to implement such fast heap dumper. Your suggestions are highly > appreciated! > > > > > > Thank you, > > > Ying > > > > > > From: Krystal Mok <rednaxel...@gmail.com> > > > Date: Friday, August 17, 2018 at 4:26 PM > > > To: Ying Su <yin...@fb.com> > > > Cc: "hotspot-...@openjdk.java.net" <hotspot-...@openjdk.java.net> > > > Subject: Re: custom/fast heap dumper > > > > > > Hi Ying, > > > > > > Side-stepping from your question a bit: is it absolutely necessary to > take a full heap dump for your use case? Or would it be more feasible to do > some of the analysis that you want online instead of taking a heap dump and > then do offline analysis? > > > > > > Some analysis that people used to do on heap dump snapshots could be > done with JFR or the new Low-overhead heap profiling feature. Would those > be sufficient, or perhaps would extending those a bit be sufficient for > your use case? > > > > > > At Azul Systems, the Zing JVM supports some of the analysis piggybacking > on the GC. Since the C4 GC is fully concurrent (*), these operations are > not interruptive at all and has very low overhead. > > > For OpenJDK, with ZGC on the horizon, this kind of feature piggybacking > on a fully concurrent GC would also be possible. > > > > > > My two cents, > > > Kris > > > > > > * C4 GC can concurrently mark and compact the heap, albeit it does still > do a few very short pauses during the whole concurrent GC cycle. Similar > story with ZGC. > > > > > > On Fri, Aug 17, 2018 at 3:51 PM, Ying Su <yin...@fb.com<mailto: > yin...@fb.com>> wrote: > > > Hi, > > > > > > We want to implement a custom fast heap dumper that should work on Java > 9 and 10, because we often need to dump huge heaps (~200GB), and it takes > 20-30 minutes with jmap (e.g. we thought about zeroing out the large arrays > and compressing the output, etc.) We’ve been looking at the following 2 > options: > > > > > > > > > 1. Modify the JVMTI demo hprof implementation in JDK8 > > > 2. Reuse/modify jdk9-dev<https://github.com/netroby/jdk9-dev > >/hotspot<https://github.com/netroby/jdk9-dev/tree/master/hotspot>/src< > https://github.com/netroby/jdk9-dev/tree/master/hotspot/src>/share< > https://github.com/netroby/jdk9-dev/tree/master/hotspot/src/share>/vm< > https://github.com/netroby/jdk9-dev/tree/master/hotspot/src/share/vm > >/services< > https://github.com/netroby/jdk9-dev/tree/master/hotspot/src/share/vm/services > >/heapDumper.cpp > > > > > > We’ve tried the first option but it’s very slow due to some shared hash > tables causing high lock contention. And the second option is more > complicated because we need to access internal JVM classes, and we don’t > know how we can deploy it to production. We’d appreciate if we can get some > expert opinion on how to best solve this problem. > > > > > > > > > Thank you very much, > > > Ying > > > > > > > > > > > > > > > > > > > > > > > > -- Thanks, Jc
