Hi Oli, I’m aware of the GC log source and I’m happy to work from that if it’s available. However, I’m also working from the heap dump in cases where a GC log may not be available. Again, this is information in the heap dump that has value and it would be disruptive for it to be stripped out.
Kind regards, Kirk > On Aug 27, 2026, at 2:31 PM, Gillespie, Oli <[email protected]> wrote: > > For humongous region waste, you can calculate the waste without observing the > fillers by using the non-filler object size and region size, as you said (an > explicit counter for this would be nice, and should be very easy to add). > > For TLAB waste, it's logged directly: "[0.365s][debug][gc,tlab] GC(0) TLAB > totals: thrds: 5 alloc-frac: 54.6% refills: 104 max: 100 slow allocs: 60 max > 60 waste: 0.9% gc: 65832B(0.9%) max: 16416B refill: 3096B(0.0%) max: 3096B", > and these counters are also available through hsPerfData. > > So it seems like the main causes will still be observable. > > Oli > > From: Kirk Pepperdine > Sent: Thursday, August 27, 2026 13:12 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > My first heuristic was finding humongous allocations and knowing the region > size one can calculate the waste. From there one can find int[] arrays that > are ~ the size of the tlab. It’s not a perfect heuristic but it’s been > helpful. > > Kind regards, > Kirk > > > On Aug 27, 2026, at 10:56 AM, Gillespie, Oli <[email protected]> wrote: > > Thanks. Could you share how you find and distinguish them? I suppose, 'large > unreachable int[]' is one heuristic. Perhaps the fact that the contents are > unzeroed heap memory makes them look quite distinctive too? What sources do > you look for and what are the remedies? I'm wondering if there's any more > direct measurement that's either already available or could be made > available. For example, for the filler at the end of humongous G1 regions, it > would be easy for G1 to track and report directly. > > Do you always operate on hprof files? The filler objects will still be > present in GC.class_histogram and JVMTI heap iteration, do those cover your > needs? > > "I would leave it up to the tooling" - my problem is that the hprof doesn't > retain the information to (perfectly) distinguish filler arrays from real int > arrays. If they were clearly marked, then I might agree. But even then, the > increased dump size and the fact that leaks random unused heap memory remain. > Those could possibly be addressed by a flag to opt-in to dumping fillers, and > to dump them zeroed, but I think that's too much. > > Oli > > > From: Kirk Pepperdine > Sent: Thursday, August 27, 2026 08:26 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > Hi Ali, > I share your frustration with filler objects, they are confusing. That said, > I have tooling that finds these filler objects and works to sort out if > anything can be done to reduce this type of heap “wastage”. I have some > heuristics that help me distinguish source and propose possible remedies. So > I would like to object to having this information hidden from me. I would > leave it up to the tooling to decide if these filler classes should filtered > out. > > Kind regards, > Kirk Pepperdine > > > On Aug 26, 2026, at 5:50 PM, Gillespie, Oli <[email protected]> wrote: > > Hearing no objections, I created https://github.com/openjdk/jdk/pull/32542 to > implement this. I found the > existinghttps://bugs.openjdk.org/browse/JDK-8372389 which had the same idea. > From: Gillespie, Oli <[email protected] <mailto:[email protected]>> > Sent: 24 August 2026 10:20 > To: [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>> > Subject: Re: Omit GC filler objects from heap dump > > Bumping this - does anyone have any objection to excluding filler objects > from heap dumps? > > Oli > From: Gillespie, Oli > Sent: 18 August 2026 15:29 > To: [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>> > Subject: Omit GC filler objects from heap dump > > Proposal: exclude filler objects from heap dumps. They confuse more than they > help. > > I've had a few cases where filler arrays in heap dumps confused me for quite > a while. It's okay in things like GC.class_histogram as they have the > identifying name of "[Ljdk.internal.vm.FillerArray", but in other tools like > Eclipse MAT they still show up as plain int arrays. Normally unreachable > objects would not be looked at much, I get that, but here are the cases I > heard about recently. 1 - HeapDumpOnOutOfMemoryError produced a heap dump, > which only had around 20% space filled with reachable objects, the rest > unreachable. I figured the unreachable stuff must be relevant to the OOM. > Actually, the 'OutOfMemoryError: java heap space' was triggered by GCLocker > retry count exceeded, the JVM was not out of heap space at all. 2 - a service > had an issue with periodic allocation spikes causing G1 full GC. They enabled > HeapDumpBeforeFullGC to understand the garbage that was triggering the full > gc. Many large int arrays seemed suspicious, but in the end they were mostly > fillers. > > I propose the simple option of excluding filler objects from heap dumps, but > if there's a way to instead make these identifiable in MAT and other tools, > that would also be fine. I wonder if Valhalla might provide any options here > to represent the filler elements not as primitives but as values? > > Feedback appreciated! > > Oli > > (Filler object background - > https://tschatzl.github.io/2022/09/26/jdk-vm-internal-fillerarray.html) > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom.
smime.p7s
Description: S/MIME cryptographic signature
