I haven't found a clean way to tag the filler arrays that would work with existing tooling. We could perhaps represent it as an FillerElement[] (so, object, not primitive) with null elements. That adds complications with oop size not always being the same as int. If you have ideas please share. Even if we do figure that out, I'm not convinced they belong, but I would agree it's a positive change.
> Right now FillerElement[] and FillerObject show up in the heap dump. Not sure what you mean by this. Filler arrays are encoded as int[]. FillerObject does show as FillerObject (which I don't care much about either way, it's well labelled so not overly confusing, and tends to be small in total volume). Oli ________________________________ From: Kirk Pepperdine Sent: Friday, August 28, 2026 16:50 To: Gillespie, Oli Cc: [email protected] Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump Hi Oli, Maybe a compromise here would be to not filter out the filler but to replace it in the heap dump as something recognizable. I’d be happier with that than having to use heuristics to sort things out. Right now FillerElement[] and FillerObject show up in the heap dump. Kind regards, Kirk On Aug 28, 2026, at 11:44 AM, Gillespie, Oli <[email protected]> wrote: Fair enough. I don't consider confusion and surprise disqualifying, but they should be weighed against value. Re flag - if there were a flag, I of course would propose that not including filler be the default - the defaults ought to be suitable for the majority, sophisticated users that can make use of it can set the flag. And by the time you're setting a flag to see filler objects, why not just monitor them through logs and metrics? If we start from the goal of monitoring TLAB waste, region tail waste etc. then I think the best solution would be direct metrics. I do get that we'd be losing information (and gaining some, since now we know that all the remaining int[]s are user objects), I just like the tradeoff. I don't know the history, but it seems to me that hprof is not intended to represent VM layout internals. It does not encode object header (even header size) nor field layout. With Valhalla's flat fields/arrays, the gap between hprof and memory representation could be huge. It also does not encode GC specifics like region boundaries, generation info, object age, GC roots. In a simple list of objects, I don't think VM filler conceptually or practically fits, and as discussed has real downsides due to the current implementation. I would for sure like a richer format where it could fit. Oli ________________________________ From: Kirk Pepperdine Sent: Friday, August 28, 2026 09:29 To: Gillespie, Oli Cc: [email protected] Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump Hi Oli, I'd like to push back on framing confusion and surprise as a reason for this information being removed. My experience with workshops that include reading heap dumps is that many developers are confused by many aspects of a heap dump. Moreover, we are often surprised at what is in a heap dump (or produced by any profiler for that matter). I don’t believe these are valid reasons for removing the data from the heap. Instead this should be motivation to improve the tooling. A heap dump tooling should help make issues visible and help and guide users to a solution. Filtering out a problem doesn’t help with visibility. Prescribed filtering creates blind spots for those that know what to do with the data. Maybe the filtering could be turned on with a feature flag? Kind regards, Kirk On Aug 28, 2026, at 9:57 AM, Gillespie, Oli <[email protected]> wrote: It has both value and cost, so I'm trying to weigh that up. So far it seems that the value it provides is smaller (only a small portion of heap dump users can usefully distinguish filler int[] from real int[], and make use of that information) than the cost (confusion, leak/disclosure of old memory, heap dump time and size, parsing time for a wider swath of users), so I'm still in favour. I'll wait a few more days for any more opinions. Oli ________________________________ From: Kirk Pepperdine Sent: Thursday, August 27, 2026 14:55 To: Gillespie, Oli Cc: [email protected] Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump Hi Oli, I’m aware of the GC log source and I’m happy to work from that if it’s available. However, I’m also working from the heap dump in cases where a GC log may not be available. Again, this is information in the heap dump that has value and it would be disruptive for it to be stripped out. Kind regards, Kirk On Aug 27, 2026, at 2:31 PM, Gillespie, Oli <[email protected]> wrote: For humongous region waste, you can calculate the waste without observing the fillers by using the non-filler object size and region size, as you said (an explicit counter for this would be nice, and should be very easy to add). For TLAB waste, it's logged directly: "[0.365s][debug][gc,tlab] GC(0) TLAB totals: thrds: 5 alloc-frac: 54.6% refills: 104 max: 100 slow allocs: 60 max 60 waste: 0.9% gc: 65832B(0.9%) max: 16416B refill: 3096B(0.0%) max: 3096B", and these counters are also available through hsPerfData. So it seems like the main causes will still be observable. Oli ________________________________ From: Kirk Pepperdine Sent: Thursday, August 27, 2026 13:12 To: Gillespie, Oli Cc: [email protected] Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump My first heuristic was finding humongous allocations and knowing the region size one can calculate the waste. From there one can find int[] arrays that are ~ the size of the tlab. It’s not a perfect heuristic but it’s been helpful. Kind regards, Kirk On Aug 27, 2026, at 10:56 AM, Gillespie, Oli <[email protected]> wrote: Thanks. Could you share how you find and distinguish them? I suppose, 'large unreachable int[]' is one heuristic. Perhaps the fact that the contents are unzeroed heap memory makes them look quite distinctive too? What sources do you look for and what are the remedies? I'm wondering if there's any more direct measurement that's either already available or could be made available. For example, for the filler at the end of humongous G1 regions, it would be easy for G1 to track and report directly. Do you always operate on hprof files? The filler objects will still be present in GC.class_histogram and JVMTI heap iteration, do those cover your needs? "I would leave it up to the tooling" - my problem is that the hprof doesn't retain the information to (perfectly) distinguish filler arrays from real int arrays. If they were clearly marked, then I might agree. But even then, the increased dump size and the fact that leaks random unused heap memory remain. Those could possibly be addressed by a flag to opt-in to dumping fillers, and to dump them zeroed, but I think that's too much. Oli ________________________________ From: Kirk Pepperdine Sent: Thursday, August 27, 2026 08:26 To: Gillespie, Oli Cc: [email protected] Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump Hi Ali, I share your frustration with filler objects, they are confusing. That said, I have tooling that finds these filler objects and works to sort out if anything can be done to reduce this type of heap “wastage”. I have some heuristics that help me distinguish source and propose possible remedies. So I would like to object to having this information hidden from me. I would leave it up to the tooling to decide if these filler classes should filtered out. Kind regards, Kirk Pepperdine On Aug 26, 2026, at 5:50 PM, Gillespie, Oli <[email protected]> wrote: Hearing no objections, I created https://github.com/openjdk/jdk/pull/32542 to implement this. I found the existinghttps://bugs.openjdk.org/browse/JDK-8372389 which had the same idea. ________________________________ From: Gillespie, Oli <[email protected]<mailto:[email protected]>> Sent: 24 August 2026 10:20 To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: Omit GC filler objects from heap dump Bumping this - does anyone have any objection to excluding filler objects from heap dumps? Oli ________________________________ From: Gillespie, Oli Sent: 18 August 2026 15:29 To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Omit GC filler objects from heap dump Proposal: exclude filler objects from heap dumps. They confuse more than they help. I've had a few cases where filler arrays in heap dumps confused me for quite a while. It's okay in things like GC.class_histogram as they have the identifying name of "[Ljdk.internal.vm.FillerArray", but in other tools like Eclipse MAT they still show up as plain int arrays. Normally unreachable objects would not be looked at much, I get that, but here are the cases I heard about recently. 1 - HeapDumpOnOutOfMemoryError produced a heap dump, which only had around 20% space filled with reachable objects, the rest unreachable. I figured the unreachable stuff must be relevant to the OOM. Actually, the 'OutOfMemoryError: java heap space' was triggered by GCLocker retry count exceeded, the JVM was not out of heap space at all. 2 - a service had an issue with periodic allocation spikes causing G1 full GC. They enabled HeapDumpBeforeFullGC to understand the garbage that was triggering the full gc. Many large int arrays seemed suspicious, but in the end they were mostly fillers. I propose the simple option of excluding filler objects from heap dumps, but if there's a way to instead make these identifiable in MAT and other tools, that would also be fine. I wonder if Valhalla might provide any options here to represent the filler elements not as primitives but as values? Feedback appreciated! Oli (Filler object background - https://tschatzl.github.io/2022/09/26/jdk-vm-internal-fillerarray.html) Amazon Development Centre (London) Ltd.Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom. Amazon Development Centre (London) Ltd.Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom. Amazon Development Centre (London) Ltd.Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom. Amazon Development Centre (London) Ltd.Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom. Amazon Development Centre (London) Ltd.Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom. Amazon Development Centre (London) Ltd. Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom.
