> On Aug 28, 2026, at 6:05 PM, Gillespie, Oli <[email protected]> wrote: > > I haven't found a clean way to tag the filler arrays that would work with > existing tooling. We could perhaps represent it as an FillerElement[] (so, > object, not primitive) with null elements. That adds complications with oop > size not always being the same as int. If you have ideas please share. Even > if we do figure that out, I'm not convinced they belong, but I would agree > it's a positive change.
I think we both agree this feels like a misused of int[]. I think the best choice is the one suggested here. That is, write out a FilterElement[] to replace the int[]. The [] is of reference so that is a full word. The [] could be empty. I believe odd length int[] is padded to a full word so that feels like it should fit. Does this work or what am I missing? > > > Right now FillerElement[] and FillerObject show up in the heap dump. My misunderstanding, I thought you said that FilterElement[] and FilterObject didn’t show up in heap dumps. Kind regards, Kirk > > From: Kirk Pepperdine > Sent: Friday, August 28, 2026 16:50 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > Hi Oli, > > Maybe a compromise here would be to not filter out the filler but to replace > it in the heap dump as something recognizable. I’d be happier with that than > having to use heuristics to sort things out. Right now FillerElement[] and > FillerObject show up in the heap dump. > > Kind regards, > Kirk > > > On Aug 28, 2026, at 11:44 AM, Gillespie, Oli <[email protected]> wrote: > > Fair enough. I don't consider confusion and surprise disqualifying, but they > should be weighed against value. Re flag - if there were a flag, I of course > would propose that not including filler be the default - the defaults ought > to be suitable for the majority, sophisticated users that can make use of it > can set the flag. And by the time you're setting a flag to see filler > objects, why not just monitor them through logs and metrics? If we start from > the goal of monitoring TLAB waste, region tail waste etc. then I think the > best solution would be direct metrics. > > I do get that we'd be losing information (and gaining some, since now we know > that all the remaining int[]s are user objects), I just like the tradeoff. I > don't know the history, but it seems to me that hprof is not intended to > represent VM layout internals. It does not encode object header (even header > size) nor field layout. With Valhalla's flat fields/arrays, the gap between > hprof and memory representation could be huge. It also does not encode GC > specifics like region boundaries, generation info, object age, GC roots. In a > simple list of objects, I don't think VM filler conceptually or practically > fits, and as discussed has real downsides due to the current implementation. > I would for sure like a richer format where it could fit. > > Oli > From: Kirk Pepperdine > Sent: Friday, August 28, 2026 09:29 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > > Hi Oli, > I'd like to push back on framing confusion and surprise as a reason for this > information being removed. My experience with workshops that include reading > heap dumps is that many developers are confused by many aspects of a heap > dump. Moreover, we are often surprised at what is in a heap dump (or produced > by any profiler for that matter). I don’t believe these are valid reasons for > removing the data from the heap. Instead this should be motivation to improve > the tooling. A heap dump tooling should help make issues visible and help and > guide users to a solution. Filtering out a problem doesn’t help with > visibility. Prescribed filtering creates blind spots for those that know what > to do with the data. Maybe the filtering could be turned on with a feature > flag? > > Kind regards, > > Kirk > > On Aug 28, 2026, at 9:57 AM, Gillespie, Oli <[email protected]> wrote: > > It has both value and cost, so I'm trying to weigh that up. So far it seems > that the value it provides is smaller (only a small portion of heap dump > users can usefully distinguish filler int[] from real int[], and make use of > that information) than the cost (confusion, leak/disclosure of old memory, > heap dump time and size, parsing time for a wider swath of users), so I'm > still in favour. > > I'll wait a few more days for any more opinions. > > Oli > > From: Kirk Pepperdine > Sent: Thursday, August 27, 2026 14:55 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > Hi Oli, > > I’m aware of the GC log source and I’m happy to work from that if it’s > available. However, I’m also working from the heap dump in cases where a GC > log may not be available. Again, this is information in the heap dump that > has value and it would be disruptive for it to be stripped out. > > Kind regards, > Kirk > > > On Aug 27, 2026, at 2:31 PM, Gillespie, Oli <[email protected]> wrote: > > For humongous region waste, you can calculate the waste without observing the > fillers by using the non-filler object size and region size, as you said (an > explicit counter for this would be nice, and should be very easy to add). > > For TLAB waste, it's logged directly: "[0.365s][debug][gc,tlab] GC(0) TLAB > totals: thrds: 5 alloc-frac: 54.6% refills: 104 max: 100 slow allocs: 60 max > 60 waste: 0.9% gc: 65832B(0.9%) max: 16416B refill: 3096B(0.0%) max: 3096B", > and these counters are also available through hsPerfData. > > So it seems like the main causes will still be observable. > > Oli > > From: Kirk Pepperdine > Sent: Thursday, August 27, 2026 13:12 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > My first heuristic was finding humongous allocations and knowing the region > size one can calculate the waste. From there one can find int[] arrays that > are ~ the size of the tlab. It’s not a perfect heuristic but it’s been > helpful. > > Kind regards, > Kirk > > > On Aug 27, 2026, at 10:56 AM, Gillespie, Oli <[email protected]> wrote: > > Thanks. Could you share how you find and distinguish them? I suppose, 'large > unreachable int[]' is one heuristic. Perhaps the fact that the contents are > unzeroed heap memory makes them look quite distinctive too? What sources do > you look for and what are the remedies? I'm wondering if there's any more > direct measurement that's either already available or could be made > available. For example, for the filler at the end of humongous G1 regions, it > would be easy for G1 to track and report directly. > > Do you always operate on hprof files? The filler objects will still be > present in GC.class_histogram and JVMTI heap iteration, do those cover your > needs? > > "I would leave it up to the tooling" - my problem is that the hprof doesn't > retain the information to (perfectly) distinguish filler arrays from real int > arrays. If they were clearly marked, then I might agree. But even then, the > increased dump size and the fact that leaks random unused heap memory remain. > Those could possibly be addressed by a flag to opt-in to dumping fillers, and > to dump them zeroed, but I think that's too much. > > Oli > > > From: Kirk Pepperdine > Sent: Thursday, August 27, 2026 08:26 > To: Gillespie, Oli > Cc: [email protected] > Subject: RE: [EXTERNAL] [External] : Re: Omit GC filler objects from heap dump > > Hi Ali, > I share your frustration with filler objects, they are confusing. That said, > I have tooling that finds these filler objects and works to sort out if > anything can be done to reduce this type of heap “wastage”. I have some > heuristics that help me distinguish source and propose possible remedies. So > I would like to object to having this information hidden from me. I would > leave it up to the tooling to decide if these filler classes should filtered > out. > > Kind regards, > Kirk Pepperdine > > > On Aug 26, 2026, at 5:50 PM, Gillespie, Oli <[email protected]> wrote: > > Hearing no objections, I created https://github.com/openjdk/jdk/pull/32542 to > implement this. I found the > existinghttps://bugs.openjdk.org/browse/JDK-8372389 which had the same idea. > From: Gillespie, Oli <[email protected] <mailto:[email protected]>> > Sent: 24 August 2026 10:20 > To: [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>> > Subject: Re: Omit GC filler objects from heap dump > > Bumping this - does anyone have any objection to excluding filler objects > from heap dumps? > > Oli > From: Gillespie, Oli > Sent: 18 August 2026 15:29 > To: [email protected] <mailto:[email protected]> > <[email protected] <mailto:[email protected]>> > Subject: Omit GC filler objects from heap dump > > Proposal: exclude filler objects from heap dumps. They confuse more than they > help. > > I've had a few cases where filler arrays in heap dumps confused me for quite > a while. It's okay in things like GC.class_histogram as they have the > identifying name of "[Ljdk.internal.vm.FillerArray", but in other tools like > Eclipse MAT they still show up as plain int arrays. Normally unreachable > objects would not be looked at much, I get that, but here are the cases I > heard about recently. 1 - HeapDumpOnOutOfMemoryError produced a heap dump, > which only had around 20% space filled with reachable objects, the rest > unreachable. I figured the unreachable stuff must be relevant to the OOM. > Actually, the 'OutOfMemoryError: java heap space' was triggered by GCLocker > retry count exceeded, the JVM was not out of heap space at all. 2 - a service > had an issue with periodic allocation spikes causing G1 full GC. They enabled > HeapDumpBeforeFullGC to understand the garbage that was triggering the full > gc. Many large int arrays seemed suspicious, but in the end they were mostly > fillers. > > I propose the simple option of excluding filler objects from heap dumps, but > if there's a way to instead make these identifiable in MAT and other tools, > that would also be fine. I wonder if Valhalla might provide any options here > to represent the filler elements not as primitives but as values? > > Feedback appreciated! > > Oli > > (Filler object background - > https://tschatzl.github.io/2022/09/26/jdk-vm-internal-fillerarray.html) > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom. > > > > > Amazon Development Centre (London) Ltd.Registered in England and Wales with > registration number 04543232 with its registered office at 1 Principal Place, > Worship Street, London EC2A 2FA, United Kingdom.
smime.p7s
Description: S/MIME cryptographic signature
