gortiz opened a new pull request, #20147: URL: https://github.com/apache/pinot/pull/20147
## Summary This PR adds the JVM garbage collection (GC) state to the logs of the heap-based query killing. The accountant compares the used heap, garbage included, against the critical and panic levels. With the new logs, an operator can tell if the heap was full of live data or full of garbage that the next collection frees. The kill decision does not change. ## Background `ResourceUsageUtils.getUsedHeapSize()` includes objects that are not reachable. With G1, the young generation can grow to 60% of the heap (`G1MaxNewSizePercent`). Just before a young collection, the used heap is approximately the old generation plus a full eden. A query that uses a lot of memory, at a moderate allocation rate, can push this peak over the critical level. The accountant then kills a query, although the next young collection frees most of that memory. The opposite case also occurs: the heap is full of live data, and the kill is necessary. Today the logs cannot show which case occurred. The JVM metrics exporters publish the GC data as gauges, but a scrape interval of 60 s is too coarse for events that last a few seconds. ## Change The new class `GcStateDescriber` has no state. It reads the MXBeans only when a log line is written: - The old generation usage after the last collection (`MemoryPoolMXBean#getCollectionUsage`). This value is an estimate of the live set. For G1, the JVM updates it after each collection, young collections included. - For each collector, the collection count and the time since the end of its last collection (`com.sun.management.GarbageCollectorMXBean#getLastGcInfo`). If the JVM does not expose this information, the time is not written. `QueryResourceAggregator` and `PerQueryCPUMemResourceUsageAccountant` add `GC state: ...` to these log lines: - The transitions to panic, to critical, and back to the safe zone. - `Killed N queries ... due to OOM panic`. - `Query: ... terminated` (`logTerminatedQuery`, with the same signature). Example with G1: ``` Heap used bytes: 1051176456 exceeds critical level: ..., killing most expensive query. GC state: oldGenUsedAfterLastGcBytes: 1049837736, collectors: [G1 Young Generation (count: 54, last ended 90 ms ago), G1 Concurrent GC (count: 6, last ended 95 ms ago), G1 Old Generation (count: 1, last ended 88 ms ago)] ``` Notes: - The class finds the old generation pool for G1, Parallel, Serial, generational ZGC, and Shenandoah (single heap pool). - The description never throws. If an MXBean fails, the line contains `unavailable: <error>`, and the kill continues. - This PR adds no metric and no configuration. ## Validation - `GcStateDescriberTest` (new, 4 tests) uses fake MXBeans: a G1 description, a collector without GC information, a collector that throws, and the old generation pool for 4 collectors. - `QueryResourceAggregatorTest`, `QueryMonitorConfigTest`, and `QueryMonitorConfigScanKillingTest` pass. - A manual run on a JVM with G1 gave the example line above. - `spotless`, `license`, and `checkstyle` pass on `pinot-core`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
