wirybeaver opened a new issue, #19666: URL: https://github.com/apache/pinot/issues/19666
## Motivation MSE hash aggregation spill currently uses `mseAggregationSpillMaxGroups` as a proxy for retained memory. Memory per group varies greatly across aggregation functions (e.g. SUM versus DISTINCTCOUNT/sketch functions) and the key representation. A single group-count value cannot provide a reliable memory bound or avoid unnecessary I/O across these queries. ## Proposed direction - Introduce an estimated retained-bytes signal for group keys and aggregation intermediate states. Define the estimate contract for `AggregationFunction` (including variable-size functions and plugin implementations) without breaking existing plugins. - Spill when the estimated memory budget or `mseAggregationSpillMaxGroups` is exceeded; name the new query option by its units (for example, `mseAggregationSpillMaxBytes`). - Evaluate the trigger during block processing when variable-size states can grow significantly within one input block. - Preserve `numGroupsLimit` as a per-hash-table hard ceiling and keep restore-side group limits distinct from the input spill trigger. - Test memory estimation, mixed aggregate types, skewed keys, and the interaction of both triggers. This is a follow-up to #19469, not part of its initial group-count-based spill feature. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
