peterxcli commented on PR #4932: URL: https://github.com/apache/datafusion-comet/pull/4932#issuecomment-5182252986
Hi @andygrove, thanks for pointing this out. After investigating, I plan to change the implementation to use a native `GroupsAccumulator` with a single `QuantileSummaries` scratch buffer shared across all groups, similar to `HLLPlusPlusGroupsAccumulator`. For retained buffer memory after compression/merge, let: - `G` = number of groups - `C` = compressed summary capacity per group - `R` = raw merge/flush workspace - `H` = empty head-buffer capacity retained after flushing The approximate memory shape is: - `main`: `G × C`, but repeatedly allocates and frees temporary buffers - Previous PR version: `G × (C + R + H)` - Updated version: `G × C + max(R)` Each group now retains only its compressed summary. The raw workspace is reused across all groups, and the head buffer is released after flushing. For the high-cardinality benchmark—1,048,576 rows, 100,000 groups, and roughly 10 values per group—the buffer payload is approximately 56 MiB for the previous PR design versus 24 MiB for the shared-scratch design, around 2.3× lower. This is a capacity model rather than measured process RSS; the raw scratch component itself goes from roughly 100,000 buffers to one. In my current local change, the seeded high-cardinality benchmark also improved from 558 ms on `main` and 523 ms on the previous PR version to 236 ms with shared scratch. The high-cardinality result was relatively stable, with a 19 ms standard deviation. I also added coverage for grouped update/merge, nulls, filters, partial emission, array results, and scratch-buffer ownership. _Assisted by an LLM._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
