comphead opened a new pull request, #7629: URL: https://github.com/apache/datafusion-comet/pull/7629
## Which issue does this PR close? Part of #7341. #7341 stays open for item 2 (a `GroupsAccumulator`) and for the `datasketches` bump once a release includes apache/datasketches-rust#317. ## Rationale for this change #7345 fixed the problem in the title of #7341. Item 3 is still real on `main`: merging a partial sketch that is already dense costs about 270 µs at lgConfigK 18, while the register-wise max it needs takes 5 µs. `datasketches` 0.6.0 rebuilds its register cache, one `f64` division per register, after every dense merge. apache/datasketches-rust#317 (merged 2026-11-04, not released yet) defers that rebuild until the next query. Comet would not benefit from it. `SparkHllUnion::merge` calls `estimate()` on the union and on the incoming sketch before every update, to keep a `dense` flag for the heap-size inference, and `heap_size()` calls `estimate()` on every accumulator `size()`. Each of those calls would force the deferred rebuild. `datasketches` 0.6.0 exposes the size directly. `HllSketch::estimated_size()` and `HllUnion::estimated_size()` read the coupon array or register array length in O(1), and the result is exact. The estimate-based inference predates that API, since 0.3.0 kept the layout private. ## What changes are included in this PR? - `SparkHllSketch::heap_size()` and `SparkHllUnion::heap_size()` return the crate's `estimated_size()`. - The `layout` module, both `dense` flags and the `preamble` module are removed, so `merge` and `size()` no longer compute estimates. - `heap_size_follows_the_crates_layout` now asserts that `heap_size()` equals the allocation at every step, instead of "never below, at most one size step above". ## How are these changes tested? The existing heap-size tests cover it. `heap_size_follows_the_crates_layout` now requires an exact match with the allocation the crate serializes at every cardinality, through each SET resize and the promotion. `high_lg_config_k_is_charged_for_what_it_holds`, `union_heap_size_follows_the_crates_layout` and `singleton_groups_at_high_lg_config_k_fit_a_small_pool` are unchanged. Sketch results do not change: only the size accounting is touched, and the probe below gave identical output bytes in every build. The numbers come from a throwaway example (not in this PR) that runs a Final `AggregateExec` over partial states at lgConfigK 18, with rows interleaved one partial per group per batch, the way a reducer reads map outputs. Release build on an Apple M-series laptop, `hll_sketch_agg`, ns per partial row, median of 5 runs over two interleaved rounds. `hll_union_agg` gives the same picture. With `datasketches` 0.6.0, as shipped by this PR: | Case | `main` | This PR | |---|---|---| | Tiny: 200K groups x 8 partials, 0 to 8 values | 211 | 169 | | SET: 2K groups x 100 partials, 50 values | 1,453 | 1,450 | | Small partials into dense groups: 20 x 400, 200 values | 3,431 | 3,445 | | Dense partials: 8 groups x 32, 30,000 values | 359,000 | 361,000 | With `datasketches` at apache/datasketches-rust@5f080f98e5 (#317, still version 0.6.0, through `[patch.crates-io]`, not part of this PR): | Case | `main` | This PR | |---|---|---| | Dense partials, per row | 351,000 ns | 24,200 ns | | One `SparkHllUnion::merge` of a dense sketch | 266 µs | 5.5 µs | A plain 2^18 register-wise max takes 5.2 µs, so with #317 a dense merge costs about what the merge itself has to do. Differences under 5% are within run-to-run noise here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
