comphead opened a new pull request, #7629:
URL: https://github.com/apache/datafusion-comet/pull/7629

   ## Which issue does this PR close?
   
   Part of #7341. #7341 stays open for item 2 (a `GroupsAccumulator`) and for 
the `datasketches` bump once a release includes apache/datasketches-rust#317.
   
   ## Rationale for this change
   
   #7345 fixed the problem in the title of #7341. Item 3 is still real on 
`main`: merging a partial sketch that is already dense costs about 270 µs at 
lgConfigK 18, while the register-wise max it needs takes 5 µs. `datasketches` 
0.6.0 rebuilds its register cache, one `f64` division per register, after every 
dense merge.
   
   apache/datasketches-rust#317 (merged 2026-11-04, not released yet) defers 
that rebuild until the next query. Comet would not benefit from it. 
`SparkHllUnion::merge` calls `estimate()` on the union and on the incoming 
sketch before every update, to keep a `dense` flag for the heap-size inference, 
and `heap_size()` calls `estimate()` on every accumulator `size()`. Each of 
those calls would force the deferred rebuild.
   
   `datasketches` 0.6.0 exposes the size directly. 
`HllSketch::estimated_size()` and `HllUnion::estimated_size()` read the coupon 
array or register array length in O(1), and the result is exact. The 
estimate-based inference predates that API, since 0.3.0 kept the layout private.
   
   ## What changes are included in this PR?
   
   - `SparkHllSketch::heap_size()` and `SparkHllUnion::heap_size()` return the 
crate's `estimated_size()`.
   - The `layout` module, both `dense` flags and the `preamble` module are 
removed, so `merge` and `size()` no longer compute estimates.
   - `heap_size_follows_the_crates_layout` now asserts that `heap_size()` 
equals the allocation at every step, instead of "never below, at most one size 
step above".
   
   ## How are these changes tested?
   
   The existing heap-size tests cover it. `heap_size_follows_the_crates_layout` 
now requires an exact match with the allocation the crate serializes at every 
cardinality, through each SET resize and the promotion. 
`high_lg_config_k_is_charged_for_what_it_holds`, 
`union_heap_size_follows_the_crates_layout` and 
`singleton_groups_at_high_lg_config_k_fit_a_small_pool` are unchanged. Sketch 
results do not change: only the size accounting is touched, and the probe below 
gave identical output bytes in every build.
   
   The numbers come from a throwaway example (not in this PR) that runs a Final 
`AggregateExec` over partial states at lgConfigK 18, with rows interleaved one 
partial per group per batch, the way a reducer reads map outputs. Release build 
on an Apple M-series laptop, `hll_sketch_agg`, ns per partial row, median of 5 
runs over two interleaved rounds. `hll_union_agg` gives the same picture.
   
   With `datasketches` 0.6.0, as shipped by this PR:
   
   | Case | `main` | This PR |
   |---|---|---|
   | Tiny: 200K groups x 8 partials, 0 to 8 values | 211 | 169 |
   | SET: 2K groups x 100 partials, 50 values | 1,453 | 1,450 |
   | Small partials into dense groups: 20 x 400, 200 values | 3,431 | 3,445 |
   | Dense partials: 8 groups x 32, 30,000 values | 359,000 | 361,000 |
   
   With `datasketches` at apache/datasketches-rust@5f080f98e5 (#317, still 
version 0.6.0, through `[patch.crates-io]`, not part of this PR):
   
   | Case | `main` | This PR |
   |---|---|---|
   | Dense partials, per row | 351,000 ns | 24,200 ns |
   | One `SparkHllUnion::merge` of a dense sketch | 266 µs | 5.5 µs |
   
   A plain 2^18 register-wise max takes 5.2 µs, so with #317 a dense merge 
costs about what the merge itself has to do. Differences under 5% are within 
run-to-run noise here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to