tisonkun opened a new pull request, #317: URL: https://github.com/apache/datasketches-rust/pull/317
Repeated HLL unions can spend most of their time copying a dense accumulator and rebuilding its estimator, even when each incoming partial is small. Persistent unions also rebuild register-derived caches after every dense input, and serializing a union currently requires cloning an intermediate sketch. This addresses the upstream costs identified in apache/datafusion-comet#7341 and complements apache/datafusion-comet#7345. ## Changes The implementation is split into three commits: 1. **Copy dense HLL8 inputs without rebuilding estimates.** When the source precision fits within the union's maximum, copy its registers and cached estimator directly. This covers empty-union adoption and the reverse-merge path that adopts a dense input before replaying coupons. 2. **Add `HllUnion::serialize()`.** Emit the same HLL8 sketch bytes as `to_sketch(HllType::Hll8).serialize()` without the intermediate sketch allocation and copy. The image retains the effective precision, not the union's maximum configuration. 3. **Defer register cache rebuilds across dense merges.** HLL8 bulk merges invalidate a `OnceLock` containing the zero count and estimator. Queries rebuild it on demand. Scalar updates maintain an initialized cache or leave an invalidated cache deferred. Copies retain that state, and restoring HIP after downsampling rebuilds the fields needed by subsequent updates. Shared-reference queries and thread safety are preserved. Dense merges still read the input registers. The benefit of deferral is avoiding repeated reconstruction when several inputs arrive before a query or serialization. Callers that query after every input still pay that reconstruction cost. ## Performance Release benchmarks on an Apple M4 Max with Rust `1.101.0-nightly (8006091eb 2026-10-17)`, 100 samples, medians. The baseline is `bb6ba57` with the same union benchmarks added. Merge benchmarks include a final estimate and HLL8 result extraction. | Workload at lg_k 18 | Baseline | This PR | |--------------------|---------:|--------:| | Adopt one dense HLL8 input into an empty union | 430.4 µs | 2.85 µs | | Recreate a union for each of 400 partials of 200 values | 79.71 ms | 3.10 ms | | Persistent union of 400 partials of 200 values | 939.6 µs | 911.6 µs | | Merge 16 dense partials, then query | 4.91 ms | 283.5 µs | | Merge 16 dense partials, querying after every input | 4.89 ms | 4.53 ms | | Merge 16 dense partials from lg_k 19 into lg_k 18 | 6.66 ms | 2.74 ms | On this PR's head, serializing an 80,000-value HLL8 union at lg_k 18 takes 4.00 µs directly versus 7.66 µs through `to_sketch`, removing one register-array allocation and copy. These are library benchmarks, not end-to-end Comet measurements. Reproduce with: ```sh cargo bench --package benchmarks --bench benchmarks -- hll:: --sample-count 100 ``` ## Validation - `cargo x prepare-testdata` - `cargo x check` - `cargo x test` - `cargo x lint` - `cargo +1.86.0 test -p tests-integration --test hll_test` — all 48 tests pass. Regression coverage includes copied HIP/composite state followed by updates; direct serialization across LIST, SET, and HLL modes and input representations; downsampling; query timing interleaved with scalar updates; copying an unqueried merged result; concurrent shared queries; and the KxQ split at register values 31/32 and the maximum six-bit value. Existing cross-language serialization tests also pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
