AhmedDusuki opened a new pull request, #335: URL: https://github.com/apache/datasketches-rust/pull/335
Addresses the empty and sparse union serialization paths in #319. ## Summary - add `CpcUnion::serialize()` with output identical to `to_sketch().serialize()` - serialize empty and sparse union results without materializing an intermediate sketch - preserve the effective `lg_k`, seed hash, and merged-result metadata - retain the existing result-sketch conversion for bit-matrix unions - add serialization regression coverage and paired benchmarks ## Details For sparse unions, `to_sketch()` clones the accumulator solely to mark the result as merged before encoding it. `CpcUnion::serialize()` now reuses the sketch encoder with merged-result metadata while borrowing the accumulator. Empty unions use the same direct path. Bit-matrix unions continue through `to_sketch().serialize()`. Encoding their window and surprise pairs directly can remain a separate follow-up as described in #319. The new method produces byte-for-byte identical serialized images to the existing path across empty, sparse, hybrid, pinned, and sliding states. The tests also cover custom seeds, reduced precision, deserialization, and subsequent union updates. ## Performance Release benchmarks compared both implementations from the same PR build on an AMD Ryzen 5 2600X with Rust 1.98.1 on Windows. Each comparison used 100 repetitions; times are medians in milliseconds, with the full observed range shown in parentheses. | `lg_k=18` union state | Rounds | Via result sketch | Direct | Ratio | |---|---:|---:|---:|---:| | Empty | 4,000,000 | 286 ms (278–327) | 218 ms (212–263) | 0.76x | | Sparse, 20,000 values | 500 | 265 ms (248–308) | 266 ms (259–290) | 1.00x | | Bit matrix, 80,000 values | 200 | 315 ms (308–377) | 315 ms (307–344) | 1.00x | Direct serialization reduced the empty-union median by 24%. Sparse-union and bit-matrix timings remained effectively unchanged. The bit-matrix case intentionally retains the existing result-sketch conversion path. A temporary counting allocator measurement produced: | `lg_k=18` union state | Via result sketch | Direct | |---|---:|---:| | Empty | 1 allocation, 40 bytes | 1 allocation, 40 bytes | | Sparse, 20,000 values | 11 allocations, 293,376 bytes | 10 allocations, 162,304 bytes | | Bit matrix, 80,000 values | 14 allocations, 815,792 bytes | 14 allocations, 815,792 bytes | For the sparse case, the direct path removes the accumulator clone, saving one allocation and 131,072 allocated bytes per serialization. Output bytes were identical in every case. The benchmark cases are included in this PR. The measurements were run with: ```sh cargo +1.98.1 x bench -- 'cpc::serde::serialize_union/0' --exact --repeat 100 --raw cargo +1.98.1 x bench -- 'cpc::serde::serialize_union/20000' --exact --repeat 100 --raw cargo +1.98.1 x bench -- 'cpc::serde::serialize_union/80000' --exact --repeat 100 --raw ``` ## Validation - full workspace test suite passed, including 145 cross-language serialization tests and 147 doctests - `cargo x check` passed - benchmark smoke tests passed - nightly Clippy passed with warnings denied - nightly formatting check passed - documentation build passed - `typos` and `hawkeye check` passed - `git diff --check` passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
