tisonkun opened a new issue, #319: URL: https://github.com/apache/datasketches-rust/issues/319
## Problem Serializing a CPC union currently requires `union.to_sketch().serialize()`. In sparse mode, `to_sketch()` clones the accumulator before marking the result as merged. In bit-matrix mode, it constructs a sliding window and a pair table; serialization then extracts pairs, sorts them, and compresses the payload. This creates opportunities to avoid intermediate copies and data structures when only bytes are needed. ## Suggested approach - Add direct serialization of the union result, sharing the CPC encoder with `CpcSketch::serialize`. - A manageable starting point is the empty/sparse path: encode the accumulator with the correct union-result metadata without cloning it. - Evaluate generating the encoding inputs from the bit matrix without constructing a complete mutable sketch and rebuilding a pair table solely to extract its entries again. Preserve the window, offset, and surprise-pair semantics. - Preserve the result's merged-estimator semantics, effective `lg_k`, and seed hash. The bytes represent a sketch result, not a checkpoint of the union operator. Start with `datasketches/src/cpc/union.rs` (`to_sketch`) and `datasketches/src/cpc/sketch.rs` (`serialize`). PR #317 provides context for the equivalent HLL use case. Performance improvements here have not yet been measured; benchmark the actual paths before making claims. ## Validation Compare against `to_sketch().serialize()` across empty, sparse, hybrid, pinned, and sliding results, including mixed input precisions and custom seeds. Verify decoded estimates and subsequent merges, and run the existing serialization compatibility tests. Add benchmarks covering both sparse and bit-matrix unions, including allocation counts. Sparse-path improvements and the bit-matrix optimization can be submitted separately under this issue. Follow CONTRIBUTING.md for changelog entries, integration-test placement, serialization fixtures, and the required checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
