tisonkun opened a new pull request, #227: URL: https://github.com/apache/datasketches-rust/pull/227
## Summary This audits the recent deserialization-hardening work by separating safety invariants from attempts to reconstruct a serializer's exact history. Each sketch is kept in a separate commit: - **REQ:** removes the mixed C++/Java floating-point state search and the historical minimum-stream-length check. Deserialization now checks the compact format's direct safety invariants, treats the level-zero sorted flag as a hint, and remains safe when opaque compactor state wraps during a later merge. - **Bloom:** treats the serialized bit count as cached metadata and derives it from the bit array, while checking the payload length before allocation. - **CPC:** makes compressed reads fallible, validates decoded indices at the point of use, fixes high-`lg_k` arithmetic, and replaces the broad random-mutation loop with truncation and targeted corruption tests. This incorporates and supersedes the implementation intent of #226; Jaideep Pyne is retained as co-author on the CPC commit. - **Theta:** checks uncompressed and compressed entry payloads, bit widths, and delta overflow before allocation or unpacking. - **Tuple:** uses the mandatory hash bytes as a payload lower bound before trusting the entry count for allocation. - **HLL:** validates mode-local capacities and payload sizes, and fixes compact HLL4 deserialization so it reads the serialized register array instead of skipping it and leaving zeroed registers. ## Complexity and performance The validation model is deliberately small: - header fields select a known mode; - counts are bounded by that mode's capacity and the remaining payload; - decoded indices are checked where they are consumed; - derived or cached state is recomputed or normalized instead of treated as independent truth. REQ production code is 68 lines smaller and no longer allocates candidate vectors or searches serializer-specific floating-point schedules during deserialization. Bloom performs the same required bit-array pass and derives the cache during that pass. Theta, Tuple, and HLL add constant-time checks before allocation. CPC adds bounds checks to compressed bit reads but does not add a second decode or validation pass. HLL4 now copies register bytes that the old code incorrectly skipped; that work is required for correct state restoration. There is no matching deserialization benchmark suite in this repository, so this PR makes no unsupported latency claim. ## Audit scope The existing Frequencies checks from #224 are direct constant-time resource bounds and were left unchanged. CountMin and T-Digest did not receive mechanical validation churn. One CountMin design tradeoff remains: an empty image carries the table configuration but no table payload, so eliminating configuration-driven allocation would require lazy table storage or a tighter public configuration limit rather than another payload check. ## Validation - `cargo x check` - `cargo x test` - `cargo x lint` - Java, C++, and Go serialization snapshots exercised by `cargo x test` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
