Xuanwo commented on code in PR #1: URL: https://github.com/apache/datasketches-rust/pull/1#discussion_r2609566228
########## src/hll/mod.rs: ########## @@ -0,0 +1,115 @@ +//! HyperLogLog sketch implementation for cardinality estimation. +//! +//! This module provides a probabilistic data structure for estimating the cardinality +//! (number of distinct elements) of large datasets with high accuracy and low memory usage. +//! +//! # Overview +//! +//! HyperLogLog (HLL) sketches use hash functions to estimate cardinality in logarithmic space. +//! This implementation follows the Apache DataSketches specification and supports multiple +//! storage modes that automatically adapt based on cardinality: +//! +//! - **List mode**: Stores individual coupons for small cardinalities +//! - **Set mode**: Uses a hash set for medium cardinalities +//! - **HLL mode**: Uses compact arrays for large cardinalities +//! +//! # HLL Types +//! +//! Three target HLL types are supported, trading precision for memory: +//! +//! - [`HllType::Hll4`]: 4 bits per bucket (most compact) +//! - [`HllType::Hll6`]: 6 bits per bucket (balanced) +//! - [`HllType::Hll8`]: 8 bits per bucket (highest precision) +//! +//! # Coupons +//! +//! A coupon is a 32-bit value encoding both a slot number (26 bits) and a value (6 bits). +//! The slot identifies which bucket to update, and the value represents the number of +//! leading zeros in the hash plus one. + +use std::io; +use std::io::Read; + +use murmur3::murmur3_x64_128; + +pub mod array4; +pub mod array6; +pub mod array8; +pub mod aux_map; +pub mod composite_interpolation; +pub mod container; +pub mod coupon_mapping; +pub mod cubic_interpolation; +pub mod estimator; +pub mod harmonic_numbers; +pub mod hash_set; +pub mod list; +pub mod serialization; +pub mod sketch; Review Comment: Do we need to publish all of them? Some of them feel more like internal implementation details. ########## Cargo.toml: ########## Review Comment: I'm thinking if we can split the basic Rust setup task into a separate PR so that we can work in parallel. ########## tests/hll_serialization_test.rs: ########## @@ -0,0 +1,230 @@ +//! HLL Sketch Serialization Compatibility Tests +//! +//! These tests verify binary compatibility with Apache DataSketches implementations: +//! - Java (datasketches-java) +//! - C++ (datasketches-cpp) +//! - Go (datasketches-go) +//! +//! Test data is generated by the reference implementations and stored in: +//! `tests/datasketches-go/serialization_test_data/` + +use std::fs; +use std::path::PathBuf; + +use datasketches::hll::sketch::HllSketch; Review Comment: Is it make sense for us to have the following import path instead? ``` use datasketches::HllSketch; ``` or ``` use datasketches::hll::HllSketch; ``` Maybe it should be our entrypoint for public API? ########## .gitmodules: ########## @@ -0,0 +1,3 @@ +[submodule "tests/datasketches-go"] Review Comment: I'm not quite happy about having a submodule for other projects. How about we make it part of CI or scripts so we can build them outside of datasketches-rust? Also, I feel we should test against Java or C++ first. Go is still under development, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
