coderfender commented on code in PR #2868:
URL: 
https://github.com/apache/datafusion-ballista/pull/2868#discussion_r4638294566


##########
ballista/core/src/hll.rs:
##########
@@ -128,97 +104,69 @@ impl HyperLogLog {
         hashes.resize(array.len(), 0);
         create_hashes([array], &HLL_RANDOM_STATE, hashes)?;
         match array.logical_nulls() {
-            None => hashes.iter().for_each(|&hash| self.add_hashed(hash)),
+            None => hashes.iter().for_each(|&hash| self.0.update(hash)),

Review Comment:
   Numbers are in the comment above: no measurable difference end to end at 
TPC-H SF10 (7.89 s on `main` vs 7.65 s with this PR; 7.96 s with sketches off). 
I'll raise a `Coupon` constructor from an existing hash upstream so we can drop 
the second hash.
   



##########
Cargo.lock:
##########
@@ -5532,7 +5539,7 @@ source = 
"registry+https://github.com/rust-lang/crates.io-index";
 checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
 dependencies = [
  "heck 0.5.0",
- "itertools 0.13.0",
+ "itertools 0.14.0",

Review Comment:
   Rebased. The lockfile now only adds `datasketches`.
   



##########
ballista/core/proto/ballista.proto:
##########
@@ -738,14 +738,8 @@ message PartitionStats {
   int64 num_rows = 1;
   int64 num_batches = 2;
   int64 num_bytes = 3;
-  repeated ColumnStats column_stats = 4;
-}
-
-message ColumnStats {
-  datafusion_common.ScalarValue min_value = 1;
-  datafusion_common.ScalarValue max_value = 2;
-  uint32 null_count = 3;
-  uint32 distinct_count = 4;
+  reserved 4;

Review Comment:
   Added a note to `docs/source/upgrading/56.0.0.md`, and added the 
`api-change` label.
   



##########
ballista/core/src/hll.rs:
##########
@@ -79,46 +74,27 @@ impl HyperLogLog {
     ///
     /// If `precision` is outside [`MIN_PRECISION`] to [`MAX_PRECISION`].
     pub fn new(precision: u8) -> Self {
-        assert!(
-            (MIN_PRECISION..=MAX_PRECISION).contains(&precision),
-            "HyperLogLog precision {precision} is outside \
-             {MIN_PRECISION}..={MAX_PRECISION}"
-        );
-        Self {
-            precision,
-            registers: vec![0; 1 << precision].into_boxed_slice(),
-        }
+        assert_precision(precision);
+        Self(HllSketch::new(precision, HllType::Hll8).expect("precision is in 
range"))
     }
 
     /// The number of hash bits that select a register.
     pub fn precision(&self) -> u8 {
-        self.precision
+        self.0.lg_config_k()
     }
 
-    /// Restores a sketch from [`Self::as_bytes`]. Returns `None` if `bytes`
-    /// is not a valid register array: its length must be `2^p` for a
-    /// supported precision `p`, and no register can exceed the largest rank
-    /// a hash can produce.
+    /// Restores a sketch from [`Self::to_bytes`]. Returns `None` if `bytes`
+    /// is not a sketch of a supported precision.
     pub fn from_bytes(bytes: &[u8]) -> Option<Self> {
-        if !bytes.len().is_power_of_two() {
-            return None;
-        }
-        let precision = u8::try_from(bytes.len().trailing_zeros()).ok()?;
-        if !(MIN_PRECISION..=MAX_PRECISION).contains(&precision) {
-            return None;
-        }
-        if bytes.iter().any(|&r| r > max_rank(precision)) {
-            return None;
-        }
-        Some(Self {
-            precision,
-            registers: bytes.into(),
-        })
+        let sketch = HllSketch::deserialize(bytes).ok()?;

Review Comment:
   Checked v0.6.0. `HllUnion::update` reads an HLL_4 source only through 
`Array4::get`, which returns `cur_min` when an aux entry is missing instead of 
panicking. The `.expect`s are in `Array4::update`, the write path, and we never 
write into a deserialized sketch; the union's own sketch is always HLL_8. So no 
panic in our path, though a corrupt sketch would under-count. Upstream doesn't 
document a guarantee.
   



##########
ballista/core/src/hll.rs:
##########
@@ -15,47 +15,46 @@
 // specific language governing permissions and limitations
 // under the License.
 
-//! HyperLogLog distinct-count sketch for shuffle output columns.
+//! HyperLogLog distinct-count sketch for shuffle output columns, backed by the
+//! Apache DataSketches HLL sketch.
 //!
-//! Shuffle writers update one sketch per sketched column, executors ship the
-//! registers to the scheduler in `TaskColumnStats.sketches`, and the scheduler
-//! merges every task's sketch for a column into a stage-level distinct count.
+//! Shuffle writers update one sketch per sketched column, executors ship it to
+//! the scheduler in `TaskColumnStats.sketches`, and the scheduler merges every
+//! task's sketch for a column into a stage-level distinct count with a
+//! [`HyperLogLogUnion`].
 //!
 //! Values are hashed with DataFusion's `HLL_RANDOM_STATE`, the hash state

Review Comment:
   Done.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to