andygrove commented on code in PR #2868:
URL: 
https://github.com/apache/datafusion-ballista/pull/2868#discussion_r4579027618


##########
ballista/core/src/hll.rs:
##########
@@ -128,97 +104,69 @@ impl HyperLogLog {
         hashes.resize(array.len(), 0);
         create_hashes([array], &HLL_RANDOM_STATE, hashes)?;
         match array.logical_nulls() {
-            None => hashes.iter().for_each(|&hash| self.add_hashed(hash)),
+            None => hashes.iter().for_each(|&hash| self.0.update(hash)),

Review Comment:
   One thing I noticed is that `HllSketch::update` hashes its input again with 
Murmur3, so each key value now gets hashed twice on the shuffle write path 
(once by `create_hashes`, once inside the sketch). A quick microbenchmark of 
`HyperLogLog::update` over 8M distinct `Int64` values at precision 12 went from 
about 15 ms to about 40 ms on my machine.
   
   Since `ballista.shuffle.collect_distinct_counts` is on by default, could you 
share before and after TPC-H numbers against `main`? Something like:
   
   ```sh
   cargo run --release --bin tpch -- benchmark ballista \
     --host localhost --port 50050 \
     --iterations 3 --path <tpch parquet dir> --format parquet
   ```
   
   at SF10 or higher would be ideal. If the overhead shows up, one option is to 
skip `create_hashes` and feed the typed values to the sketch directly so each 
value is only hashed once. `Coupon` can't be built from an existing hash since 
`pack` is private, so that might be worth raising upstream too.



##########
Cargo.lock:
##########
@@ -5532,7 +5539,7 @@ source = 
"registry+https://github.com/rust-lang/crates.io-index";
 checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
 dependencies = [
  "heck 0.5.0",
- "itertools 0.13.0",
+ "itertools 0.14.0",

Review Comment:
   Could you rebase on `main`? `Cargo.lock` conflicts right now, and it also 
picks up some unrelated bumps here (`itertools` 0.14 for prost-build and 
`getrandom` 0.4 for tempfile). `datasketches` has no dependencies, so taking 
`main`'s lockfile and letting cargo add just that entry should keep the diff 
minimal.



##########
ballista/core/proto/ballista.proto:
##########
@@ -738,14 +738,8 @@ message PartitionStats {
   int64 num_rows = 1;
   int64 num_batches = 2;
   int64 num_bytes = 3;
-  repeated ColumnStats column_stats = 4;
-}
-
-message ColumnStats {
-  datafusion_common.ScalarValue min_value = 1;
-  datafusion_common.ScalarValue max_value = 2;
-  uint32 null_count = 3;
-  uint32 distinct_count = 4;
+  reserved 4;

Review Comment:
   Reserving field 4 is the right call on the wire side. Since `ColumnStats` 
and `PartitionStats::column_stats` were public in 55.0.0, downstream code that 
builds a `PartitionStats` with a struct literal will stop compiling. Could you 
add a short note to `docs/source/upgrading/56.0.0.md`? We should also add the 
`api change` label.



##########
ballista/core/src/hll.rs:
##########
@@ -15,47 +15,46 @@
 // specific language governing permissions and limitations
 // under the License.
 
-//! HyperLogLog distinct-count sketch for shuffle output columns.
+//! HyperLogLog distinct-count sketch for shuffle output columns, backed by the
+//! Apache DataSketches HLL sketch.
 //!
-//! Shuffle writers update one sketch per sketched column, executors ship the
-//! registers to the scheduler in `TaskColumnStats.sketches`, and the scheduler
-//! merges every task's sketch for a column into a stage-level distinct count.
+//! Shuffle writers update one sketch per sketched column, executors ship it to
+//! the scheduler in `TaskColumnStats.sketches`, and the scheduler merges every
+//! task's sketch for a column into a stage-level distinct count with a
+//! [`HyperLogLogUnion`].
 //!
 //! Values are hashed with DataFusion's `HLL_RANDOM_STATE`, the hash state

Review Comment:
   Nit: since the sketch hashes the values again, `HLL_RANDOM_STATE` doesn't 
really decide anything about the result anymore. Might be worth saying so here, 
so nobody assumes the sketch is compatible with DataFusion's `approx_distinct`.



##########
ballista/core/src/hll.rs:
##########
@@ -79,46 +74,27 @@ impl HyperLogLog {
     ///
     /// If `precision` is outside [`MIN_PRECISION`] to [`MAX_PRECISION`].
     pub fn new(precision: u8) -> Self {
-        assert!(
-            (MIN_PRECISION..=MAX_PRECISION).contains(&precision),
-            "HyperLogLog precision {precision} is outside \
-             {MIN_PRECISION}..={MAX_PRECISION}"
-        );
-        Self {
-            precision,
-            registers: vec![0; 1 << precision].into_boxed_slice(),
-        }
+        assert_precision(precision);
+        Self(HllSketch::new(precision, HllType::Hll8).expect("precision is in 
range"))
     }
 
     /// The number of hash bits that select a register.
     pub fn precision(&self) -> u8 {
-        self.precision
+        self.0.lg_config_k()
     }
 
-    /// Restores a sketch from [`Self::as_bytes`]. Returns `None` if `bytes`
-    /// is not a valid register array: its length must be `2^p` for a
-    /// supported precision `p`, and no register can exceed the largest rank
-    /// a hash can produce.
+    /// Restores a sketch from [`Self::to_bytes`]. Returns `None` if `bytes`
+    /// is not a sketch of a supported precision.
     pub fn from_bytes(bytes: &[u8]) -> Option<Self> {
-        if !bytes.len().is_power_of_two() {
-            return None;
-        }
-        let precision = u8::try_from(bytes.len().trailing_zeros()).ok()?;
-        if !(MIN_PRECISION..=MAX_PRECISION).contains(&precision) {
-            return None;
-        }
-        if bytes.iter().any(|&r| r > max_rank(precision)) {
-            return None;
-        }
-        Some(Self {
-            precision,
-            registers: bytes.into(),
-        })
+        let sketch = HllSketch::deserialize(bytes).ok()?;

Review Comment:
   Small question. The old `from_bytes` rejected any register above `max_rank`, 
and now we rely on `HllSketch::deserialize`. That checks lengths and counts, 
but `Array4::update` still has an `.expect(...)` for an `AUX_TOKEN` register 
with no aux entry. Executor to scheduler traffic is trusted, so this isn't a 
big worry. Do you know if upstream guarantees that a sketch that deserializes 
OK can't panic later in `HllUnion::update`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to