comphead commented on code in PR #7691:
URL: https://github.com/apache/datafusion-comet/pull/7691#discussion_r4645582477


##########
spark/src/test/resources/sql-tests/expressions/hash/hash_fixed_width_nulls.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements.  See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership.  The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License.  You may obtain a copy of the License at
+--
+--   http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied.  See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- Plain encoding keeps the columns off the dictionary path, so they reach the 
64-row chunk loop
+-- in `hash_fixed_width_with_nulls`.
+-- Config: parquet.enable.dictionary=false
+
+-- One file of 1300 rows, so each block of 256 rows is four aligned 64-row 
chunks and the last 20
+-- rows are a partial chunk. Each block has its own validity pattern. Blocks 1 
and 2 have 16 and
+-- 15 valid rows per chunk, one at and one below `MIN_VALID_FOR_DENSE`, so 
they choose different
+-- branches. Update them if that constant changes.
+statement
+CREATE TABLE test_hash_fixed_width_nulls USING parquet AS
+SELECT /*+ COALESCE(1) */ id, IF(is_null, NULL, CAST(id * 7919 - 5000000 AS 
INT)) AS i,
+  IF(is_null, NULL, id * 1000000007 - 1000000000000) AS l,
+  IF(is_null, NULL, CASE id % 97 WHEN 0 THEN double('-0.0') WHEN 1 THEN 
double('NaN')
+    ELSE id / 7.0 - 50 END) AS d
+FROM (
+  SELECT id, CASE CAST(id DIV 256 AS INT)

Review Comment:
   The table only reaches the chunk loop through `int`, `bigint` and `double`, 
and only with `-0.0` and NaN as special values. The same loop also serves 
`float`, `tinyint`, `smallint`, `date` and `timestamp` through 
`hash_array_primitive` and `hash_array_primitive_float`, where `float` has its 
own 4-byte `hash_input` path. Adding a `CAST(d AS FLOAT)` column, a `smallint` 
column that includes its bounds, and a couple of `double` rows with `Infinity` 
and `-Infinity` would exercise those instantiations against Spark. The unit 
test only uses a `u64` closure, so it does not cover them.



##########
native/spark-expr/src/hash_funcs/utils.rs:
##########
@@ -18,6 +18,72 @@
 //! This includes utilities for hashing and murmur3 hashing.
 
 use arrow::array::Array;
+use arrow::buffer::NullBuffer;
+use datafusion::common::{internal_err, Result};
+
+/// The positions of the set bits of `bits`, lowest first.
+#[inline]
+fn set_bits(mut bits: u64) -> impl Iterator<Item = usize> {
+    std::iter::from_fn(move || {
+        (bits != 0).then(|| {
+            let i = bits.trailing_zeros() as usize;
+            bits &= bits - 1;
+            i
+        })
+    })
+}

Review Comment:
   `set_bits` is the same `trailing_zeros` and `bits &= bits - 1` loop that 
`native/spark-expr/src/conditional_funcs/case_when.rs` already writes out 
twice, in `copy_rows` and `fill_rows`. Rather than a third copy, this could 
live in one shared place (for example next to the other bit helpers in 
`datafusion-comet-common`) and be used by all three. If that is out of scope 
for this PR, a follow-up that switches `case_when.rs` to it would do. I have 
not checked whether the iterator form generates the same code as the open-coded 
`while` loops.



##########
native/spark-expr/src/hash_funcs/utils.rs:
##########
@@ -18,6 +18,72 @@
 //! This includes utilities for hashing and murmur3 hashing.
 
 use arrow::array::Array;
+use arrow::buffer::NullBuffer;
+use datafusion::common::{internal_err, Result};
+
+/// The positions of the set bits of `bits`, lowest first.
+#[inline]
+fn set_bits(mut bits: u64) -> impl Iterator<Item = usize> {
+    std::iter::from_fn(move || {
+        (bits != 0).then(|| {
+            let i = bits.trailing_zeros() as usize;
+            bits &= bits - 1;
+            i
+        })
+    })
+}
+
+/// The fewest valid rows in a 64-row chunk for which hashing every row of the 
chunk and restoring
+/// its null rows costs less than hashing only its valid rows one at a time.
+/// The null rates from 25% to 85% in the `hash_nulls` benchmark put random 
chunks around this
+/// count, so rerun them when changing the value.
+const MIN_VALID_FOR_DENSE: u32 = 16;

Review Comment:
   One constant serves both hashes, but the chunk copy is 256 bytes for murmur3 
and 512 bytes for xxhash64, and your reply on the review says the xxhash64 copy 
becomes a `memcpy` call. The break-even can therefore differ between them. 
Before settling on 16, the sweep needs the `xxhash64/int64/random_*` rows of 
`hash_nulls` as well as the murmur3 ones, and if they disagree this wants one 
constant per hash width. I have not measured this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to