comphead commented on code in PR #7691: URL: https://github.com/apache/datafusion-comet/pull/7691#discussion_r4645582477
########## spark/src/test/resources/sql-tests/expressions/hash/hash_fixed_width_nulls.sql: ########## @@ -0,0 +1,47 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +-- Plain encoding keeps the columns off the dictionary path, so they reach the 64-row chunk loop +-- in `hash_fixed_width_with_nulls`. +-- Config: parquet.enable.dictionary=false + +-- One file of 1300 rows, so each block of 256 rows is four aligned 64-row chunks and the last 20 +-- rows are a partial chunk. Each block has its own validity pattern. Blocks 1 and 2 have 16 and +-- 15 valid rows per chunk, one at and one below `MIN_VALID_FOR_DENSE`, so they choose different +-- branches. Update them if that constant changes. +statement +CREATE TABLE test_hash_fixed_width_nulls USING parquet AS +SELECT /*+ COALESCE(1) */ id, IF(is_null, NULL, CAST(id * 7919 - 5000000 AS INT)) AS i, + IF(is_null, NULL, id * 1000000007 - 1000000000000) AS l, + IF(is_null, NULL, CASE id % 97 WHEN 0 THEN double('-0.0') WHEN 1 THEN double('NaN') + ELSE id / 7.0 - 50 END) AS d +FROM ( + SELECT id, CASE CAST(id DIV 256 AS INT) Review Comment: The table only reaches the chunk loop through `int`, `bigint` and `double`, and only with `-0.0` and NaN as special values. The same loop also serves `float`, `tinyint`, `smallint`, `date` and `timestamp` through `hash_array_primitive` and `hash_array_primitive_float`, where `float` has its own 4-byte `hash_input` path. Adding a `CAST(d AS FLOAT)` column, a `smallint` column that includes its bounds, and a couple of `double` rows with `Infinity` and `-Infinity` would exercise those instantiations against Spark. The unit test only uses a `u64` closure, so it does not cover them. ########## native/spark-expr/src/hash_funcs/utils.rs: ########## @@ -18,6 +18,72 @@ //! This includes utilities for hashing and murmur3 hashing. use arrow::array::Array; +use arrow::buffer::NullBuffer; +use datafusion::common::{internal_err, Result}; + +/// The positions of the set bits of `bits`, lowest first. +#[inline] +fn set_bits(mut bits: u64) -> impl Iterator<Item = usize> { + std::iter::from_fn(move || { + (bits != 0).then(|| { + let i = bits.trailing_zeros() as usize; + bits &= bits - 1; + i + }) + }) +} Review Comment: `set_bits` is the same `trailing_zeros` and `bits &= bits - 1` loop that `native/spark-expr/src/conditional_funcs/case_when.rs` already writes out twice, in `copy_rows` and `fill_rows`. Rather than a third copy, this could live in one shared place (for example next to the other bit helpers in `datafusion-comet-common`) and be used by all three. If that is out of scope for this PR, a follow-up that switches `case_when.rs` to it would do. I have not checked whether the iterator form generates the same code as the open-coded `while` loops. ########## native/spark-expr/src/hash_funcs/utils.rs: ########## @@ -18,6 +18,72 @@ //! This includes utilities for hashing and murmur3 hashing. use arrow::array::Array; +use arrow::buffer::NullBuffer; +use datafusion::common::{internal_err, Result}; + +/// The positions of the set bits of `bits`, lowest first. +#[inline] +fn set_bits(mut bits: u64) -> impl Iterator<Item = usize> { + std::iter::from_fn(move || { + (bits != 0).then(|| { + let i = bits.trailing_zeros() as usize; + bits &= bits - 1; + i + }) + }) +} + +/// The fewest valid rows in a 64-row chunk for which hashing every row of the chunk and restoring +/// its null rows costs less than hashing only its valid rows one at a time. +/// The null rates from 25% to 85% in the `hash_nulls` benchmark put random chunks around this +/// count, so rerun them when changing the value. +const MIN_VALID_FOR_DENSE: u32 = 16; Review Comment: One constant serves both hashes, but the chunk copy is 256 bytes for murmur3 and 512 bytes for xxhash64, and your reply on the review says the xxhash64 copy becomes a `memcpy` call. The break-even can therefore differ between them. Before settling on 16, the sweep needs the `xxhash64/int64/random_*` rows of `hash_nulls` as well as the murmur3 ones, and if they disagree this wants one constant per hash width. I have not measured this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
