mbutrovich commented on code in PR #7691:
URL: https://github.com/apache/datafusion-comet/pull/7691#discussion_r4645700409
##########
native/spark-expr/src/hash_funcs/utils.rs:
##########
@@ -18,6 +18,72 @@
//! This includes utilities for hashing and murmur3 hashing.
use arrow::array::Array;
+use arrow::buffer::NullBuffer;
+use datafusion::common::{internal_err, Result};
+
+/// The positions of the set bits of `bits`, lowest first.
+#[inline]
+fn set_bits(mut bits: u64) -> impl Iterator<Item = usize> {
+ std::iter::from_fn(move || {
+ (bits != 0).then(|| {
+ let i = bits.trailing_zeros() as usize;
+ bits &= bits - 1;
+ i
+ })
+ })
+}
+
+/// The fewest valid rows in a 64-row chunk for which hashing every row of the
chunk and restoring
+/// its null rows costs less than hashing only its valid rows one at a time.
+/// The null rates from 25% to 85% in the `hash_nulls` benchmark put random
chunks around this
+/// count, so rerun them when changing the value.
+const MIN_VALID_FOR_DENSE: u32 = 16;
Review Comment:
> Before settling on 16, the sweep needs the `xxhash64/int64/random_*` rows
of `hash_nulls` as well as the murmur3 ones, and if they disagree this wants
one constant per hash width.
Agreed. The sweep commands in the description run all of `hash_nulls`, so
each run includes the `xxhash64/int64/random_*` rows at every null rate. If
xxhash64's break-even differs from murmur3's, I'll split the constant by hash.
As you noted in
https://github.com/apache/datafusion-comet/pull/7691#pullrequestreview-5982563394,
this loop sees xxhash64 at the default seed only for float and struct
arguments and `approx_count_distinct`, so those are the workloads a separate
value would serve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]