jinjin1919 commented on code in PR #7569:
URL: https://github.com/apache/datafusion-comet/pull/7569#discussion_r4559481640
##########
native/spark-expr/src/hash_funcs/murmur3.rs:
##########
@@ -1148,4 +1148,202 @@ mod tests {
"a dictionary child must keep the per-element path"
);
}
+
+ /// `create_hashes_internal!` specialises maps of (Utf8, Int32), (Int32,
Utf8), (Utf8, Utf8)
+ /// and (Int32, Int32), each with a loop for input without nulls and a
null-aware loop. Covers
+ /// both loops of every pair for both hashes. The null shape has a null
map that still owns an
+ /// entry, which must not be hashed, and a null value, which is skipped
while its key is not.
+ /// Expected values are Spark 4.1.3's `hash` and `xxhash64` (seed 42) on
the same rows.
+ #[test]
+ fn test_specialised_map_pairs_match_spark() {
+ use crate::hash_funcs::create_xxhash64_hashes;
+ use arrow::array::{Array, MapArray, StructArray};
+ use arrow::buffer::{NullBuffer, OffsetBuffer};
+ use arrow::datatypes::{DataType, Field, Fields};
+
+ const STR_KEYS: [&str; 7] = ["a", "b", "", "日本", "x", "yz", "hidden"];
+ const INT_KEYS: [i32; 7] = [1, -1, 0, i32::MAX, i32::MIN, 7, 99];
+ const STR_VALUES: [&str; 7] = ["v1", "", "é", "w", "long value", "z",
"hidden"];
+ const INT_VALUES: [i32; 7] = [10, -20, 0, i32::MIN, i32::MAX, 3, 99];
+
+ // The first `len` entries of a pool, with entry `null_at` null.
+ fn column(
+ utf8: bool,
+ strs: &[&str],
+ ints: &[i32],
+ len: usize,
+ null_at: Option<usize>,
+ ) -> ArrayRef {
+ let valid = |i: usize| Some(i) != null_at;
+ if utf8 {
+ Arc::new(
+ (0..len)
+ .map(|i| valid(i).then(|| strs[i]))
+ .collect::<StringArray>(),
+ )
+ } else {
+ Arc::new(
+ (0..len)
+ .map(|i| valid(i).then(|| ints[i]))
+ .collect::<Int32Array>(),
+ )
+ }
+ }
+
+ fn map(key_utf8: bool, value_utf8: bool, shape: Shape) -> ArrayRef {
+ let (lengths, valid, null_value) = shape;
+ let len = lengths.iter().sum();
+ let keys = column(key_utf8, &STR_KEYS, &INT_KEYS, len, None);
+ let values = column(value_utf8, &STR_VALUES, &INT_VALUES, len,
null_value);
+ let entry_fields = Fields::from(vec![
+ Field::new("key", keys.data_type().clone(), false),
+ Field::new("value", values.data_type().clone(), true),
+ ]);
+ let entries = StructArray::new(entry_fields.clone(), vec![keys,
values], None);
+ Arc::new(MapArray::new(
+ Arc::new(Field::new("entries", DataType::Struct(entry_fields),
false)),
+ OffsetBuffer::from_lengths(lengths.iter().copied()),
+ entries,
+ valid
+ .contains(&false)
+ .then(|| NullBuffer::from(valid.to_vec())),
+ false,
+ ))
+ }
+
+ // Entries per row, map validity per row, and the entry whose value is
null.
+ type Shape = (&'static [usize], &'static [bool], Option<usize>);
+ const NO_NULLS: Shape = (&[2, 0, 1, 3], &[true; 4], None);
+ const WITH_NULLS: Shape = (&[2, 0, 1, 1, 3], &[true, true, false,
true, true], Some(3));
+
+ // (pair, key is Utf8, value is Utf8, [murmur3, xxhash64] without
nulls, the same with nulls)
+ #[allow(clippy::type_complexity)]
+ let cases: [(&str, bool, bool, (Vec<u32>, Vec<u64>), (Vec<u32>,
Vec<u64>)); 4] = [
Review Comment:
Thanks, I made a new commit (#decfaff) to this PR based on your feedback.
I added map cases to expressions/hash/hash.sql for all four specialised pairs:
one table with no nulls at all, and one with a null map and null values. Those
are compared with Spark at run time, so I removed the hardcoded murmur3 and
xxhash64 constants.
I also confirmed that a Parquet map column with no nulls reaches the fast
loop. A temporary panic in that loop fired for the no-null query. Swapping key
and value in the loop makes hash.sql fail.
The case SQL can't build, a null map row that still owns an entry, is now
covered by `test_specialised_map_null_row_skips_owned_entries`. It checks that
the null row keeps the seed, and that the other rows hash the same as they do
with that row filtered out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]