kosiew commented on code in PR #26609:
URL: https://github.com/apache/datafusion/pull/26609#discussion_r4582752550
##########
datafusion/physical-expr/src/expressions/case/literal_lookup_table/primitive_lookup_table.rs:
##########
@@ -12,6 +12,26 @@ use half::f16;
use std::fmt::Debug;
use std::hash::Hash;
+/// The maximum total size in bytes of the literals that are looked up by
comparing each
+/// value with all of them, rather than with a hash map.
+///
+/// The cost of the comparisons grows with the number of literals and with
their size,
+/// as fewer wider values fit in a vector register.
+const MAX_LINEAR_LOOKUP_BYTES: usize = 128;
+
+/// The maximum number of literals that are looked up by comparing each value
with all
+/// of them, for values narrow enough for many of them to fit
[MAX_LINEAR_LOOKUP_BYTES].
+const MAX_LINEAR_LOOKUP_LITERALS: usize = 32;
+
+/// The maximum number of literals of type `K` that are looked up by comparing
each
+/// value with all of them, rather than with a hash map.
+fn max_linear_lookup_literals<K>() -> usize {
Review Comment:
The 128-byte threshold admits eight Decimal128 literals, but the latest
same-case benchmark regresses from 47.5 to 64.4 µs (~36% slower); earlier local
runs also showed a reproducible ~45% slowdown with 50% nulls. Please lower the
wide-key limit or improve the kernel, then benchmark all-valid and nullable
inputs against hash lookup. A representation-specific four-key cap is a
reasonable candidate given the measured improvement at four literals, but
please verify it without unnecessarily reducing the i32/i64 limits.
##########
datafusion/physical-expr/src/expressions/case.rs:
##########
@@ -3289,6 +3289,37 @@ mod tests {
);
}
+ /// Covers both the linear and the hash lookup of primitive literals, with
inputs
+ /// spanning several chunks of the linear lookup.
+ #[rstest::rstest]
+ fn test_case_when_literal_lookup_int64_by_number_of_literals(
Review Comment:
Optional test hardening: could we evaluate a nullable `Int64Array` slice
starting at a non-byte-aligned offset, with matches and misses spanning the
256-row boundary? The current test covers chunking but only unsliced arrays;
reusing the ELSE/no-ELSE helper would verify that values and validity remain
aligned. No offset bug was found.
##########
datafusion/physical-expr/src/expressions/case/literal_lookup_table/primitive_lookup_table.rs:
##########
@@ -180,7 +269,19 @@ impl_to_hashable_key!(@float | f64 => u64);
#[cfg(test)]
mod tests {
- use super::ToHashableKey;
+ use super::{ToHashableKey, max_linear_lookup_literals};
+ use arrow::datatypes::{IntervalMonthDayNano, i256};
+
+ #[test]
+ fn max_linear_lookup_literals_by_type() {
+ assert_eq!(max_linear_lookup_literals::<i8>(), 32);
+ assert_eq!(max_linear_lookup_literals::<u16>(), 32);
+ assert_eq!(max_linear_lookup_literals::<i32>(), 32);
+ assert_eq!(max_linear_lookup_literals::<i64>(), 16);
+ assert_eq!(max_linear_lookup_literals::<i128>(), 8);
+ assert_eq!(max_linear_lookup_literals::<IntervalMonthDayNano>(), 8);
+ assert_eq!(max_linear_lookup_literals::<i256>(), 4);
+ }
use arrow::array::downcast_primitive;
Review Comment:
Minor cleanup: `use arrow::array::downcast_primitive;` now appears after the
new test. Could we move it alongside the other imports at the top of the test
module?
##########
datafusion/physical-expr/src/expressions/case/literal_lookup_table/primitive_lookup_table.rs:
##########
@@ -180,7 +269,19 @@ impl_to_hashable_key!(@float | f64 => u64);
#[cfg(test)]
mod tests {
- use super::ToHashableKey;
+ use super::{ToHashableKey, max_linear_lookup_literals};
+ use arrow::datatypes::{IntervalMonthDayNano, i256};
+
+ #[test]
+ fn max_linear_lookup_literals_by_type() {
+ assert_eq!(max_linear_lookup_literals::<i8>(), 32);
+ assert_eq!(max_linear_lookup_literals::<u16>(), 32);
+ assert_eq!(max_linear_lookup_literals::<i32>(), 32);
+ assert_eq!(max_linear_lookup_literals::<i64>(), 16);
+ assert_eq!(max_linear_lookup_literals::<i128>(), 8);
+ assert_eq!(max_linear_lookup_literals::<IntervalMonthDayNano>(), 8);
+ assert_eq!(max_linear_lookup_literals::<i256>(), 4);
Review Comment:
Optional coverage: could we add a Decimal256 execution test with four and
five unique literals, including values with high bits beyond i64, nulls, and
misses? This would exercise wide-key equality and both sides of the linear/hash
boundary rather than only asserting the four-key threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]