jackylee-ch commented on code in PR #795:
URL: https://github.com/apache/paimon-rust/pull/795#discussion_r3949949643
##########
crates/paimon/src/predicate_stats.rs:
##########
@@ -146,6 +144,19 @@ pub(crate) fn data_leaf_may_match<T: StatsAccessor>(
&& !matches!(literal.partial_cmp(&max_value),
Some(Ordering::Greater))
}
PredicateOperator::NotEq => !(min_value == *literal && max_value ==
*literal),
+ // The n-ary form of `NotEq` above: a file can only be skipped when
every
+ // non-null value in it is one forbidden literal, i.e. some literal
equals
+ // both bounds. Java `NotIn#test` does the same (`compareLiteral(lit,
min)
+ // == 0 && compareLiteral(lit, max) == 0`). Its extra `literal ==
null` arm
+ // has no counterpart because `Datum` has no null variant at all: the
REST
+ // parser folds `NOT IN (.., null)` into `AlwaysFalse`, and DataFusion
+ // declines to push the predicate down. Files that are entirely null
were
+ // already skipped by the shared `all_null` check, and files that are
+ // partly null are still pruned -- a null row satisfies neither
`NotIn` nor
+ // the equality that prunes it.
+ PredicateOperator::NotIn => !literals
+ .iter()
+ .any(|literal| min_value == *literal && max_value == *literal),
Review Comment:
Confirmed, and it already bites `<>` on main: on a file holding `[1.0, NaN]`
(min == max == 1.0, NaN counted as neither bound nor null) `<> 1.0` returned
`[3]` while the filter alone returns `[2, 3]`; on a `-0.0` file `<> 0.0`
returned nothing. `datum_cmp` is IEEE where the residual is bitwise, and Java's
`compareLiteral` uses `Double.compareTo`. Both operators now fail open for
FLOAT/DOUBLE, so this removes the existing loss rather than extending it.
Ordering rules and `Eq` are untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
