QuakeWang commented on code in PR #795:
URL: https://github.com/apache/paimon-rust/pull/795#discussion_r3949577442
##########
crates/paimon/src/predicate_stats.rs:
##########
@@ -146,6 +144,19 @@ pub(crate) fn data_leaf_may_match<T: StatsAccessor>(
&& !matches!(literal.partial_cmp(&max_value),
Some(Ordering::Greater))
}
PredicateOperator::NotEq => !(min_value == *literal && max_value ==
*literal),
+ // The n-ary form of `NotEq` above: a file can only be skipped when
every
+ // non-null value in it is one forbidden literal, i.e. some literal
equals
+ // both bounds. Java `NotIn#test` does the same (`compareLiteral(lit,
min)
+ // == 0 && compareLiteral(lit, max) == 0`). Its extra `literal ==
null` arm
+ // has no counterpart because `Datum` has no null variant at all: the
REST
+ // parser folds `NOT IN (.., null)` into `AlwaysFalse`, and DataFusion
+ // declines to push the predicate down. Files that are entirely null
were
+ // already skipped by the shared `all_null` check, and files that are
+ // partly null are still pruned -- a null row satisfies neither
`NotIn` nor
+ // the equality that prunes it.
+ PredicateOperator::NotIn => !literals
+ .iter()
+ .any(|literal| min_value == *literal && max_value == *literal),
Review Comment:
This can drop matching FLOAT/DOUBLE rows: `Datum` treats signed zeros as
equal while the residual distinguishes them, and Parquet min/max excludes NaNs.
For example, `[1.0, NaN] NOT IN (1.0, 2.0)` is pruned entirely instead of
retaining the NaN row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
