jackylee-ch opened a new pull request, #795: URL: https://github.com/apache/paimon-rust/pull/795
On the stats rule, `data_leaf_may_match` returned `true` for `NotIn` unconditionally, so `col NOT IN (...)` never used min/max or the null count to skip anything. That was deliberate — #382 scoped `IN` pruning and said `NOT IN` would keep failing open — and this lifts it. The rule is the n-ary form of the `NotEq` rule just above it: skip a file only when some literal equals both bounds, i.e. every non-null row is forbidden. Java `NotIn#test` does the same. Its extra null-literal arm has no counterpart because `Datum` has no null variant at all — the REST parser folds `NOT IN (.., null)` into `AlwaysFalse`, and DataFusion declines to push such a predicate down. Files that are partly null are still pruned: a null row satisfies neither `NotIn` nor the equality that prunes it. Not gated on `supports_in_min_max_pruning`, unlike `IN`. `NotEq` is not gated either, and `x NOT IN (5)` has to answer like `x <> 5`; a test pins that equivalence across missing, inverted, unknown-null and all-null stats. The consequence is that every consumer of this function gains the rule at once — manifest partition stats, key stats, Parquet row groups and page index, and mosaic — not only data-file stats as in #382. Results do not change, only how many files are read. On a three-file table where one file holds a single value, `value NOT IN (20)` moves `manifest_entries_pruned_by_data_stats` from 0 to 1 and `final_files` from 3 to 2. `data_leaf_must_match` is untouched; the reverse direction deserves its own tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
