liujiwen-up opened a new pull request, #488: URL: https://github.com/apache/paimon-rust/pull/488
### Purpose Linked issue: none. This PR adds conservative SQL `NOT` pushdown after the exact residual reader semantics introduced around filter pushdown. SQL `NOT` cannot be treated as exact because Paimon's predicate evaluation uses two-valued predicate semantics while DataFusion must preserve SQL three-valued logic, especially for NULL values. ### Brief change log - Push translatable SQL `NOT(expr)` predicates to Paimon as `Predicate::Not`, while marking the DataFusion filter pushdown result as `Inexact` so the residual filter is preserved. - Push case-sensitive `NOT LIKE` as an inexact negated LIKE predicate when the positive LIKE pattern can be translated; `ILIKE` remains unsupported. - Track whether translated predicates require a residual filter, so partition-only `NOT` is not incorrectly classified as exact. - Skip limit pushdown when the pushed filter requires a residual filter. - Add conservative stats pruning for `Predicate::Not(inner)`: prune only when the inner predicate must match every row in the file or data-evolution group; otherwise fail open. - Fix the btree global index builder constant reference from `BTREE_INDEX_TYPE` to `BTREE_GLOBAL_INDEX_TYPE`, which otherwise prevents the `paimon` crate tests from compiling on this branch. ### Tests - `cargo fmt -- --check`: passed - `git diff --check`: passed - `cargo test -p paimon-datafusion filter_pushdown --lib`: passed, 36 tests - `cargo test -p paimon-datafusion table::tests::test_scan_pushes_not_as_inexact_and_skips_limit_hint --lib`: passed - `cargo test -p paimon-datafusion --test sql_context_tests test_not_filter_pushdown_keeps_sql_null_semantics`: passed - `cargo test -p paimon predicate_stats --lib`: passed, 19 tests - `cargo test -p paimon table_scan::tests::test_data_file_matches_not --lib`: passed, 3 tests - `cargo test -p paimon table_scan::tests::test_data_evolution_group_matches_not --lib`: passed ### API and Format No public API, manifest, snapshot, or storage format changes are introduced. ### Documentation No user-facing documentation changes are required. The behavior is covered by DataFusion pushdown and core stats unit tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
