peterxcli opened a new pull request, #25000: URL: https://github.com/apache/datafusion/pull/25000
## Which issue does this PR close? - Closes #19193. - Closes #18860; page-level LIMIT pruning is the epic's last open sub-issue. ## Rationale for this change Row-group LIMIT pruning cannot skip partially matching row groups even when a few pages within them contain enough guaranteed matches. For example, with pages `[0, 10, 0]`, `[5, 6, 7]`, and `[0, 8, 0]`, `WHERE a >= 5 LIMIT 3` can read only the middle page. This extends the row-group optimization introduced in #18868 to page ranges for scans that do not need to preserve order. ## What changes are included in this PR? - Share the null-safe inverse-predicate construction between row-group and page pruning in a private Parquet module. - Use page statistics to prove fully matching row ranges, intersect them across all supported conjuncts and existing row selections, and combine them with fully matched row groups. - Rewrite the access plan only when the guaranteed rows satisfy the limit. Preserve existing full-match flags and retain the ordinary plan when proofs are unavailable or insufficient. - Disable the page-level LIMIT rewrite for order-preserving scans, including when the optimizer removes an already-satisfied sort. - Add and document the lazily registered `limit_pruned_rows` metric. - Fix a pre-existing all-features Clippy warning in the PostgreSQL test helper by borrowing its decimal argument. ## What is the testing strategy for this PR? The Parquet integration tests cover selecting a fully matched page, combining pages and row groups, NULLs, unsupported conjuncts, insufficient guaranteed rows, missing page indexes, and an eliminated `ORDER BY` sort. Unit tests cover existing row selections, full-match flags, and the opener's order-preservation gate. The full-match assertion was verified to fail before its fix. Temporarily removing the page-level order guard also makes the SQL ordering regression fail with different row IDs. Local validation: - 95 page-pruning integration tests passed. - 16 row-group filter unit tests passed, along with the access-plan and opener ordering regressions. - `cargo fmt --all`. - `cargo clippy --all-targets --all-features -- -D warnings`. - `./dev/rust_lint.sh`. Performance benchmarks are not yet available. The inverse-statistics pass adds work when it cannot prove enough matches, so both successful-pruning and no-benefit cases need measurement; the tests demonstrate pruning behavior, not a measured speedup. ## Are there any user-facing changes? Eligible unordered Parquet LIMIT queries can skip additional row ranges. EXPLAIN ANALYZE reports `limit_pruned_rows` when this optimization removes rows. No new configuration or public API is introduced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
