jensholdgaard opened a new pull request, #24770: URL: https://github.com/apache/datafusion/pull/24770
> [!NOTE] > This fix was investigated and written with AI assistance (Claude Code), posted with the account owner's review and consent. ## Which issue does this PR close? - Closes #24769. ## Rationale for this change A query like `SELECT ... WHERE col = 'x'` over a Parquet file whose statistics say `col` is entirely NULL cannot match any row, and DataFusion 54 pruned such row groups from the footer without reading them. On 55 the same query reads them. The cause is an interaction rather than a bug in pruning itself. `constant_columns_from_stats` substitutes columns that file statistics prove constant, and its all-NULL branch folds such a column to a NULL literal. Once substituted, the predicate simplifies to a bare constant — `NULL` here — at which point `build_pruning_predicates` returns `None` (there are no column references left to build a pruning predicate over), and `prune_row_groups` falls through with no pruning at all. So for exactly the files where the statistics carry the *most* information, the substitution is strictly counterproductive: before it, the pruning predicate's own `col_null_count != row_count` conjunct proved the row group empty and skipped it. This surfaced as a regression when bisecting a real workload (a Parquet log store where a `body` column is NULL for the vast majority of rows) from 54 → 55. The bisect lands on #22969, which removed `ListingOptions::collect_stat` in favour of the session's `execution.collect_statistics` — default `true`. That change is correct in itself; it simply began feeding per-file statistics to the substitution on paths that previously had none, exposing the gap. Setting `execution.collect_statistics = false` restores pruning on 55, which is a useful confirmation but obviously not a fix. Results were never wrong — a filter drops NULL and false rows alike — but the scan work is real: row groups that used to be skipped from the footer are now decoded in full. ## What changes are included in this PR? `prune_row_groups` now recognises the collapsed-to-constant case: if the (post-substitution, post-simplification) predicate is a `false` or NULL literal, every remaining row group is skipped and the skip is credited to `row_groups_pruned_statistics`. A small `RowGroupAccessPlanFilter::skip_all` helper is added alongside the existing `prune_by_*` methods. The check is deliberately narrow — a downcast to `Literal` plus a NULL/`false` value test — so it cannot affect predicates that still reference columns; those take the existing path unchanged. ## Are these changes tested? Yes: `test_prune_all_null_column_equality_from_file_statistics` in `opener/mod.rs`, modelled on the neighbouring `test_prune_on_partition_values_and_file_statistics`. It fails on current `main` (3 rows scanned, 0 pruned) and passes with this change. One note on how it asserts, in case it saves a reviewer time: it checks the `row_groups_pruned_statistics` metric rather than the returned row count. A row-count assertion cannot distinguish "pruned" from "scanned, then row-filtered" — both give zero rows — and an earlier draft of this test passed against the unfixed code for exactly that reason. Verified `cargo test -p datafusion-datasource-parquet --lib opener::` — 47 passed. The 16 pre-existing `bloom_filter::tests` failures on this crate are unrelated and reproduce identically on an untouched checkout. ## Are there any user-facing changes? No API or result changes. Queries that were already correct stay correct; affected scans read less data. Plans may show more row groups pruned, and `row_groups_pruned_statistics` increases correspondingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
