alamb opened a new pull request, #26418: URL: https://github.com/apache/datafusion/pull/26418
## Which issue does this PR close? - Part of https://github.com/apache/datafusion/issues/3463 (enable Parquet filter pushdown by default) - Related to https://github.com/apache/datafusion/issues/24393 **Note: this PR builds on https://github.com/apache/datafusion/pull/26384 (pin to arrow-rs `main`) and https://github.com/apache/datafusion/pull/26387 (`progressive_io` via `ParquetPushDecoder::scan_plan`) and can not be merged until the next arrow release.** Only the last commit is specific to this PR. ## Rationale for this change Parquet filter pushdown ("late materialization") has been off by default since it was added, mostly because of the performance regressions it causes on some workloads. With the I/O pattern now decoupled from filter evaluation (#26387) and the recent work on adaptive filters, it is time to measure what enabling it by default looks like. This PR flips two defaults so the benchmarks and the full test suite run with the new behavior: | option | old default | new default | |---|---|---| | `datafusion.execution.parquet.pushdown_filters` | `false` | `true` | | `datafusion.execution.parquet.progressive_io` | `false` | `true` | With both enabled, filters are evaluated during Parquet decoding, and when a file has an offset index the filter columns are fetched before the remaining projected columns (only for the rows that passed). Files without an offset index (such as the ClickBench dataset) are still read with one request per row group. ## What changes are included in this PR? - `pushdown_filters` and `progressive_io` default to `true` in `ParquetOptions` (plus the generated `configs.md`, `information_schema.slt` and the `format_options.md` table) - A note in the 56.0.0 upgrade guide, including how to restore the previous behavior - Comments and docs that described `pushdown_filters` as off by default are updated - Expected plans in 22 sqllogictest files are regenerated: the `FilterExec` above a Parquet `DataSourceExec` moves into the scan as `predicate=...`. Two files that reset `pushdown_filters` to `false` at the end now reset it to `true` so they leave the default configuration in place. There are no result (data) changes, only plan changes. - `parquet_max_row_group_bytes.slt` used the `FilterExec` line as the anchor for its `EXPLAIN ANALYZE` checks; the row-group count is now matched directly on the `DataSourceExec` line ## What is the testing strategy for this PR? Existing tests, regenerated where plans changed: the full sqllogictest suite, the `datafusion` core tests (including `parquet_integration` and `object_store_access`), and the `datafusion-datasource-parquet`, `datafusion-common`, `datafusion-proto` and `datafusion-cli` tests. ## Are there any user-facing changes? Yes: Parquet filter pushdown is on by default. Users who hit a regression can set `datafusion.execution.parquet.pushdown_filters = false` (see the upgrade guide). Benchmarks: `run benchmark clickbench_partitioned` on this PR compares the new defaults against `main`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
