alamb opened a new issue, #3462: URL: https://github.com/apache/arrow-datafusion/issues/3462
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** DataFusion offers sophisticated "filter pushdown" optimizations into `LogicalPlan::TableScan` by passing predicates into TableProvider::scan. This ticket tracks the work to make use of these predicates in the table provider for parquet files, [`ParquetFileReader`](https://github.com/apache/arrow-datafusion/blob/d16457a0ba129b077935078e5cf89d028f598e0b/datafusion/core/src/physical_plan/file_format/parquet.rs#L693). Much of this work has been completed by the writing of this ticket, but I wanted to try and capture it here to both show how far DataFusion has come as well as how close we are to done There are three types of predicate pushdown: - [ ] Prune row groups based on statistics (do not fetch or decode any pages) - [ ] Prune column pages based on page level statistics, skip decode of corresponding positions in other columns - [ ] Prune row indexes based on `Expr` predicates, and skip decode of corresponding positions in other columns Work Items - [ ] https://github.com/apache/arrow-datafusion/issues/3360 - [ ] https://github.com/apache/arrow-datafusion/issues/3457 - [ ] https://github.com/apache/arrow-datafusion/issues/3456 - [ ] https://github.com/apache/arrow-datafusion/issues/847 - [ ] https://github.com/apache/arrow-datafusion/pull/2946 Related arrow-rs items: - [ ] https://github.com/apache/arrow-rs/issues/2110 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
