feniljain opened a new issue, #18171: URL: https://github.com/apache/datafusion/issues/18171
### Describe the bug For queries like this: ```sql select sum(ss_quantity) from store_sales, date_dim where ss_sold_date_sk=d_date_sk and d_year='2001'; ``` where in `store_sales` table, `ss_sold_date_sk` is a partition column. For these queries, dynamic filter pushdown does not cross [try_pushdown_filters](https://github.com/e6data/e6-datafusion/blob/149d3e9533acadfb92329f5445d7deb72f6f679c/datafusion/datasource-parquet/src/source.rs#L697) of `DataSourceExec` because we don't pushdown filters containing filter columns [here](https://github.com/e6data/e6-datafusion/blob/149d3e9533acadfb92329f5445d7deb72f6f679c/datafusion/datasource-parquet/src/source.rs#L722-L726). We do another round of file pruning for dynamic filters [here](https://github.com/e6data/e6-datafusion/blob/149d3e9533acadfb92329f5445d7deb72f6f679c/datafusion/datasource-parquet/src/opener.rs#L177-L193) in `FileOpener`, this does skip files. Although it would be better to update partitions to be read using dynamic filters bounds, but this is enough for filtering them down for now at least. ### To Reproduce _No response_ ### Expected behavior _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
