adriangb commented on PR #15566: URL: https://github.com/apache/datafusion/pull/15566#issuecomment-2809592722
> I've also another question. When parquet_options.pushdown_filters is true, we can pushdown filters and remove the FilterExec's from the plan. When parquet_options.pushdown_filters is false, we can still pushdown filters but cannot remove the FilterExec's from the plan. > > Is this our motivation? It's a bit more complicated than that, and it may be hard to match the current behavior exactly. For now for this PR I'd say what we want is: - If `parquet_options.pushdown_filters = true` then `ParquetSource` accepts filter pushdown as exact - If `parquet_options.pushdown_filters = false` then `ParquetSource` does not accept filter pushdown at all \* \* In theory there's little harm in accepting pushdown only into stats pruning but I think that might be hard to get right / require new APIs and realistically this is mostly a temporary issue because once #3463 gets resolved the default will be `parquet_options.pushdown_filters = true` so if `false` is suboptimal it's not a big deal -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
