alamb commented on issue #20135: URL: https://github.com/apache/datafusion/issues/20135#issuecomment-4481410138
> Arrow-rs' Parquet reader doesn't let you push down virtual column filters anyway, unless I'm misunderstanding you [@AdamGS](https://github.com/AdamGS). From [#22026](https://github.com/apache/datafusion/pull/22026) > > > ParquetSource::try_pushdown_filters classifies filters against the file+partition schema (not the full table schema) so predicates referencing virtual columns are reported as PushedDown::No and the FilterExec stays above the scan — arrow-rs's RowFilter addresses parquet leaves only and can't evaluate virtual-column refs, so silently pushing them would produce wrong results. I think we added that feature recently: https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderOptions.html#method.with_virtual_columns -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
