cloud-fan commented on pull request #35395: URL: https://github.com/apache/spark/pull/35395#issuecomment-1070968323
> I think we can achieve that by reusing the same Scan object in the main and runtime filtering queries. This may be suboptimal because it's two different scans, with different column pruning and predicate pushdown. The scan to collect group ids only needs to return columns referred by the DELETE condition and the hidden column, while the scan to get the input data requires all the columns. Different predicate pushdown is also an important factor. Some data sources have multiple levels to handle predicate, such as file level and parquet row group in Delta Lake. We need to create a fresh `ScanBuilder` to apply dedicated column pruning and predicate pushdown required by the query. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
