LuciferYang commented on PR #60053: URL: https://github.com/apache/spark/pull/60053#issuecomment-6624279810
Thanks for the review. Your second question holds: the DSv2 contracts only require that the rows a scan returns match a filter it fully accepted. A source can accept `part = 3`, apply it row by row in its readers, and leave the pruning of its planned partitions to DPP, so removing the DPP filter there can turn a pruned scan into a full one. Spark can count on that pruning only for its own file scans, which apply partition filters to the file listing. Those scans get runtime filtering in #55579, so on master the narrowed rule would change nothing. I'm closing this PR and will make the change in #55579 for the built-in file scans only, along with the Scaladoc and test-name fixes you suggested. The fixture nit goes away, since #55579 does not need the in-memory fixture. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
