peter-toth commented on PR #59715: URL: https://github.com/apache/spark/pull/59715#issuecomment-6398087156
@dongjoon-hyun @szehon-ho @viirya @cloud-fan @aokolnychyi, this is now ready for review. It extends late materialization to data source V2 scans, the next step after the V1 Parquet support in #58895 (SPARK-59620). A DSv2 reader can then evaluate the runtime bloom filter inside the scan, and skip reading the other columns of the rows it drops. Your feedback would be much appreciated. Since this adds to the DSv2 API, I'm happy to write an SPIP if you think it needs one. The API follows the shape of `PartitionPredicate` (SPARK-55596). It is an opaque predicate that Spark implements, and the connector only supplies the values it reads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
