stczwd edited a comment on pull request #35669: URL: https://github.com/apache/spark/pull/35669#issuecomment-1061558999
> How hard it is to support file source v1? This is a big PR and it's a pity if it only works for file source v2 which is off by default, @cloud-fan Thanks for your attension. Actually, there are not many changes in this PR, most of the changes are in the benchmark results. As for supporting os file source v1, we split dataFilters and partitionFilters at [FileSourceStrategy.apply](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L188), thus we need a way to push whole filters to `Fileformat.buildReaderWithPartitionValues`. Not a big PR, but it can be used in ORC v1 file source. Maybe another PR is better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
