huaxingao commented on pull request #35669: URL: https://github.com/apache/spark/pull/35669#issuecomment-1061470359
Yes, I mean [FileScanBuilder.pushFilters](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala#L71) should split the partitionFilter and dataFilter, and only data filter should be pushed down to parquet/orc. But not splitting doesn't cause any issue anyways because later on Spark checks the filter and only creates parquet/orc filters for data filters. I will leave this as is so you can use it for your optimization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
