huaxingao commented on a change in pull request #33650:
URL: https://github.com/apache/spark/pull/33650#discussion_r692317321
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
##########
@@ -57,6 +64,27 @@ abstract class FileScanBuilder(
StructType(fields)
}
+ def pushFilters(partitionFilters: Seq[Expression], dataFilters:
Seq[Expression]): Unit = {
+ this.partitionFilters = partitionFilters
+ this.dataFilters = dataFilters
+ this.pushedDataFilters = pushDataFilters(translateDataFilter)
+ }
+
+ protected def pushDataFilters(dataFilters: Array[Filter]): Array[Filter] =
Array.empty[Filter]
+
+ protected def translateDataFilter(): Array[Filter] = {
Review comment:
Done. Thanks!
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
##########
@@ -57,6 +64,27 @@ abstract class FileScanBuilder(
StructType(fields)
}
+ def pushFilters(partitionFilters: Seq[Expression], dataFilters:
Seq[Expression]): Unit = {
+ this.partitionFilters = partitionFilters
+ this.dataFilters = dataFilters
+ this.pushedDataFilters = pushDataFilters(translateDataFilter)
+ }
+
+ protected def pushDataFilters(dataFilters: Array[Filter]): Array[Filter] =
Array.empty[Filter]
Review comment:
`pushFilters` means to push the filters (both `partitionFilters` and
`dataFilters`) from spark to the file source. Since now file source has all the
`dataFilters`, it can call the `pushDataFilters` method to find out which data
filters can be processed in the file source, and set the `pushedDataFilters` to
be these filters, so they later can be converted to data source filters and
pushed down to the data sources. Hope this is clear :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
