sunchao commented on a change in pull request #33584:
URL: https://github.com/apache/spark/pull/33584#discussion_r680130284
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala
##########
@@ -74,6 +74,34 @@ object PushDownUtils extends PredicateHelper {
}
}
+ /**
+ * Pushes down partition filters and data filters to the data source reader
+ *
+ * @return pushed partition filters.
+ */
+ def pushPartitionFilters(
Review comment:
I think this name is a bit confusing: it doesn't **push** the partition
filters but rather just set the filters in the scan builder?
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala
##########
@@ -74,6 +74,34 @@ object PushDownUtils extends PredicateHelper {
}
}
+ /**
+ * Pushes down partition filters and data filters to the data source reader
+ *
+ * @return pushed partition filters.
+ */
+ def pushPartitionFilters(
+ scanBuilder: ScanBuilder,
+ relation: DataSourceV2Relation,
+ normalizedFilters: Seq[Expression]): Seq[Expression] = {
+ scanBuilder match {
+ case fileBuilder: FileScanBuilder =>
+ val partitionColumns = relation.resolve(
Review comment:
can we move this into a separate util method? so that we can share it
between here and `PruneFileSourcePartitions`?
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
##########
@@ -57,6 +60,15 @@ abstract class FileScanBuilder(
StructType(fields)
}
+ def setFilters(pFilters: Seq[Expression], dFilters: Seq[Expression]): Unit =
{
+ partitionFilters = pFilters
+ dataFilters = dFilters
+ }
+
+ def getPartitionFilters(): Seq[Expression] = partitionFilters
Review comment:
nit: remove `()`.
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
##########
@@ -57,6 +60,15 @@ abstract class FileScanBuilder(
StructType(fields)
}
+ def setFilters(pFilters: Seq[Expression], dFilters: Seq[Expression]): Unit =
{
Review comment:
should we make these two fields immutable and have a `withFilters`
method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
