cloud-fan commented on code in PR #56195:
URL: https://github.com/apache/spark/pull/56195#discussion_r3321892304
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:
##########
@@ -112,19 +112,26 @@ object PushDownUtils extends Logging {
}
}
- val rejectedFilters = r.pushPredicates(translatedFilters.toArray).map
{ predicate =>
- DataSourceV2Strategy.rebuildExpressionFromFilter(predicate,
translatedFilterToExpr)
- }
-
- val remainingFilters = (rejectedFilters ++ untranslatableExprs).toSeq
+ val postScanPredicates = r.pushPredicates(translatedFilters.toArray)
val postScanFilters =
+ rebuildExpressions(postScanPredicates.toSeq, translatedFilterToExpr)
++ untranslatableExprs
Review Comment:
Minor: `postScanFilters` is only consumed by the non-iterative branch below
— the iterative branch rebuilds the expressions again (partitioned into
pushed/not-pushed), so this value is dead work whenever iterative pushdown is
active. Slightly cleaner to compute it inside the non-iterative branch:
```
val finalPostScanFilters =
if (!partitionFields.exists(_.nonEmpty) || !r.supportsIterativePushdown) {
rebuildExpressions(postScanPredicates.toSeq, translatedFilterToExpr) ++
untranslatableExprs
} else {
...
```
Non-blocking.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:
##########
@@ -134,6 +141,18 @@ object PushDownUtils extends Logging {
}
}
+ /**
+ * Rebuilds the Catalyst [[Expression]]s for a sequence of pushed-down
[[Predicate]]s, using the
Review Comment:
Nit: every call site passes post-scan / returned predicates (the full
returned set, the not-pushed subset, and the partially-pushed subset) — most of
which are not pushed down. "pushed-down" is misleading here.
```suggestion
* Rebuilds the Catalyst [[Expression]]s for a sequence of data source
[[Predicate]]s, using the
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
