szehon-ho commented on code in PR #56195:
URL: https://github.com/apache/spark/pull/56195#discussion_r3322828852
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:
##########
@@ -112,19 +112,26 @@ object PushDownUtils extends Logging {
}
}
- val rejectedFilters = r.pushPredicates(translatedFilters.toArray).map
{ predicate =>
- DataSourceV2Strategy.rebuildExpressionFromFilter(predicate,
translatedFilterToExpr)
- }
-
- val remainingFilters = (rejectedFilters ++ untranslatableExprs).toSeq
+ val postScanPredicates = r.pushPredicates(translatedFilters.toArray)
val postScanFilters =
+ rebuildExpressions(postScanPredicates.toSeq, translatedFilterToExpr)
++ untranslatableExprs
Review Comment:
Good catch, done. Moved the post-scan filter computation into the
non-iterative branch so it's not computed when iterative pushdown is active.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:
##########
@@ -134,6 +141,18 @@ object PushDownUtils extends Logging {
}
}
+ /**
+ * Rebuilds the Catalyst [[Expression]]s for a sequence of pushed-down
[[Predicate]]s, using the
Review Comment:
Done, reworded to "a sequence of data source predicates".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
