cloud-fan commented on code in PR #37176:
URL: https://github.com/apache/spark/pull/37176#discussion_r920746311
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:
##########
@@ -117,13 +117,24 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with
PredicateHelper with J
* do not add a subquery that might have an expensive computation
*/
private def isSelectiveFilterOverScan(plan: LogicalPlan): Boolean = {
- val ret = plan match {
- case PhysicalOperation(_, filters, child) if
child.isInstanceOf[LeafNode] =>
Review Comment:
The old behavior was, collecting all the filters above a leaf node and
checking if all filters are simple expressions and at least one of them is
selective.
I've rewritten the code to keep the old behavior but in a more efficient
way: simply traverse the plan tree to check filter predicates, instead of
merging expressions and collecting filters that may duplicate complicated
expressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
