SaurabhChawla100 commented on a change in pull request #33232:
URL: https://github.com/apache/spark/pull/33232#discussion_r665089953
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1442,6 +1442,12 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
pushDownPredicate(filter, u.child) { predicate =>
u.withNewChildren(Seq(Filter(predicate, u.child)))
}
+
+ // Push down filter predicates in case filter having child as TypedFilter.
+ // In this scenario inorder to push the filter predicates there is need to
+ // to push Filter beneath the TypedFilter.
+ case Filter(condition, typeFilter @ TypedFilter(_, _, _, _, _)) =>
+ typeFilter.copy(child = Filter(condition, typeFilter.child))
Review comment:
@HyukjinKwon - Thank you for reviewing this PR.
Yes that makes sense for some function that are selecting the first n
records on each task and applying the filter on top of that.If the filter
sequence change the output will be different.
In my scenario which is having typeFilter on checking the value that exist
in the bloomFilter. I have to add this rule as the part of
"SparkSession.experimental.extraOptimizations = Seq()" then.
Any other suggestion on this.How to take on these kind of scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
