[
https://issues.apache.org/jira/browse/SPARK-36878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asif resolved SPARK-36878.
--------------------------
Resolution: Not A Bug
Further inputs from my company colleague, who pointed out that there might be
some other rule which would be handling the case, it turns out
BooleanSimplificationRule which runs before PruneFilter would reduce the
composite condition into a single condition ( by prunning unnecessary
conditions) as a result the current PruneFilter code is sufficient. It does not
have to handle the composite conditions containing nullls or false or true.
> Optimization in PushDownPredicates to push all filters in a single iteration
> has broken some optimizations in PruneFilter rule
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-36878
> URL: https://issues.apache.org/jira/browse/SPARK-36878
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1
> Reporter: Asif
> Priority: Major
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> It appears that the optimization in PushDownPredicates rule to try to push
> all filters in a single pass to reduce iteration has broken the PruneFilter
> rule to substitute with EmptyRelation when the filter condition is a
> composite and statically evaluates to false either because one of the non
> redundant predicate is Literal(false) or all the non redundant predicates are
> null.
> The new PushDownPredicate rule is created by chaining CombineFilters,
> PushPredicateThroughNonJoin and PushPredicateThroughJoin.
> so individual filters will get combined as a single filter while being pushed.
> But the PruneFilters rule does not substitute it with empty relation if the
> filter is composite. It is coded to handle single predicates.
> The test is falsely passing as it is testing PushPredicateThroughNonJoin,
> which does not combine filters.
> While the actual rule in action has an effect produced by CombineFilters.
> In fact I believe all the places in other tests which are testing
> individually for PushDownPredicateThroughNonJoin or
> PushDownPredicateThroughJoin should be corrected ( may be with rule
> PushPredicates) & re tested.
> I will add a bug test & open PR.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
