[
https://issues.apache.org/jira/browse/SPARK-13840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li updated SPARK-13840:
----------------------------
Summary: Split Optimizer Rule ColumnPruning to ColumnPruning and
EliminateOperator (was: Disable Project Push Down Through Filter)
> Split Optimizer Rule ColumnPruning to ColumnPruning and EliminateOperator
> -------------------------------------------------------------------------
>
> Key: SPARK-13840
> URL: https://issues.apache.org/jira/browse/SPARK-13840
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Xiao Li
> Assignee: Xiao Li
> Priority: Critical
> Fix For: 2.0.0
>
>
> Now, ColumnPruning and PushPredicateThroughProject reverse each other's
> effect. Optimizer rules always reach the max iteration when optimizing some
> queries. Thus, we should disable columnPruning when hitting Filter for
> enabling predicate push down through project.
> The issue becomes worse when having another rule NullFiltering, which add
> extra Filters.
> In addition, columnPruning should not push Project through non-deterministic
> Filters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
