[
https://issues.apache.org/jira/browse/SPARK-19665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-19665:
------------------------------------
Assignee: Apache Spark
> Improve constraint propagation
> ------------------------------
>
> Key: SPARK-19665
> URL: https://issues.apache.org/jira/browse/SPARK-19665
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Liang-Chi Hsieh
> Assignee: Apache Spark
>
> If there are aliased expression in the projection, we propagate constraints
> by completely expanding the original constraints with aliases.
> This expanding costs much computation time when the number of aliases
> increases.
> Another issue is we actually don't need the additional constraints at most of
> time. For example, if there is a constraint "a > b", and "a" is aliased to
> "c" and "d". When we use this constraint in filtering, we don't need all
> constraints "a > b", "c > b", "d > b". We only need "a > b" because if it is
> false, it is guaranteed that all other constraints are false too.
> Fully expanding all constraints at all the time makes iterative ML algorithms
> where a ML pipeline with many stages runs very slow.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
