[jira] [Commented] (SPARK-13919) Resolving the Conflicts of ColumnPruning and PushPredicateThroughProject

2016-03-19 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-13919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202116#comment-15202116
 ] 

Apache Spark commented on SPARK-13919:
--

User 'davies' has created a pull request for this issue:
https://github.com/apache/spark/pull/11828

> Resolving the Conflicts of ColumnPruning and PushPredicateThroughProject 
> -
>
> Key: SPARK-13919
> URL: https://issues.apache.org/jira/browse/SPARK-13919
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Xiao Li
>
> Now, {{ColumnPruning}} and {{PushPredicateThroughProject}} reverse each 
> other's effect. Although it will not cause the max iteration now, some 
> queries are not optimized to the best. 
> For example, in the following query, 
> {code}
> val input = LocalRelation('a.int, 'b.string, 'c.double, 'd.int)
> val originalQuery =
>   input.select('a, 'b, 'c, 'd,
> WindowExpression(
>   AggregateExpression(Count('b), Complete, isDistinct = false),
>   WindowSpecDefinition( 'a :: Nil,
> SortOrder('b, Ascending) :: Nil,
> UnspecifiedFrame)).as('window)).where('window > 1).select('a, 'c)
> {code}
> After multiple iteration of two rules of {{ColumnPruning}} and 
> {{PushPredicateThroughProject}}, the optimized plan we generated is like:
> {code}
> Project [a#0,c#0] 
>   
>  
> +- Filter (window#0L > cast(1 as bigint)) 
>   
>  
>+- Project [a#0,c#0,window#0L] 
>   
>  
>   +- Window [(count(b#0),mode=Complete,isDistinct=false) 
> windowspecdefinition(a#0, b#0 ASC, RANGE BETWEEN UNBOUNDED PRECEDING AND 
> CURRENT ROW) AS window#0L], [a#0], [b#0 ASC]   
>  +- LocalRelation [a#0,b#0,c#0,d#0]   
>   
>  
> {code}
> However, the expected optimized plan should be like:
> {code}
> Project [a#0,c#0]
> +- Filter (window#0L > cast(1 as bigint))
>+- Project [a#0,c#0,window#0L]
>   +- Window [(count(b#0),mode=Complete,isDistinct=false) 
> windowspecdefinition(a#0, b#0 ASC, RANGE BETWEEN UNBOUNDED PRECEDING AND 
> CURRENT ROW) AS window#0L], [a#0], [b#0 ASC]
>  +- Project [a#0,b#0,c#0]
> +- LocalRelation [a#0,b#0,c#0,d#0]
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-13919) Resolving the Conflicts of ColumnPruning and PushPredicateThroughProject

2016-03-15 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-13919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196535#comment-15196535
 ] 

Apache Spark commented on SPARK-13919:
--

User 'gatorsmile' has created a pull request for this issue:
https://github.com/apache/spark/pull/11745

> Resolving the Conflicts of ColumnPruning and PushPredicateThroughProject 
> -
>
> Key: SPARK-13919
> URL: https://issues.apache.org/jira/browse/SPARK-13919
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.0.0
>Reporter: Xiao Li
>
> Now, {{ColumnPruning}} and {{PushPredicateThroughProject}} reverse each 
> other's effect. Although it will not cause the max iteration now, some 
> queries are not optimized to the best. 
> For example, in the following query, 
> {code}
> val input = LocalRelation('a.int, 'b.string, 'c.double, 'd.int)
> val originalQuery =
>   input.select('a, 'b, 'c, 'd,
> WindowExpression(
>   AggregateExpression(Count('b), Complete, isDistinct = false),
>   WindowSpecDefinition( 'a :: Nil,
> SortOrder('b, Ascending) :: Nil,
> UnspecifiedFrame)).as('window)).where('window > 1).select('a, 'c)
> {code}
> After multiple iteration of two rules of {{ColumnPruning}} and 
> {{PushPredicateThroughProject}}, the optimized plan we generated is like:
> {code}
> Project [a#0,c#0] 
>   
>  
> +- Filter (window#0L > cast(1 as bigint)) 
>   
>  
>+- Project [a#0,c#0,window#0L] 
>   
>  
>   +- Window [(count(b#0),mode=Complete,isDistinct=false) 
> windowspecdefinition(a#0, b#0 ASC, RANGE BETWEEN UNBOUNDED PRECEDING AND 
> CURRENT ROW) AS window#0L], [a#0], [b#0 ASC]   
>  +- LocalRelation [a#0,b#0,c#0,d#0]   
>   
>  
> {code}
> However, the expected optimized plan should be like:
> {code}
> Project [a#0,c#0]
> +- Filter (window#0L > cast(1 as bigint))
>+- Project [a#0,c#0,window#0L]
>   +- Window [(count(b#0),mode=Complete,isDistinct=false) 
> windowspecdefinition(a#0, b#0 ASC, RANGE BETWEEN UNBOUNDED PRECEDING AND 
> CURRENT ROW) AS window#0L], [a#0], [b#0 ASC]
>  +- Project [a#0,b#0,c#0]
> +- LocalRelation [a#0,b#0,c#0,d#0]
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org