[
https://issues.apache.org/jira/browse/SPARK-23985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16625876#comment-16625876
]
Yuming Wang edited comment on SPARK-23985 at 9/26/18 1:54 AM:
--------------------------------------------------------------
[~uzadude] Seem we should not push down predicate. Pelase see these test case:
[https://github.com/apache/spark/blob/2c73d2a948bdde798aaf0f87c18846281deb05fd/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala#L1086-L1144]
Here is an example:
{code:scala}
spark.range(10).selectExpr("cast(id % 5 as string) as a", "id as
b").write.saveAsTable("t1")
val w1 = spark.sql(
"select * from (select *, row_number() over (partition by alit order by b) as
rn from " +
"(select *, a % 4 as alit from t1) x) y where a>2 order by a")
w1.show
val w2 = spark.sql(
"select * from (select *, row_number() over (partition by alit order by b) as
rn from " +
"(select *, a % 4 as alit from t1 where a> 2) x) y order by a")
w2.show
{code}
output:
{noformat}
+---+---+----+---+
| a| b|alit| rn|
+---+---+----+---+
| 3| 3| 3.0| 1|
| 3| 8| 3.0| 2|
| 4| 4| 0.0| 2|
| 4| 9| 0.0| 4|
+---+---+----+---+
+---+---+----+---+
| a| b|alit| rn|
+---+---+----+---+
| 3| 3| 3.0| 1|
| 3| 8| 3.0| 2|
| 4| 4| 0.0| 1|
| 4| 9| 0.0| 2|
+---+---+----+---+
{noformat}
was (Author: q79969786):
[~uzadude] Seem we should not push down predicate. Pelase see these test case:
https://github.com/apache/spark/blob/2c73d2a948bdde798aaf0f87c18846281deb05fd/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala#L1086-L1144
> predicate push down doesn't work with simple compound partition spec
> --------------------------------------------------------------------
>
> Key: SPARK-23985
> URL: https://issues.apache.org/jira/browse/SPARK-23985
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Ohad Raviv
> Priority: Minor
>
> while predicate push down works with this query:
> {code:sql}
> select * from (
> select *, row_number() over (partition by a order by b) from t1
> )z
> where a>1
> {code}
> it dowsn't work with:
> {code:sql}
> select * from (
> select *, row_number() over (partition by concat(a,'lit') order by b) from
> t1
> )z
> where a>1
> {code}
>
> I added a test to FilterPushdownSuite which I think recreates the problem:
> {code}
> test("Window: predicate push down -- ohad") {
> val winExpr = windowExpr(count('b),
> windowSpec(Concat('a :: Nil) :: Nil, 'b.asc :: Nil, UnspecifiedFrame))
> val originalQuery = testRelation.select('a, 'b, 'c,
> winExpr.as('window)).where('a > 1)
> val correctAnswer = testRelation
> .where('a > 1).select('a, 'b, 'c)
> .window(winExpr.as('window) :: Nil, 'a :: Nil, 'b.asc :: Nil)
> .select('a, 'b, 'c, 'window).analyze
> comparePlans(Optimize.execute(originalQuery.analyze), correctAnswer)
> }
> {code}
> will try to create a PR with a correction
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
