Ohad Raviv created SPARK-23985:

             Summary: predicate push down doesn't work with simple compound 
partition spec
                 Key: SPARK-23985
                 URL: https://issues.apache.org/jira/browse/SPARK-23985
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.4.0
            Reporter: Ohad Raviv

while predicate push down works with this query: 
select *, row_number() over (partition by a order by b) from t1 where a>1
it dowsn't work with:
select *, row_number() over (partition by concat(a,'lit') order by b) from t1 
where a>1
I added a test to FilterPushdownSuite which I think recreates the problem:
  test("Window: predicate push down -- ohad") {
    val winExpr = windowExpr(count('b),
      windowSpec(Concat('a :: Nil) :: Nil, 'b.asc :: Nil, UnspecifiedFrame))

    val originalQuery = testRelation.select('a, 'b, 'c, 
winExpr.as('window)).where('a > 1)
    val correctAnswer = testRelation
      .where('a > 1).select('a, 'b, 'c)
      .window(winExpr.as('window) :: Nil, 'a :: Nil, 'b.asc :: Nil)
      .select('a, 'b, 'c, 'window).analyze

    comparePlans(Optimize.execute(originalQuery.analyze), correctAnswer)

will try to create a PR with a correction

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to