zhengruifeng commented on a change in pull request #34504:
URL: https://github.com/apache/spark/pull/34504#discussion_r755892984
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1548,6 +1548,31 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
filter
}
+ case filter @ Filter(condition, w: Window) if w.partitionSpec.isEmpty =>
+ w.windowExpressions match {
+ case Seq(alias @ Alias(WindowExpression(_: RowNumber,
WindowSpecDefinition(Nil, orderSpec,
+ SpecifiedWindowFrame(RowFrame, UnboundedPreceding, CurrentRow))),
_)) =>
Review comment:
I am afraid we can not relax restrictions here, suppose this case:
```
val df1 = spark.range(0, 10000, 1, 9).select(when('id < 2500, 2499).when('id
>= 7500, 10000).otherwise('id).as("key1"), 'id as "value1").withColumn("hash1",
abs(hash(col("key1"))).mod(100))
val df2 = df1.withColumn("rank",
row_number().over(Window.orderBy("value1"))).withColumn("lag3", lag("value1",
3).over(Window.orderBy("value1"))).withColumn("count",
count("value1").over(Window.orderBy("value1").rowsBetween(-10,
10))).where(col("rank") <= 1)
println(df2.queryExecution.optimizedPlan)
Filter (rank#176 <= 1)
+- Window [row_number() windowspecdefinition(value1#168L ASC NULLS FIRST,
specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS
rank#176, lag(value1#168L, -3, null) windowspecdefinition(value1#168L ASC NULLS
FIRST, specifiedwindowframe(RowFrame, -3, -3)) AS lag3#181L, count(1)
windowspecdefinition(value1#168L ASC NULLS FIRST,
specifiedwindowframe(RowFrame, -10, 10)) AS count#189L], [value1#168L ASC NULLS
FIRST]
+- Project [CASE WHEN (id#165L < 2500) THEN 2499 WHEN (id#165L >= 7500)
THEN 10000 ELSE id#165L END AS key1#167L, id#165L AS value1#168L,
(abs(hash(CASE WHEN (id#165L < 2500) THEN 2499 WHEN (id#165L >= 7500) THEN
10000 ELSE id#165L END, 42), false) % 100) AS hash1#171]
+- Range (0, 10000, step=1, splits=Some(9))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
