wangyum commented on a change in pull request #32475:
URL: https://github.com/apache/spark/pull/32475#discussion_r634390447
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/LimitPushDownThroughWindow.scala
##########
@@ -42,17 +42,19 @@ object LimitPushDownThroughWindow extends Rule[LogicalPlan]
{
_.containsAllPatterns(WINDOW, LIMIT), ruleId) {
// Adding an extra Limit below WINDOW when the partitionSpec of all window
functions is empty.
case LocalLimit(limitExpr @ IntegerLiteral(limit),
- window @ Window(windowExpressions, Nil, orderSpec, child))
+ window @ Window(windowExpressions, partitionSpec, orderSpec, child))
if supportsPushdownThroughWindow(windowExpressions) &&
child.maxRows.forall(_ > limit) &&
limit < conf.topKSortFallbackThreshold =>
// Sort is needed here because we need global sort.
- window.copy(child = Limit(limitExpr, Sort(orderSpec, true, child)))
+ window.copy(child = Limit(limitExpr,
+ Sort(partitionSpec.map(SortOrder(_, Ascending)) ++ orderSpec, true,
child)))
Review comment:
@cloud-fan Do we need to changed the sort order from `NULLS FIRST` to
`NULLS LAST`? Impala have changed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
