Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19393#discussion_r142311845
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -296,13 +296,20 @@ object LimitPushDown extends Rule[LogicalPlan] {
}
}
- private def maybePushLimit(limitExp: Expression, plan: LogicalPlan):
LogicalPlan = {
- (limitExp, plan.maxRows) match {
- case (IntegerLiteral(maxRow), Some(childMaxRows)) if maxRow <
childMaxRows =>
+ private def maybePushLocalLimit(limitExp: Expression, plan:
LogicalPlan): LogicalPlan = {
+ (limitExp, plan.maxRowsPerPartition) match {
+ case (IntegerLiteral(newLimit), Some(childMaxRows)) if newLimit <
childMaxRows =>
+ // If the child has a cap on max rows per partition and the cap is
smaller than
+ // the new limit, put a new LocalLimit there.
--- End diff --
I think it is `the cap is larger than the new limit`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
