maytasm commented on PR #37330:
URL: https://github.com/apache/spark/pull/37330#issuecomment-1867341537
@ulysses-you
Sorry for the late reply. That is true unless the following condition is
True for the `Sort global`:
` case s @ Sort(orders, _, child) if orders.isEmpty ||
orders.exists(_.child.foldable) =>
val newOrders = orders.filterNot(_.child.foldable)
if (newOrders.isEmpty) {
applyLocally.lift(child).getOrElse(child)
}
`
In the case that the above condition is True, then this `Sort global` will
be removed. However, with this change (in this PR), the `Sort global` will not
be removed as it would have been turn into a RepartitionByExpression
Consider the following case:
`
+- Sort [date#0 ASC NULLS FIRST], false
+- Project [...]
+- Sort [20231221 ASC NULLS FIRST], true
+- Aggregate [...]
+- RelationV2[...] some_table
`
with the change from this PR, the `Sort [20231221 ASC NULLS FIRST], true`
would be remove. We would end up with
`
+- Sort [date#0 ASC NULLS FIRST], false
+- Project [...]
+- Aggregate [...]
+- RelationV2[...] some_table
`
However, with the change in this PR:
`
+- Sort [date#0 ASC NULLS FIRST], false
+- Project [...]
+- Exchange rangepartition (20231221 ASC NULL FIRST)
+- Aggregate [...]
+- RelationV2[...] some_table
`
The problem here is that `Exchange rangepartition` reduces the partition to
1. Then the stages after that like the Sort on date#0 ran with a single task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
