allisonwang-db commented on a change in pull request #30093:
URL: https://github.com/apache/spark/pull/30093#discussion_r509898482
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1052,11 +1052,11 @@ object CombineFilters extends Rule[LogicalPlan] with
PredicateHelper {
* function is order irrelevant
*/
object EliminateSorts extends Rule[LogicalPlan] {
- def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
Review comment:
Yes. This case is tricky. Let's consider this example:
`df.sortBy('a.asc).sortBy('a.asc).sortBy('a.asc)`
```
Sort(a, false, _) // A
Sort(a, false, _) // B
Sort(a, false, _) // C
...
```
If we don't use `transformUp`here, the rule will keep both Sort B and C
since it will not apply the rule to A's child (B) again. Plan after running
this rule will be:
```
Sort(a, false, _) // B
Sort(a, false, _) // C
...
```
And the `Once` strategy's idempotency will be broken since Sort B will be
removed if we run this rule again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
