Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21252
Instead of touching inside of `TakeOrderedAndProjectExec`, how about we
don't replace `Sort` + `Limit` with `TakeOrderedAndProjectExec` when reaching
the threshold? A.k.a:
```scala
object SpecialLimits extends Strategy {
override def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
case ReturnAnswer(rootPlan) => rootPlan match {
case Limit(IntegerLiteral(limit), Sort(order, true, child)) if
limit < conf.sortInMemForLimitThreshold =>
TakeOrderedAndProjectExec(limit, order, child.output,
planLater(child)) :: Nil
...
}
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
