Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22630#discussion_r223182235
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SortExec.scala ---
@@ -132,6 +132,13 @@ case class SortExec(
// a stop check before sorting.
override def needStopCheck: Boolean = false
+ // Sort is a blocking operator. It needs to consume all the inputs
before producing any output.
+ // This means, Limit operator after Sort will never reach its limit
during the execution of Sort's
+ // upstream operators. Here we override this method to return Nil, so
that upstream operators will
+ // not generate useless conditions (which are always evaluated to false)
for the Limit operators
+ // after Sort.
+ override def limitNotReachedChecks: Seq[String] = Nil
--- End diff --
It's only done in Sort and Aggregate currently. I don't want to overdesign
it until there are more use cases.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
