jin xing created SPARK-24193: -------------------------------- Summary: Sort by disk when number of limit is big in TakeOrderedAndProjectExec Key: SPARK-24193 URL: https://issues.apache.org/jira/browse/SPARK-24193 Project: Spark Issue Type: New Feature Components: SQL Affects Versions: 2.3.0 Reporter: jin xing
Physical plan of "_select colA from t order by colB limit M_" is _TakeOrderedAndProject_; Currently _TakeOrderedAndProject_ sorts data in memory, see https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala#L158 Shall we add a config -- if the number of limit (M) is too big, we can sort by disk ? Thus memory issue can be resolved. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org