Henry Robinson created SPARK-23973: -------------------------------------- Summary: Remove consecutive sorts Key: SPARK-23973 URL: https://issues.apache.org/jira/browse/SPARK-23973 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.4.0 Reporter: Henry Robinson
As a follow-on from SPARK-23375, it would be easy to remove redundant sorts in the following kind of query: {code} Seq((1), (3)).toDF("int").orderBy('int.asc).orderBy('int.desc).explain() == Physical Plan == *(2) Sort [int#35 DESC NULLS LAST], true, 0 +- Exchange rangepartitioning(int#35 DESC NULLS LAST, 200) +- *(1) Sort [int#35 ASC NULLS FIRST], true, 0 +- Exchange rangepartitioning(int#35 ASC NULLS FIRST, 200) +- LocalTableScan [int#35] {code} There's no need to perform {{(1) Sort}}. Since the sort operator isn't stable, AFAIK, it should be ok to remove a sort on any column that gets 'overwritten' by a subsequent one in this way. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org