GitHub user henryr opened a pull request:
https://github.com/apache/spark/pull/21049
[SPARK-23957][SQL] Remove redundant sort operators from subqueries
## What changes were proposed in this pull request?
Subqueries (at least in SQL) have 'bag of tuples' semantics. Ordering
them is therefore redundant (unless combined with a limit). This patch
adds a new optimizer rule that removes sort operators that are directly
below subqueries (or some combination of projection and filtering below
a subquery).
## How was this patch tested?
New unit tests. All sql unit tests pass.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/henryr/spark spark-23957
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21049.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21049
----
commit bb992c2058863322a9183b2985806a87729e4168
Author: Henry Robinson <henry@...>
Date: 2018-04-12T03:44:36Z
[SPARK-23957][SQL] Remove redundant sort operators from subqueries
## What changes were proposed in this pull request?
Subqueries (at least in SQL) have 'bag of tuples' semantics. Ordering
them is therefore redundant (unless combined with a limit). This patch
adds a new optimizer rule that removes sort operators that are directly
below subqueries (or some combination of projection and filtering below
a subquery).
## How was this patch tested?
New unit tests. All sql unit tests pass.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
