GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/20453
[SPARK-23281][SQL] Query produces results in incorrect order when a
composite order by clause refers to both original columns and aliases
## What changes were proposed in this pull request?
Here is the test snippet.
``` SQL
scala> Seq[(Integer, Integer)](
| (1, 1),
| (1, 3),
| (2, 3),
| (3, 3),
| (4, null),
| (5, null)
| ).toDF("key", "value").createOrReplaceTempView("src")
scala> sql(
| """
| |SELECT MAX(value) as value, key as col2
| |FROM src
| |GROUP BY key
| |ORDER BY value desc, key
| """.stripMargin).show
+-----+----+
|value|col2|
+-----+----+
| 3| 3|
| 3| 2|
| 3| 1|
| null| 5|
| null| 4|
+-----+----+
```SQL
Here is the explain output :
```SQL
== Parsed Logical Plan ==
'Sort ['value DESC NULLS LAST, 'key ASC NULLS FIRST], true
+- 'Aggregate ['key], ['MAX('value) AS value#9, 'key AS col2#10]
+- 'UnresolvedRelation `src`
== Analyzed Logical Plan ==
value: int, col2: int
Project [value#9, col2#10]
+- Sort [value#9 DESC NULLS LAST, col2#10 DESC NULLS LAST], true
+- Aggregate [key#5], [max(value#6) AS value#9, key#5 AS col2#10]
+- SubqueryAlias src
+- Project [_1#2 AS key#5, _2#3 AS value#6]
+- LocalRelation [_1#2, _2#3]
``` SQL
The sort direction is being wrongly changed from ASC to DSC while resolving
```Sort``` in
resolveAggregateFunctions.
## How was this patch tested?
A few tests are added in SQLQuerySuite.
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dilipbiswal/spark local_spark
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20453.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20453
----
commit 24e858c707b01982762e4695ea552ae517abb2cc
Author: Dilip Biswal <dbiswal@...>
Date: 2018-01-31T08:59:24Z
[SPARK-23281] Query produces results in incorrect order when a composite
order by clause refers to both original columns and aliases
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
