[
https://issues.apache.org/jira/browse/SPARK-54833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lixiaolong04 updated SPARK-54833:
---------------------------------
Description:
{{MODE() WITHIN GROUP (ORDER BY ...)}} executes correctly for both {{ASC}} and
{{{}DESC{}}}, but the SQL string representation displays an incorrect or
missing sort direction.
This causes a mismatch between the executed semantics and the displayed SQL
In {{{}Mode.scala{}}}, the SQL string representation is generated based on
{{{}reverseOpt{}}}:
[https://github.com/apache/spark/blob/e316d28be78d23b03f8615bda3919d2d212aeb17/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Mode.scala#L173-L180]
{code:java}
override def sql(isDistinct: Boolean): String = {
reverseOpt.map {
reverse =>
if (reverse) {
s"$prettyName() WITHIN GROUP (ORDER BY ${child.sql} DESC)"
} else {
s"$prettyName() WITHIN GROUP (ORDER BY ${child.sql})"
}
}.getOrElse(super.sql(isDistinct))
}{code}
However, {{reverseOpt}} is derived from {{withOrderingWithinGroup}} and does
not directly reflect the original {{ORDER BY}} direction. As a result, the SQL
string can display a reversed or missing sort direction, even though the
execution semantics are correct.
*Proposed Fix*
Swap the {{ASC}} / {{DESC}} handling in {{sql()}} so that the displayed ORDER
BY direction matches the actual execution semantics
was:
{{MODE() WITHIN GROUP (ORDER BY ...)}} executes correctly for both {{ASC}} and
{{{}DESC{}}}, but the SQL string representation displays an incorrect or
missing sort direction.
This causes a mismatch between the executed semantics and the displayed SQL
In {{{}Mode.scala{}}}, the SQL string representation is generated based on
{{{}reverseOpt{}}}:
[https://github.com/apache/spark/blob/e316d28be78d23b03f8615bda3919d2d212aeb17/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Mode.scala#L173-L180]
{code:java}
override def sql(isDistinct: Boolean): String = {
reverseOpt.map {
reverse =>
if (reverse) {
s"$prettyName() WITHIN GROUP (ORDER BY ${child.sql} DESC)"
} else {
s"$prettyName() WITHIN GROUP (ORDER BY ${child.sql})"
}
}.getOrElse(super.sql(isDistinct))
}{code}
However, {{reverseOpt}} is derived from {{withOrderingWithinGroup}} and does
not directly reflect the original {{ORDER BY}} direction. As a result, the SQL
string can display a reversed or missing sort direction, even though the
execution semantics are correct.
> MODE() WITHIN GROUP ORDER BY displays incorrect sort direction in SQL string
> ----------------------------------------------------------------------------
>
> Key: SPARK-54833
> URL: https://issues.apache.org/jira/browse/SPARK-54833
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0, 4.2.0
> Reporter: lixiaolong04
> Priority: Minor
>
> {{MODE() WITHIN GROUP (ORDER BY ...)}} executes correctly for both {{ASC}}
> and {{{}DESC{}}}, but the SQL string representation displays an incorrect or
> missing sort direction.
> This causes a mismatch between the executed semantics and the displayed SQL
> In {{{}Mode.scala{}}}, the SQL string representation is generated based on
> {{{}reverseOpt{}}}:
> [https://github.com/apache/spark/blob/e316d28be78d23b03f8615bda3919d2d212aeb17/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Mode.scala#L173-L180]
> {code:java}
> override def sql(isDistinct: Boolean): String = {
> reverseOpt.map {
> reverse =>
> if (reverse) {
> s"$prettyName() WITHIN GROUP (ORDER BY ${child.sql} DESC)"
> } else {
> s"$prettyName() WITHIN GROUP (ORDER BY ${child.sql})"
> }
> }.getOrElse(super.sql(isDistinct))
> }{code}
>
> However, {{reverseOpt}} is derived from {{withOrderingWithinGroup}} and does
> not directly reflect the original {{ORDER BY}} direction. As a result, the
> SQL string can display a reversed or missing sort direction, even though the
> execution semantics are correct.
> *Proposed Fix*
> Swap the {{ASC}} / {{DESC}} handling in {{sql()}} so that the displayed ORDER
> BY direction matches the actual execution semantics
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
