[
https://issues.apache.org/jira/browse/SPARK-46349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Majid Hajiheidari updated SPARK-46349:
--------------------------------------
Description:
Applying a sort direction to an expression that is already a sort expression
creates a nested SortOrder. Analysis accepts it, and the query only fails at
execution time with an internal error that does not point at the user's code.
Scala:
{code:scala}
spark.range(10).orderBy($"id".desc.asc).show()
{code}
Python:
{code:python}
spark.range(10).orderBy(f.desc("id"), ascending=False).show()
{code}
Both fail with:
{noformat}
org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot generate code for
expression: input[0, bigint, false] DESC NULLS LAST SQLSTATE: XX000
{noformat}
Analysis should reject a nested sort order with a clear error instead.
was:
Hello everyone,
This is my first contribution to the project. I welcome any feedback and edits
to improve this pull request.Currently, it's possible to create redundant sort
expressions in both Scala and Python APIs, leading to potentially incorrect and
confusing SQL statements. For example:
Scala:
{code:java}
spark.range(10).orderBy($"id".desc.asc).show(){code}
Python:
{code:java}
spark.range(10).orderBy(f.desc('id'), ascending=False).show(){code}
Such usage generates SQL like order by id DESC NULLS LAST DESC NULLS LAST,
causing non-descriptive error messages.
I look forward to your feedback and thank you for considering this contribution.
Summary: Fail analysis with a clear error for nested sort orders (was:
Prevent Multiple SortOrders for an Expression)
> Fail analysis with a clear error for nested sort orders
> -------------------------------------------------------
>
> Key: SPARK-46349
> URL: https://issues.apache.org/jira/browse/SPARK-46349
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 4.0.0
> Reporter: Majid Hajiheidari
> Priority: Minor
> Labels: pull-request-available
>
> Applying a sort direction to an expression that is already a sort expression
> creates a nested SortOrder. Analysis accepts it, and the query only fails at
> execution time with an internal error that does not point at the user's code.
> Scala:
> {code:scala}
> spark.range(10).orderBy($"id".desc.asc).show()
> {code}
> Python:
> {code:python}
> spark.range(10).orderBy(f.desc("id"), ascending=False).show()
> {code}
> Both fail with:
> {noformat}
> org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot generate code for
> expression: input[0, bigint, false] DESC NULLS LAST SQLSTATE: XX000
> {noformat}
> Analysis should reject a nested sort order with a clear error instead.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]