[ 
https://issues.apache.org/jira/browse/SPARK-46349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Majid Hajiheidari updated SPARK-46349:
--------------------------------------
    Description: 
Applying a sort direction to an expression that is already a sort expression 
creates a nested SortOrder. Analysis accepts it, and the query only fails at 
execution time with an internal error that does not point at the user's code.

Scala:
{code:scala}
spark.range(10).orderBy($"id".desc.asc).show()
{code}
Python:
{code:python}
spark.range(10).orderBy(f.desc("id"), ascending=False).show()
{code}
Both fail with:
{noformat}
org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot generate code for 
expression: input[0, bigint, false] DESC NULLS LAST SQLSTATE: XX000
{noformat}
Analysis should reject a nested sort order with a clear error instead.

  was:
Hello everyone,

This is my first contribution to the project. I welcome any feedback and edits 
to improve this pull request.Currently, it's possible to create redundant sort 
expressions in both Scala and Python APIs, leading to potentially incorrect and 
confusing SQL statements. For example:

Scala:
{code:java}
spark.range(10).orderBy($"id".desc.asc).show(){code}
Python:
{code:java}
spark.range(10).orderBy(f.desc('id'), ascending=False).show(){code}
 
Such usage generates SQL like order by id DESC NULLS LAST DESC NULLS LAST, 
causing non-descriptive error messages.

 

I look forward to your feedback and thank you for considering this contribution.

        Summary: Fail analysis with a clear error for nested sort orders  (was: 
Prevent Multiple SortOrders for an Expression)

> Fail analysis with a clear error for nested sort orders
> -------------------------------------------------------
>
>                 Key: SPARK-46349
>                 URL: https://issues.apache.org/jira/browse/SPARK-46349
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, SQL
>    Affects Versions: 4.0.0
>            Reporter: Majid Hajiheidari
>            Priority: Minor
>              Labels: pull-request-available
>
> Applying a sort direction to an expression that is already a sort expression 
> creates a nested SortOrder. Analysis accepts it, and the query only fails at 
> execution time with an internal error that does not point at the user's code.
> Scala:
> {code:scala}
> spark.range(10).orderBy($"id".desc.asc).show()
> {code}
> Python:
> {code:python}
> spark.range(10).orderBy(f.desc("id"), ascending=False).show()
> {code}
> Both fail with:
> {noformat}
> org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot generate code for 
> expression: input[0, bigint, false] DESC NULLS LAST SQLSTATE: XX000
> {noformat}
> Analysis should reject a nested sort order with a clear error instead.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to