[
https://issues.apache.org/jira/browse/SPARK-46349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146387#comment-18146387
]
Majid Hajiheidari commented on SPARK-46349:
-------------------------------------------
My original PR #44283 was closed as stale and can't be reopened, so the fix is
now in https://github.com/apache/spark/pull/60520.
Rather than rejecting nested sort orders in the SortOrder constructor, it adds
a NESTED_SORT_ORDER analysis error, checked in both CheckAnalysis and the
single-pass resolver. That covers the Scala and Python APIs, both classic and
Spark Connect.
> Fail analysis with a clear error for nested sort orders
> -------------------------------------------------------
>
> Key: SPARK-46349
> URL: https://issues.apache.org/jira/browse/SPARK-46349
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 4.0.0
> Reporter: Majid Hajiheidari
> Priority: Minor
> Labels: pull-request-available
>
> Applying a sort direction to an expression that is already a sort expression
> creates a nested SortOrder. Analysis accepts it, and the query only fails at
> execution time with an internal error that does not point at the user's code.
> Scala:
> {code:scala}
> spark.range(10).orderBy($"id".desc.asc).show()
> {code}
> Python:
> {code:python}
> spark.range(10).orderBy(f.desc("id"), ascending=False).show()
> {code}
> Both fail with:
> {noformat}
> org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot generate code for
> expression: input[0, bigint, false] DESC NULLS LAST SQLSTATE: XX000
> {noformat}
> Analysis should reject a nested sort order with a clear error instead.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]