[
https://issues.apache.org/jira/browse/SPARK-61199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61199:
-----------------------------------
Labels: pull-request-available (was: )
> Keep the sort order of a pickled sort Column, so that orderBy, Window.orderBy
> and repartitionByRange sort by it, in the Python client as PySpark does
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61199
> URL: https://issues.apache.org/jira/browse/SPARK-61199
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> A Column pickles as its expression (SPARK-61167). For a sort order
> (Column.asc(), desc(), asc_nulls_last() and so on) the unpickled Column is an
> opaque expression, not a sort order:
> - On a Python worker of the server, such as in a foreachBatch function that
> captures F.col("a").desc(), PySpark's DataFrame.sort and orderBy wrap the
> unpickled Column in asc(), so the server gets a sort order of a sort order
> and fails with INTERNAL_ERROR "Cannot evaluate expression: input[0, int,
> true] DESC NULLS LAST".
> - On the client, Window.orderBy and repartitionByRange of an unpickled sort
> Column also wrap it in ASC NULLS FIRST:
> Window.orderBy(pickle.loads(pickle.dumps(F.col("a").desc()))) renders as
> WindowSpec(OrderBy(a DESC NULLS LAST ASC NULLS FIRST)), and a window over a
> partition of two or more rows fails on the server the same way.
> PySpark Connect's SortOrder pickles as its child and its direction, so the
> unpickled Column is a sort order again. The fix pickles a sort order as its
> child Column and the sort method, which gives back the sort order on the
> client and on the worker.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
