Hyukjin Kwon created SPARK-61199:
------------------------------------
Summary: Keep the sort order of a pickled sort Column, so that
orderBy, Window.orderBy and repartitionByRange sort by it, in the Python client
as PySpark does
Key: SPARK-61199
URL: https://issues.apache.org/jira/browse/SPARK-61199
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A Column pickles as its expression (SPARK-61167). For a sort order
(Column.asc(), desc(), asc_nulls_last() and so on) the unpickled Column is an
opaque expression, not a sort order:
- On a Python worker of the server, such as in a foreachBatch function that
captures F.col("a").desc(), PySpark's DataFrame.sort and orderBy wrap the
unpickled Column in asc(), so the server gets a sort order of a sort order and
fails with INTERNAL_ERROR "Cannot evaluate expression: input[0, int, true] DESC
NULLS LAST".
- On the client, Window.orderBy and repartitionByRange of an unpickled sort
Column also wrap it in ASC NULLS FIRST:
Window.orderBy(pickle.loads(pickle.dumps(F.col("a").desc()))) renders as
WindowSpec(OrderBy(a DESC NULLS LAST ASC NULLS FIRST)), and a window over a
partition of two or more rows fails on the server the same way.
PySpark Connect's SortOrder pickles as its child and its direction, so the
unpickled Column is a sort order again. The fix pickles a sort order as its
child Column and the sort method, which gives back the sort order on the client
and on the worker.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
