[ 
https://issues.apache.org/jira/browse/SPARK-61758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61758:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Fix the PySpark differences that the review of the last 200 commits found in 
> the types, Columns, windows, table arguments, UDF arguments and copies of the 
> Python client, and the gaps of its CI
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61758
>                 URL: https://issues.apache.org/jira/browse/SPARK-61758
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> The review of the last 200 commits of spark-connect-rust found these 
> differences from PySpark 4.3.0 in the Python client (outside ML), and gaps of 
> its CI. This fixes them:
> - JSON type strings are parsed as PySpark's prefix regexes parse them; null 
> and calendarinterval are rejected as PySpark rejects them; 
> CalendarIntervalType pickles as "interval"; the hash of a DataType is 
> hash(str(self)), as PySpark's.
> - The schema of a server UDT keeps its sqlType (df.schema.json(), 
> createDataFrame([], df.schema)).
> - No origin on the CASE WHEN `when`; the origins of between, getItem, asc and 
> desc are those of PySpark's dunders; alias and name take bytes and give 
> protobuf's TypeError.
> - TableArg checks its state before its columns, and its copies are PySpark's; 
> the literal arguments of a UDF raise PySpark's AssertionError; the UDTF 
> `name` that PySpark does not have is removed.
> - The copies of Column, GroupedData, Catalog and Window keep their `__dict__`.
> - The runtime abort is documented and tested; the core integration test runs 
> in the connect-e2e job.
> - CI guards that every e2e file runs, and the path filters of the docs and 
> wheel jobs.
> - Docs: deep plans, call-time conversions, the pickle of a TableArg, the 
> order of getMessageParameters, char/varchar.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to