[ 
https://issues.apache.org/jira/browse/SPARK-62081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62081:
-----------------------------------
    Labels: pull-request-available  (was: )

> Send the metadata of the fields of a type as PySpark's json.dumps writes it, 
> and raise the errors of createDataFrame of a user-defined type of no JSON in 
> PySpark's order
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62081
>                 URL: https://issues.apache.org/jira/browse/SPARK-62081
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Two documented differences of the drop-in (pyspark-client-rust) from 
> pyspark-client 4.3.0, of the metadata of the fields of a type and of 
> createDataFrame of user-defined types:
> 1. The metadata of a StructField. The core held it as serde_json values, so a 
> plan and createDataFrame of data sent the core's JSON of it: an int of more 
> than 64 bits as a float (an Arrow-optimized UDF of a return type of such 
> metadata failed with ARROW_TYPE_MISMATCH where PySpark's runs), a float that 
> is not finite raised serde_json's ValueError when the field was made, keys 
> other than str in another order, ties of floats rounded up 
> (74334700308107.625 written as ...63, where CPython's repr writes ...62), and 
> metadata that json.dumps refuses (a set, a tuple key) raised when the field 
> was made. PySpark sends json.dumps of the dict, and json() of the schema for 
> createDataFrame, also of the metadata of the fields of a UserDefinedType's 
> sqlType(), in its Arrow schema too, and raises the errors of json.dumps where 
> it writes it. The core now holds the metadata as it was written (a Metadata 
> type of the exact JSON text of each value, with Python's json writers), and 
> every place that sends a type sends PySpark's bytes (casts, UDF/UDTF return 
> types, map/apply schemas, DataFrame.to, from_json, readers, createDataFrame's 
> schema and Arrow schemas), with PySpark's errors and order of them (the 
> NOT_IMPLEMENTED of the six map/apply functions after the AnalyzePlan of the 
> columns). The pandas fast paths strip the metadata of their Arrow schema and 
> name their tables by the deduplicated names, as PySpark's pandas path does, 
> and a UDT's SQL type keeps its duplicate field names.
> 2. createDataFrame of a type of a UserDefinedType whose module() or sqlType() 
> is not implemented raised the NOT_IMPLEMENTED of its jsonValue(), where 
> PySpark's converters raise that of its sqlType() or serialize, and PySpark's 
> inference errors (CANNOT_MERGE_TYPE, CANNOT_INFER_TYPE_FOR_FIELD) come first. 
> It now raises PySpark's errors in PySpark's order, through PySpark's own 
> conversion of the data.
> The Rust API changes (StructField::metadata is a Metadata, which is not Eq, 
> Hash, Serialize or Deserialize; its accessors) are in docs/compatibility.md's 
> notes of the 4.4 line. The constructors of the types calling a UDT's methods 
> when a field of it is made is documented in dev/parity/known-differences.md.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to