Hyukjin Kwon created SPARK-62081:
------------------------------------

             Summary: Send the metadata of the fields of a type as PySpark's 
json.dumps writes it, and raise the errors of createDataFrame of a user-defined 
type of no JSON in PySpark's order
                 Key: SPARK-62081
                 URL: https://issues.apache.org/jira/browse/SPARK-62081
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Two documented differences of the drop-in (pyspark-client-rust) from 
pyspark-client 4.3.0, of the metadata of the fields of a type and of 
createDataFrame of user-defined types:

1. The metadata of a StructField. The core held it as serde_json values, so a 
plan and createDataFrame of data sent the core's JSON of it: an int of more 
than 64 bits as a float (an Arrow-optimized UDF of a return type of such 
metadata failed with ARROW_TYPE_MISMATCH where PySpark's runs), a float that is 
not finite raised serde_json's ValueError when the field was made, keys other 
than str in another order, ties of floats rounded up (74334700308107.625 
written as ...63, where CPython's repr writes ...62), and metadata that 
json.dumps refuses (a set, a tuple key) raised when the field was made. PySpark 
sends json.dumps of the dict, and json() of the schema for createDataFrame, 
also of the metadata of the fields of a UserDefinedType's sqlType(), in its 
Arrow schema too, and raises the errors of json.dumps where it writes it. The 
core now holds the metadata as it was written (a Metadata type of the exact 
JSON text of each value, with Python's json writers), and every place that 
sends a type sends PySpark's bytes (casts, UDF/UDTF return types, map/apply 
schemas, DataFrame.to, from_json, readers, createDataFrame's schema and Arrow 
schemas), with PySpark's errors and order of them (the NOT_IMPLEMENTED of the 
six map/apply functions after the AnalyzePlan of the columns). The pandas fast 
paths strip the metadata of their Arrow schema and name their tables by the 
deduplicated names, as PySpark's pandas path does, and a UDT's SQL type keeps 
its duplicate field names.

2. createDataFrame of a type of a UserDefinedType whose module() or sqlType() 
is not implemented raised the NOT_IMPLEMENTED of its jsonValue(), where 
PySpark's converters raise that of its sqlType() or serialize, and PySpark's 
inference errors (CANNOT_MERGE_TYPE, CANNOT_INFER_TYPE_FOR_FIELD) come first. 
It now raises PySpark's errors in PySpark's order, through PySpark's own 
conversion of the data.

The Rust API changes (StructField::metadata is a Metadata, which is not Eq, 
Hash, Serialize or Deserialize; its accessors) are in docs/compatibility.md's 
notes of the 4.4 line. The constructors of the types calling a UDT's methods 
when a field of it is made is documented in dev/parity/known-differences.md.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to