[
https://issues.apache.org/jira/browse/SPARK-62081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62081:
------------------------------------
Assignee: Hyukjin Kwon
> Send the metadata of the fields of a type as PySpark's json.dumps writes it,
> and raise the errors of createDataFrame of a user-defined type of no JSON in
> PySpark's order
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62081
> URL: https://issues.apache.org/jira/browse/SPARK-62081
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Two documented differences of the drop-in (pyspark-client-rust) from
> pyspark-client 4.3.0, of the metadata of the fields of a type and of
> createDataFrame of user-defined types:
> 1. The metadata of a StructField. The core held it as serde_json values, so a
> plan and createDataFrame of data sent the core's JSON of it: an int of more
> than 64 bits as a float (an Arrow-optimized UDF of a return type of such
> metadata failed with ARROW_TYPE_MISMATCH where PySpark's runs), a float that
> is not finite raised serde_json's ValueError when the field was made, keys
> other than str in another order, ties of floats rounded up
> (74334700308107.625 written as ...63, where CPython's repr writes ...62), and
> metadata that json.dumps refuses (a set, a tuple key) raised when the field
> was made. PySpark sends json.dumps of the dict, and json() of the schema for
> createDataFrame, also of the metadata of the fields of a UserDefinedType's
> sqlType(), in its Arrow schema too, and raises the errors of json.dumps where
> it writes it. The core now holds the metadata as it was written (a Metadata
> type of the exact JSON text of each value, with Python's json writers), and
> every place that sends a type sends PySpark's bytes (casts, UDF/UDTF return
> types, map/apply schemas, DataFrame.to, from_json, readers, createDataFrame's
> schema and Arrow schemas), with PySpark's errors and order of them (the
> NOT_IMPLEMENTED of the six map/apply functions after the AnalyzePlan of the
> columns). The pandas fast paths strip the metadata of their Arrow schema and
> name their tables by the deduplicated names, as PySpark's pandas path does,
> and a UDT's SQL type keeps its duplicate field names.
> 2. createDataFrame of a type of a UserDefinedType whose module() or sqlType()
> is not implemented raised the NOT_IMPLEMENTED of its jsonValue(), where
> PySpark's converters raise that of its sqlType() or serialize, and PySpark's
> inference errors (CANNOT_MERGE_TYPE, CANNOT_INFER_TYPE_FOR_FIELD) come first.
> It now raises PySpark's errors in PySpark's order, through PySpark's own
> conversion of the data.
> The Rust API changes (StructField::metadata is a Metadata, which is not Eq,
> Hash, Serialize or Deserialize; its accessors) are in docs/compatibility.md's
> notes of the 4.4 line. The constructors of the types calling a UDT's methods
> when a field of it is made is documented in dev/parity/known-differences.md.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]