Hyukjin Kwon created SPARK-62081:
------------------------------------
Summary: Send the metadata of the fields of a type as PySpark's
json.dumps writes it, and raise the errors of createDataFrame of a user-defined
type of no JSON in PySpark's order
Key: SPARK-62081
URL: https://issues.apache.org/jira/browse/SPARK-62081
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Two documented differences of the drop-in (pyspark-client-rust) from
pyspark-client 4.3.0, of the metadata of the fields of a type and of
createDataFrame of user-defined types:
1. The metadata of a StructField. The core held it as serde_json values, so a
plan and createDataFrame of data sent the core's JSON of it: an int of more
than 64 bits as a float (an Arrow-optimized UDF of a return type of such
metadata failed with ARROW_TYPE_MISMATCH where PySpark's runs), a float that is
not finite raised serde_json's ValueError when the field was made, keys other
than str in another order, ties of floats rounded up (74334700308107.625
written as ...63, where CPython's repr writes ...62), and metadata that
json.dumps refuses (a set, a tuple key) raised when the field was made. PySpark
sends json.dumps of the dict, and json() of the schema for createDataFrame,
also of the metadata of the fields of a UserDefinedType's sqlType(), in its
Arrow schema too, and raises the errors of json.dumps where it writes it. The
core now holds the metadata as it was written (a Metadata type of the exact
JSON text of each value, with Python's json writers), and every place that
sends a type sends PySpark's bytes (casts, UDF/UDTF return types, map/apply
schemas, DataFrame.to, from_json, readers, createDataFrame's schema and Arrow
schemas), with PySpark's errors and order of them (the NOT_IMPLEMENTED of the
six map/apply functions after the AnalyzePlan of the columns). The pandas fast
paths strip the metadata of their Arrow schema and name their tables by the
deduplicated names, as PySpark's pandas path does, and a UDT's SQL type keeps
its duplicate field names.
2. createDataFrame of a type of a UserDefinedType whose module() or sqlType()
is not implemented raised the NOT_IMPLEMENTED of its jsonValue(), where
PySpark's converters raise that of its sqlType() or serialize, and PySpark's
inference errors (CANNOT_MERGE_TYPE, CANNOT_INFER_TYPE_FOR_FIELD) come first.
It now raises PySpark's errors in PySpark's order, through PySpark's own
conversion of the data.
The Rust API changes (StructField::metadata is a Metadata, which is not Eq,
Hash, Serialize or Deserialize; its accessors) are in docs/compatibility.md's
notes of the 4.4 line. The constructors of the types calling a UDT's methods
when a field of it is made is documented in dev/parity/known-differences.md.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]