Hyukjin Kwon created SPARK-61960:
------------------------------------
Summary: Make the concrete data types subclassable as PySpark's,
and compare the metadata of fields as Python compares the dicts
Key: SPARK-61960
URL: https://issues.apache.org/jira/browse/SPARK-61960
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
In pyspark-client 4.3.0, the data types of pyspark.sql.types are plain Python
classes, so they can be subclassed. For example, `class MyString(StringType):
...` gives a type whose typeName, simpleString and JSON come from the
subclass's name, whose repr and equality follow PySpark's (DataType.__eq__ is
isinstance(other, type(self)) and the __dict__), and which pickles and copies
as its class. The drop-in's concrete types (NullType ... GeographyType, 27
classes) were PyO3 classes without subclass support, so `type("X",
(StringType,), {})` raised "type 'StringType' is not an acceptable base type".
`class X(IntegralType)` raised "cannot create".
Also (a round-3 finding on SPARK-61912): StructField metadata was compared as
JSON values, where PySpark compares the dicts with Python's ==. So the drop-in
called equal:
- {"k": (1, 2)} and {"k": [1, 2]};
- {1: "a"} and {"1": "a"};
- 10**30 and 10**30 + 1.
json() also wrote 1e+30 where PySpark writes the exact int.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]