[
https://issues.apache.org/jira/browse/SPARK-61960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61960:
------------------------------------
Assignee: Hyukjin Kwon
> Make the concrete data types subclassable as PySpark's, and compare the
> metadata of fields as Python compares the dicts
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61960
> URL: https://issues.apache.org/jira/browse/SPARK-61960
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> In pyspark-client 4.3.0, the data types of pyspark.sql.types are plain Python
> classes, so they can be subclassed. For example, `class MyString(StringType):
> ...` gives a type whose typeName, simpleString and JSON come from the
> subclass's name, whose repr and equality follow PySpark's (DataType.__eq__ is
> isinstance(other, type(self)) and the __dict__), and which pickles and copies
> as its class. The drop-in's concrete types (NullType ... GeographyType, 27
> classes) were PyO3 classes without subclass support, so `type("X",
> (StringType,), {})` raised "type 'StringType' is not an acceptable base
> type". `class X(IntegralType)` raised "cannot create".
> Also (a round-3 finding on SPARK-61912): StructField metadata was compared as
> JSON values, where PySpark compares the dicts with Python's ==. So the
> drop-in called equal:
> - {"k": (1, 2)} and {"k": [1, 2]};
> - {1: "a"} and {"1": "a"};
> - 10**30 and 10**30 + 1.
> json() also wrote 1e+30 where PySpark writes the exact int.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]