[ 
https://issues.apache.org/jira/browse/SPARK-61960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61960:
------------------------------------

    Assignee: Hyukjin Kwon

> Make the concrete data types subclassable as PySpark's, and compare the 
> metadata of fields as Python compares the dicts
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61960
>                 URL: https://issues.apache.org/jira/browse/SPARK-61960
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> In pyspark-client 4.3.0, the data types of pyspark.sql.types are plain Python 
> classes, so they can be subclassed. For example, `class MyString(StringType): 
> ...` gives a type whose typeName, simpleString and JSON come from the 
> subclass's name, whose repr and equality follow PySpark's (DataType.__eq__ is 
> isinstance(other, type(self)) and the __dict__), and which pickles and copies 
> as its class. The drop-in's concrete types (NullType ... GeographyType, 27 
> classes) were PyO3 classes without subclass support, so `type("X", 
> (StringType,), {})` raised "type 'StringType' is not an acceptable base 
> type". `class X(IntegralType)` raised "cannot create".
> Also (a round-3 finding on SPARK-61912): StructField metadata was compared as 
> JSON values, where PySpark compares the dicts with Python's ==. So the 
> drop-in called equal:
> - {"k": (1, 2)} and {"k": [1, 2]};
> - {1: "a"} and {"1": "a"};
> - 10**30 and 10**30 + 1.
> json() also wrote 1e+30 where PySpark writes the exact int.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to