Hyukjin Kwon created SPARK-61960:
------------------------------------

             Summary: Make the concrete data types subclassable as PySpark's, 
and compare the metadata of fields as Python compares the dicts
                 Key: SPARK-61960
                 URL: https://issues.apache.org/jira/browse/SPARK-61960
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


In pyspark-client 4.3.0, the data types of pyspark.sql.types are plain Python 
classes, so they can be subclassed. For example, `class MyString(StringType): 
...` gives a type whose typeName, simpleString and JSON come from the 
subclass's name, whose repr and equality follow PySpark's (DataType.__eq__ is 
isinstance(other, type(self)) and the __dict__), and which pickles and copies 
as its class. The drop-in's concrete types (NullType ... GeographyType, 27 
classes) were PyO3 classes without subclass support, so `type("X", 
(StringType,), {})` raised "type 'StringType' is not an acceptable base type". 
`class X(IntegralType)` raised "cannot create".

Also (a round-3 finding on SPARK-61912): StructField metadata was compared as 
JSON values, where PySpark compares the dicts with Python's ==. So the drop-in 
called equal:
- {"k": (1, 2)} and {"k": [1, 2]};
- {1: "a"} and {"1": "a"};
- 10**30 and 10**30 + 1.

json() also wrote 1e+30 where PySpark writes the exact int.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to