Hyukjin Kwon created SPARK-61917:
------------------------------------

             Summary: Compare and render nested data types by their own 
objects, so that types of the same JSON stay distinct
                 Key: SPARK-61917
                 URL: https://issues.apache.org/jira/browse/SPARK-61917
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


The drop-in keeps a type nested in an ArrayType, a MapType or a StructType as a 
core type, and finds its Python object again through a registry keyed by the 
type's JSON. So two nested types of the same JSON are confused, where PySpark 
compares and renders the objects themselves:

- t1, t2 = AtomicType(), AtomicType(); t1.tag, t2.tag = 1, 2: ArrayType(t1) == 
ArrayType(t2) is True on the drop-in and False on pyspark-client 4.3.0, and the 
same for StructType.
- Two Python subclasses of DataType whose typeName stems are equal but whose 
simpleString differ: StructType([StructField("a", X1()), StructField("b", 
X2())]).simpleString() is struct<a:x2,b:x2> on the drop-in and 
struct<a:x1,b:x2> on PySpark, and ArrayType(X1()) == ArrayType(X2()) is True.
- The same happens for UserDefinedType subclasses with parameters: 
ArrayType(ArgUDT(2)) == ArrayType(ArgUDT(3)) is True.

Nested children should be compared and rendered through the objects that the 
containers keep, as PySpark does.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to