[ 
https://issues.apache.org/jira/browse/SPARK-61917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61917:
-----------------------------------
    Labels: pull-request-available  (was: )

> Compare and render nested data types by their own objects, so that types of 
> the same JSON stay distinct
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61917
>                 URL: https://issues.apache.org/jira/browse/SPARK-61917
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> The drop-in keeps a type nested in an ArrayType, a MapType or a StructType as 
> a core type, and finds its Python object again through a registry keyed by 
> the type's JSON. So two nested types of the same JSON are confused, where 
> PySpark compares and renders the objects themselves:
> - t1, t2 = AtomicType(), AtomicType(); t1.tag, t2.tag = 1, 2: ArrayType(t1) 
> == ArrayType(t2) is True on the drop-in and False on pyspark-client 4.3.0, 
> and the same for StructType.
> - Two Python subclasses of DataType whose typeName stems are equal but whose 
> simpleString differ: StructType([StructField("a", X1()), StructField("b", 
> X2())]).simpleString() is struct<a:x2,b:x2> on the drop-in and 
> struct<a:x1,b:x2> on PySpark, and ArrayType(X1()) == ArrayType(X2()) is True.
> - The same happens for UserDefinedType subclasses with parameters: 
> ArrayType(ArgUDT(2)) == ArrayType(ArgUDT(3)) is True.
> Nested children should be compared and rendered through the objects that the 
> containers keep, as PySpark does.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to