[
https://issues.apache.org/jira/browse/SPARK-61917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61917:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Compare and render nested data types by their own objects, so that types of
> the same JSON stay distinct
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61917
> URL: https://issues.apache.org/jira/browse/SPARK-61917
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> The drop-in keeps a type nested in an ArrayType, a MapType or a StructType as
> a core type, and finds its Python object again through a registry keyed by
> the type's JSON. So two nested types of the same JSON are confused, where
> PySpark compares and renders the objects themselves:
> - t1, t2 = AtomicType(), AtomicType(); t1.tag, t2.tag = 1, 2: ArrayType(t1)
> == ArrayType(t2) is True on the drop-in and False on pyspark-client 4.3.0,
> and the same for StructType.
> - Two Python subclasses of DataType whose typeName stems are equal but whose
> simpleString differ: StructType([StructField("a", X1()), StructField("b",
> X2())]).simpleString() is struct<a:x2,b:x2> on the drop-in and
> struct<a:x1,b:x2> on PySpark, and ArrayType(X1()) == ArrayType(X2()) is True.
> - The same happens for UserDefinedType subclasses with parameters:
> ArrayType(ArgUDT(2)) == ArrayType(ArgUDT(3)) is True.
> Nested children should be compared and rendered through the objects that the
> containers keep, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]