Hyukjin Kwon created SPARK-61917:
------------------------------------
Summary: Compare and render nested data types by their own
objects, so that types of the same JSON stay distinct
Key: SPARK-61917
URL: https://issues.apache.org/jira/browse/SPARK-61917
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
The drop-in keeps a type nested in an ArrayType, a MapType or a StructType as a
core type, and finds its Python object again through a registry keyed by the
type's JSON. So two nested types of the same JSON are confused, where PySpark
compares and renders the objects themselves:
- t1, t2 = AtomicType(), AtomicType(); t1.tag, t2.tag = 1, 2: ArrayType(t1) ==
ArrayType(t2) is True on the drop-in and False on pyspark-client 4.3.0, and the
same for StructType.
- Two Python subclasses of DataType whose typeName stems are equal but whose
simpleString differ: StructType([StructField("a", X1()), StructField("b",
X2())]).simpleString() is struct<a:x2,b:x2> on the drop-in and
struct<a:x1,b:x2> on PySpark, and ArrayType(X1()) == ArrayType(X2()) is True.
- The same happens for UserDefinedType subclasses with parameters:
ArrayType(ArgUDT(2)) == ArrayType(ArgUDT(3)) is True.
Nested children should be compared and rendered through the objects that the
containers keep, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]