Hyukjin Kwon created SPARK-61849:
------------------------------------

             Summary: Load the data types that PySpark pickled, and make the 
parameters of a type its attributes as PySpark's
                 Key: SPARK-61849
                 URL: https://issues.apache.org/jira/browse/SPARK-61849
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


PySpark's data types are plain classes, which pickle as their class and their 
__dict__ (copyreg.__newobj__ at protocols 2 and up). The drop-in's native type 
classes require their constructor arguments and have no __setstate__, so the 
pickle of a type that official PySpark made does not load right in the drop-in:
- DecimalType(10,2) loads as DecimalType(10,0), StringType('UTF8_LCASE') as 
StringType(), TimeType(3) as TimeType(6), DayTimeIntervalType(1,2) as 
DayTimeIntervalType(0,3);
- ArrayType, MapType, StructField, StructType, CharType, VarcharType, 
GeometryType and GeographyType raise TypeError ("missing 1 required positional 
argument"), so a pickled df.schema does not load at all.

Also, the parameters of a type (DecimalType.scale, StringType.collation, 
CharType.length, TimeType.precision, the interval fields, ...) are plain 
attributes in PySpark, whose assignment changes the type; in the drop-in they 
are read-only.

Make PySpark's pickles of the types load as the same types at protocols 2 to 5, 
and the parameters assignable as PySpark's.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to