Hyukjin Kwon created SPARK-62065:
------------------------------------

             Summary: [Connect][Python] Drop-in: hold PySpark's user-defined 
type objects in the cached schemas of set operations and pickles, raise the 
conversion errors of the schema of the server where PySpark reads it, and make 
DataType.simpleString the typeName
                 Key: SPARK-62065
                 URL: https://issues.apache.org/jira/browse/SPARK-62065
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The drop-in's cached schema of a DataFrame, which a set operation compares and 
a pickle carries, and some reads of the schema of the server do not hold or 
convert the user-defined type (UDT) objects as pyspark-client 4.3.0 does.

1. Set operations. PySpark's _merge_cached_schema keeps the cached schema of 
`this` when it equals that of `other` by ==, which compares the __dict__ of a 
UDT (python/pyspark/sql/connect/dataframe.py:1203-1208). The cached schema of a 
DataFrame of the schema of the server is of new UDT objects 
(UserDefinedType.fromJson, python/pyspark/sql/types.py:1986-2004). The drop-in 
compared it with the UDT object that it had lowered last of its JSON, often the 
user's own.

    u = PtUDT(); u.marker = "made"
    server = spark.createDataFrame([((1.0, 2.0),)], 
StructType([StructField("p", PtUDT())]))
    server.columns
    cast = server.to(StructType([StructField("p", u)]))
    hasattr(cast.union(server).schema["p"].dataType, "marker")
    # PySpark: False (the cached schemas differ, the union caches none, its 
schema is the server's)
    # drop-in: True (the deep copy of the schema of `to`)

2. Pickles. PySpark's __reduce__ carries _cached_schema as its objects 
(dataframe.py:150-170), which the unpickled DataFrame keeps: dtypes, columns 
and the methods that copy the cached schema read them, while schema is the one 
that its unpickle requested. The drop-in lowered the pickled StructType into a 
core type and dropped it, and pickled the registry's objects of a DataFrame of 
the server.

    
cloudpickle.loads(cloudpickle.dumps(cast)).filter("true").schema["p"].dataType.marker
    # PySpark: "made"; drop-in: AttributeError (a new object)

3. The reads of the schema. PySpark's _schema converts the schema of the server 
(dataframe.py:2024-2038, python/pyspark/sql/connect/types.py:320-327), so every 
read of it raises the errors of that conversion: the TypeError of a UDT whose 
__init__ needs arguments, the KeyError of a UDT of no Python class, 
UNSUPPORTED_OPERATION of a nanosecond timestamp. The drop-in raised them for 
schema, dtypes and the rows, but not for columns, printSchema, df[0], dir(df), 
a lookup of a name of two underscores, the columns of mapInPandas and 
applyInPandas, copy.copy, copy.deepcopy, an unpickle, or GroupedData.sum() and 
the other numeric aggregations, which read self._df.schema 
(python/pyspark/sql/connect/group.py:160).

    class ArgUDT(PtUDT):
        def __init__(self, scale): self.scale = scale
    df = spark.createDataFrame([((1.0, 2.0), 1)], StructType([StructField("p", 
ArgUDT(2)), StructField("i", LongType())]))
    df.columns            # PySpark: TypeError: ArgUDT.__init__() missing 1 
required positional argument: 'scale'; drop-in: ['p', 'i']
    copy.copy(df)         # PySpark: the same TypeError; drop-in: a DataFrame
    df.groupBy("i").sum() # PySpark: the same TypeError; drop-in: a DataFrame

4. DataType.simpleString. PySpark's base method is self.typeName() 
(types.py:134-135), which the types of a string of their own override, so 
DataType.simpleString(StructType([...])) is "struct" and 
DataType.simpleString(LongType()) is "long". The drop-in's gave the simple 
string of the type ("struct<a:bigint>", "bigint"), of the registry's UDT 
objects for a struct, array or map, and did not read the typeName() of a 
subclass.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to