Hyukjin Kwon created SPARK-62065:
------------------------------------
Summary: [Connect][Python] Drop-in: hold PySpark's user-defined
type objects in the cached schemas of set operations and pickles, raise the
conversion errors of the schema of the server where PySpark reads it, and make
DataType.simpleString the typeName
Key: SPARK-62065
URL: https://issues.apache.org/jira/browse/SPARK-62065
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The drop-in's cached schema of a DataFrame, which a set operation compares and
a pickle carries, and some reads of the schema of the server do not hold or
convert the user-defined type (UDT) objects as pyspark-client 4.3.0 does.
1. Set operations. PySpark's _merge_cached_schema keeps the cached schema of
`this` when it equals that of `other` by ==, which compares the __dict__ of a
UDT (python/pyspark/sql/connect/dataframe.py:1203-1208). The cached schema of a
DataFrame of the schema of the server is of new UDT objects
(UserDefinedType.fromJson, python/pyspark/sql/types.py:1986-2004). The drop-in
compared it with the UDT object that it had lowered last of its JSON, often the
user's own.
u = PtUDT(); u.marker = "made"
server = spark.createDataFrame([((1.0, 2.0),)],
StructType([StructField("p", PtUDT())]))
server.columns
cast = server.to(StructType([StructField("p", u)]))
hasattr(cast.union(server).schema["p"].dataType, "marker")
# PySpark: False (the cached schemas differ, the union caches none, its
schema is the server's)
# drop-in: True (the deep copy of the schema of `to`)
2. Pickles. PySpark's __reduce__ carries _cached_schema as its objects
(dataframe.py:150-170), which the unpickled DataFrame keeps: dtypes, columns
and the methods that copy the cached schema read them, while schema is the one
that its unpickle requested. The drop-in lowered the pickled StructType into a
core type and dropped it, and pickled the registry's objects of a DataFrame of
the server.
cloudpickle.loads(cloudpickle.dumps(cast)).filter("true").schema["p"].dataType.marker
# PySpark: "made"; drop-in: AttributeError (a new object)
3. The reads of the schema. PySpark's _schema converts the schema of the server
(dataframe.py:2024-2038, python/pyspark/sql/connect/types.py:320-327), so every
read of it raises the errors of that conversion: the TypeError of a UDT whose
__init__ needs arguments, the KeyError of a UDT of no Python class,
UNSUPPORTED_OPERATION of a nanosecond timestamp. The drop-in raised them for
schema, dtypes and the rows, but not for columns, printSchema, df[0], dir(df),
a lookup of a name of two underscores, the columns of mapInPandas and
applyInPandas, copy.copy, copy.deepcopy, an unpickle, or GroupedData.sum() and
the other numeric aggregations, which read self._df.schema
(python/pyspark/sql/connect/group.py:160).
class ArgUDT(PtUDT):
def __init__(self, scale): self.scale = scale
df = spark.createDataFrame([((1.0, 2.0), 1)], StructType([StructField("p",
ArgUDT(2)), StructField("i", LongType())]))
df.columns # PySpark: TypeError: ArgUDT.__init__() missing 1
required positional argument: 'scale'; drop-in: ['p', 'i']
copy.copy(df) # PySpark: the same TypeError; drop-in: a DataFrame
df.groupBy("i").sum() # PySpark: the same TypeError; drop-in: a DataFrame
4. DataType.simpleString. PySpark's base method is self.typeName()
(types.py:134-135), which the types of a string of their own override, so
DataType.simpleString(StructType([...])) is "struct" and
DataType.simpleString(LongType()) is "long". The drop-in's gave the simple
string of the type ("struct<a:bigint>", "bigint"), of the registry's UDT
objects for a struct, array or map, and did not read the typeName() of a
subclass.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]