Hyukjin Kwon created SPARK-62016:
------------------------------------
Summary: Treat user-defined type objects as PySpark does: keep
them in the types that hold them until a use needs their JSON, make new ones
for the schemas of the server, and render casts by their simpleString()
Key: SPARK-62016
URL: https://issues.apache.org/jira/browse/SPARK-62016
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
The drop-in differs from pyspark-client 4.3.0 in four ways on user-defined type
(UDT) objects:
1. The ArrayType, MapType, StructField and StructType of a UDT whose module()
or sqlType() is not
implemented, such as UserDefinedType() itself, raise when they are made.
PySpark keeps the type
object as it is and reads it only for a use that needs its JSON or its
conversion. So in
PySpark ArrayType(UserDefinedType()) is made, its repr, simpleString, ==,
hash, pickles and
tree string work, and only json(), a plan or toInternal raise.
2. The schema of a DataFrame holds the UDT object that the client last lowered
of its JSON: the
user's own object, with its attributes. Lowering an unrelated UDT of the
same JSON later changes
the schema of an untouched DataFrame. PySpark's UserDefinedType.fromJson
makes a new UDT() of the
class (python/pyspark/sql/types.py:1986-2005), which raises TypeError for a
class whose __init__
takes arguments, for df.schema, df.dtypes and df.collect().
3. A cast to a UDT of its own simpleString() shows UDT. PySpark's
CastExpression.__repr__ shows
simpleString().upper() of the type object
(python/pyspark/sql/connect/expressions.py:1004-1016).
The same goes for a type of a Python subclass, and for the
pyspark.mllib.linalg VectorUDT and
MatrixUDT, of which the core names only the pyspark.ml.linalg classes.
4. DataFrame.to of a struct caches the core's type rather than the struct as
given, as PySpark's
res._cached_schema = schema does.
Repro (master 50594b2e, against a Spark 4.3.0 server)
>>> ArrayType(UserDefinedType())
PySparkNotImplementedError: [NOT_IMPLEMENTED] module() is not implemented.
>>> u = PtUDT(); u.marker = 1
>>> df = spark.createDataFrame([(Pt(1.0, 2.0),)],
StructType([StructField("p", u)]))
>>> df.schema["p"].dataType is u
True
>>> spark.createDataFrame([(Pt(1.0, 2.0),)], StructType([StructField("p",
ArgUDT(2))])).schema
StructType([StructField('p', ArgUDT(), True)]) # ArgUDT.__init__ takes
a scale
>>> F.col("x").cast(ScaleUDT(2))
Column<'CAST(x AS UDT)'>
>>> F.col("x").cast(pyspark.mllib.linalg.VectorUDT())
Column<'CAST(x AS UDT)'>
Expected (as with pyspark-client 4.3.0)
ArrayType(UserDefinedType(), True), and its json() raises the NOT_IMPLEMENTED
of module()
False (a new PtUDT(), without the attribute)
TypeError: ArgUDT.__init__() missing 1 required positional argument: 'scale'
Column<'CAST(x AS SCALE2)'>
Column<'CAST(x AS VECTOR)'>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]