[
https://issues.apache.org/jira/browse/SPARK-62016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62016:
------------------------------------
Assignee: Hyukjin Kwon
> Treat user-defined type objects as PySpark does: keep them in the types that
> hold them until a use needs their JSON, make new ones for the schemas of the
> server, and render casts by their simpleString()
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62016
> URL: https://issues.apache.org/jira/browse/SPARK-62016
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Problem
> The drop-in differs from pyspark-client 4.3.0 in four ways on user-defined
> type (UDT) objects:
> 1. The ArrayType, MapType, StructField and StructType of a UDT whose module()
> or sqlType() is not
> implemented, such as UserDefinedType() itself, raise when they are made.
> PySpark keeps the type
> object as it is and reads it only for a use that needs its JSON or its
> conversion. So in
> PySpark ArrayType(UserDefinedType()) is made, its repr, simpleString, ==,
> hash, pickles and
> tree string work, and only json(), a plan or toInternal raise.
> 2. The schema of a DataFrame holds the UDT object that the client last
> lowered of its JSON: the
> user's own object, with its attributes. Lowering an unrelated UDT of the
> same JSON later changes
> the schema of an untouched DataFrame. PySpark's UserDefinedType.fromJson
> makes a new UDT() of the
> class (python/pyspark/sql/types.py:1986-2005), which raises TypeError for
> a class whose __init__
> takes arguments, for df.schema, df.dtypes and df.collect().
> 3. A cast to a UDT of its own simpleString() shows UDT. PySpark's
> CastExpression.__repr__ shows
> simpleString().upper() of the type object
> (python/pyspark/sql/connect/expressions.py:1004-1016).
> The same goes for a type of a Python subclass, and for the
> pyspark.mllib.linalg VectorUDT and
> MatrixUDT, of which the core names only the pyspark.ml.linalg classes.
> 4. DataFrame.to of a struct caches the core's type rather than the struct as
> given, as PySpark's
> res._cached_schema = schema does.
> Repro (master 50594b2e, against a Spark 4.3.0 server)
> >>> ArrayType(UserDefinedType())
> PySparkNotImplementedError: [NOT_IMPLEMENTED] module() is not implemented.
> >>> u = PtUDT(); u.marker = 1
> >>> df = spark.createDataFrame([(Pt(1.0, 2.0),)],
> StructType([StructField("p", u)]))
> >>> df.schema["p"].dataType is u
> True
> >>> spark.createDataFrame([(Pt(1.0, 2.0),)], StructType([StructField("p",
> ArgUDT(2))])).schema
> StructType([StructField('p', ArgUDT(), True)]) # ArgUDT.__init__
> takes a scale
> >>> F.col("x").cast(ScaleUDT(2))
> Column<'CAST(x AS UDT)'>
> >>> F.col("x").cast(pyspark.mllib.linalg.VectorUDT())
> Column<'CAST(x AS UDT)'>
> Expected (as with pyspark-client 4.3.0)
> ArrayType(UserDefinedType(), True), and its json() raises the
> NOT_IMPLEMENTED of module()
> False (a new PtUDT(), without the attribute)
> TypeError: ArgUDT.__init__() missing 1 required positional argument: 'scale'
> Column<'CAST(x AS SCALE2)'>
> Column<'CAST(x AS VECTOR)'>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]