Hyukjin Kwon created SPARK-62016:
------------------------------------

             Summary: Treat user-defined type objects as PySpark does: keep 
them in the types that hold them until a use needs their JSON, make new ones 
for the schemas of the server, and render casts by their simpleString()
                 Key: SPARK-62016
                 URL: https://issues.apache.org/jira/browse/SPARK-62016
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
The drop-in differs from pyspark-client 4.3.0 in four ways on user-defined type 
(UDT) objects:

1. The ArrayType, MapType, StructField and StructType of a UDT whose module() 
or sqlType() is not
   implemented, such as UserDefinedType() itself, raise when they are made. 
PySpark keeps the type
   object as it is and reads it only for a use that needs its JSON or its 
conversion. So in
   PySpark ArrayType(UserDefinedType()) is made, its repr, simpleString, ==, 
hash, pickles and
   tree string work, and only json(), a plan or toInternal raise.
2. The schema of a DataFrame holds the UDT object that the client last lowered 
of its JSON: the
   user's own object, with its attributes. Lowering an unrelated UDT of the 
same JSON later changes
   the schema of an untouched DataFrame. PySpark's UserDefinedType.fromJson 
makes a new UDT() of the
   class (python/pyspark/sql/types.py:1986-2005), which raises TypeError for a 
class whose __init__
   takes arguments, for df.schema, df.dtypes and df.collect().
3. A cast to a UDT of its own simpleString() shows UDT. PySpark's 
CastExpression.__repr__ shows
   simpleString().upper() of the type object 
(python/pyspark/sql/connect/expressions.py:1004-1016).
   The same goes for a type of a Python subclass, and for the 
pyspark.mllib.linalg VectorUDT and
   MatrixUDT, of which the core names only the pyspark.ml.linalg classes.
4. DataFrame.to of a struct caches the core's type rather than the struct as 
given, as PySpark's
   res._cached_schema = schema does.

Repro (master 50594b2e, against a Spark 4.3.0 server)
  >>> ArrayType(UserDefinedType())
  PySparkNotImplementedError: [NOT_IMPLEMENTED] module() is not implemented.
  >>> u = PtUDT(); u.marker = 1
  >>> df = spark.createDataFrame([(Pt(1.0, 2.0),)], 
StructType([StructField("p", u)]))
  >>> df.schema["p"].dataType is u
  True
  >>> spark.createDataFrame([(Pt(1.0, 2.0),)], StructType([StructField("p", 
ArgUDT(2))])).schema
  StructType([StructField('p', ArgUDT(), True)])        # ArgUDT.__init__ takes 
a scale
  >>> F.col("x").cast(ScaleUDT(2))
  Column<'CAST(x AS UDT)'>
  >>> F.col("x").cast(pyspark.mllib.linalg.VectorUDT())
  Column<'CAST(x AS UDT)'>

Expected (as with pyspark-client 4.3.0)
  ArrayType(UserDefinedType(), True), and its json() raises the NOT_IMPLEMENTED 
of module()
  False (a new PtUDT(), without the attribute)
  TypeError: ArgUDT.__init__() missing 1 required positional argument: 'scale'
  Column<'CAST(x AS SCALE2)'>
  Column<'CAST(x AS VECTOR)'>




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to