Hyukjin Kwon created SPARK-61969:
------------------------------------
Summary: Serialize the values of local rows with the user-defined
type objects of the schema in createDataFrame, as PySpark does
Key: SPARK-61969
URL: https://issues.apache.org/jira/browse/SPARK-61969
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
PySpark 4.3.0's createDataFrame of local rows serializes each value of a
user-defined type with
the type object that the schema holds:
LocalDataToArrowConversion._create_converter binds
`udt = dataType` and calls `udt.serialize(value)`
(python/pyspark/sql/conversion.py:846-877).
The drop-in's converter (udt_of of crates/pyspark-rs/src/infer.rs) used the
value's __UDT__ when
it was of the class, and otherwise made a new instance of the class with no
arguments. So:
- a UDT of a parameter that its JSON does not hold, such as a scale, silently
serialized with
the parameter's default: ScaleUDT(2) and ScaleUDT(10) of one schema both sent
the values of
ScaleUDT(), in a field, an array, a map, a struct, and the SQL type of
another UDT;
- a UDT whose constructor needs an argument could not be used at all:
createDataFrame raised
TypeError: __init__() missing 1 required positional argument.
pandas DataFrames go through PySpark's own conversion, and are right.
Repro (master 3f743c32, against a Spark 4.3.0 server)
class ScaleUDT(UserDefinedType):
def __init__(self, scale=1): self.scale = scale
@classmethod
def sqlType(cls): return LongType()
@classmethod
def module(cls): return __name__
def serialize(self, obj): return obj * self.scale
def deserialize(self, datum): return datum // self.scale
schema = StructType([StructField("a", ScaleUDT(2)), StructField("b",
ScaleUDT(10)),
StructField("c", ArrayType(ScaleUDT(10))),
StructField("d", MapType(StringType(), ScaleUDT(2)))])
>>> spark.createDataFrame([(3, 4, [5], {"k": 6})],
schema).toArrow().to_pylist()
[{'a': 3, 'b': 4, 'c': [5], 'd': [('k', 6)]}]
With a UDT OffsetUDT whose __init__ takes `offset` without a default, the
createDataFrame call
raises TypeError: OffsetUDT.__init__() missing 1 required positional argument:
'offset'.
Expected (as with pyspark-client 4.3.0)
[{'a': 6, 'b': 40, 'c': [50], 'd': [('k', 12)]}]
and the UDT whose constructor needs an argument serializes with the object of
the schema, as
the other UDTs do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]