[
https://issues.apache.org/jira/browse/SPARK-61969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61969:
------------------------------------
Assignee: Hyukjin Kwon
> Serialize the values of local rows with the user-defined type objects of the
> schema in createDataFrame, as PySpark does
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61969
> URL: https://issues.apache.org/jira/browse/SPARK-61969
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark 4.3.0's createDataFrame of local rows serializes each value of a
> user-defined type with
> the type object that the schema holds:
> LocalDataToArrowConversion._create_converter binds
> `udt = dataType` and calls `udt.serialize(value)`
> (python/pyspark/sql/conversion.py:846-877).
> The drop-in's converter (udt_of of crates/pyspark-rs/src/infer.rs) used the
> value's __UDT__ when
> it was of the class, and otherwise made a new instance of the class with no
> arguments. So:
> - a UDT of a parameter that its JSON does not hold, such as a scale, silently
> serialized with
> the parameter's default: ScaleUDT(2) and ScaleUDT(10) of one schema both
> sent the values of
> ScaleUDT(), in a field, an array, a map, a struct, and the SQL type of
> another UDT;
> - a UDT whose constructor needs an argument could not be used at all:
> createDataFrame raised
> TypeError: __init__() missing 1 required positional argument.
> pandas DataFrames go through PySpark's own conversion, and are right.
> Repro (master 3f743c32, against a Spark 4.3.0 server)
> class ScaleUDT(UserDefinedType):
> def __init__(self, scale=1): self.scale = scale
> @classmethod
> def sqlType(cls): return LongType()
> @classmethod
> def module(cls): return __name__
> def serialize(self, obj): return obj * self.scale
> def deserialize(self, datum): return datum // self.scale
> schema = StructType([StructField("a", ScaleUDT(2)), StructField("b",
> ScaleUDT(10)),
> StructField("c", ArrayType(ScaleUDT(10))),
> StructField("d", MapType(StringType(), ScaleUDT(2)))])
> >>> spark.createDataFrame([(3, 4, [5], {"k": 6})],
> schema).toArrow().to_pylist()
> [{'a': 3, 'b': 4, 'c': [5], 'd': [('k', 6)]}]
> With a UDT OffsetUDT whose __init__ takes `offset` without a default, the
> createDataFrame call
> raises TypeError: OffsetUDT.__init__() missing 1 required positional
> argument: 'offset'.
> Expected (as with pyspark-client 4.3.0)
> [{'a': 6, 'b': 40, 'c': [50], 'd': [('k', 12)]}]
> and the UDT whose constructor needs an argument serializes with the object of
> the schema, as
> the other UDTs do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]