[ 
https://issues.apache.org/jira/browse/SPARK-61969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61969:
-----------------------------------
    Labels: pull-request-available  (was: )

> Serialize the values of local rows with the user-defined type objects of the 
> schema in createDataFrame, as PySpark does
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61969
>                 URL: https://issues.apache.org/jira/browse/SPARK-61969
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark 4.3.0's createDataFrame of local rows serializes each value of a 
> user-defined type with
> the type object that the schema holds: 
> LocalDataToArrowConversion._create_converter binds
> `udt = dataType` and calls `udt.serialize(value)` 
> (python/pyspark/sql/conversion.py:846-877).
> The drop-in's converter (udt_of of crates/pyspark-rs/src/infer.rs) used the 
> value's __UDT__ when
> it was of the class, and otherwise made a new instance of the class with no 
> arguments. So:
> - a UDT of a parameter that its JSON does not hold, such as a scale, silently 
> serialized with
>   the parameter's default: ScaleUDT(2) and ScaleUDT(10) of one schema both 
> sent the values of
>   ScaleUDT(), in a field, an array, a map, a struct, and the SQL type of 
> another UDT;
> - a UDT whose constructor needs an argument could not be used at all: 
> createDataFrame raised
>   TypeError: __init__() missing 1 required positional argument.
> pandas DataFrames go through PySpark's own conversion, and are right.
> Repro (master 3f743c32, against a Spark 4.3.0 server)
>   class ScaleUDT(UserDefinedType):
>       def __init__(self, scale=1): self.scale = scale
>       @classmethod
>       def sqlType(cls): return LongType()
>       @classmethod
>       def module(cls): return __name__
>       def serialize(self, obj): return obj * self.scale
>       def deserialize(self, datum): return datum // self.scale
>   schema = StructType([StructField("a", ScaleUDT(2)), StructField("b", 
> ScaleUDT(10)),
>                        StructField("c", ArrayType(ScaleUDT(10))),
>                        StructField("d", MapType(StringType(), ScaleUDT(2)))])
>   >>> spark.createDataFrame([(3, 4, [5], {"k": 6})], 
> schema).toArrow().to_pylist()
>   [{'a': 3, 'b': 4, 'c': [5], 'd': [('k', 6)]}]
> With a UDT OffsetUDT whose __init__ takes `offset` without a default, the 
> createDataFrame call
> raises TypeError: OffsetUDT.__init__() missing 1 required positional 
> argument: 'offset'.
> Expected (as with pyspark-client 4.3.0)
>   [{'a': 6, 'b': 40, 'c': [50], 'd': [('k', 12)]}]
> and the UDT whose constructor needs an argument serializes with the object of 
> the schema, as
> the other UDTs do.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to