Hyukjin Kwon created SPARK-61969:
------------------------------------

             Summary: Serialize the values of local rows with the user-defined 
type objects of the schema in createDataFrame, as PySpark does
                 Key: SPARK-61969
                 URL: https://issues.apache.org/jira/browse/SPARK-61969
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
PySpark 4.3.0's createDataFrame of local rows serializes each value of a 
user-defined type with
the type object that the schema holds: 
LocalDataToArrowConversion._create_converter binds
`udt = dataType` and calls `udt.serialize(value)` 
(python/pyspark/sql/conversion.py:846-877).
The drop-in's converter (udt_of of crates/pyspark-rs/src/infer.rs) used the 
value's __UDT__ when
it was of the class, and otherwise made a new instance of the class with no 
arguments. So:
- a UDT of a parameter that its JSON does not hold, such as a scale, silently 
serialized with
  the parameter's default: ScaleUDT(2) and ScaleUDT(10) of one schema both sent 
the values of
  ScaleUDT(), in a field, an array, a map, a struct, and the SQL type of 
another UDT;
- a UDT whose constructor needs an argument could not be used at all: 
createDataFrame raised
  TypeError: __init__() missing 1 required positional argument.
pandas DataFrames go through PySpark's own conversion, and are right.

Repro (master 3f743c32, against a Spark 4.3.0 server)
  class ScaleUDT(UserDefinedType):
      def __init__(self, scale=1): self.scale = scale
      @classmethod
      def sqlType(cls): return LongType()
      @classmethod
      def module(cls): return __name__
      def serialize(self, obj): return obj * self.scale
      def deserialize(self, datum): return datum // self.scale

  schema = StructType([StructField("a", ScaleUDT(2)), StructField("b", 
ScaleUDT(10)),
                       StructField("c", ArrayType(ScaleUDT(10))),
                       StructField("d", MapType(StringType(), ScaleUDT(2)))])
  >>> spark.createDataFrame([(3, 4, [5], {"k": 6})], 
schema).toArrow().to_pylist()
  [{'a': 3, 'b': 4, 'c': [5], 'd': [('k', 6)]}]
With a UDT OffsetUDT whose __init__ takes `offset` without a default, the 
createDataFrame call
raises TypeError: OffsetUDT.__init__() missing 1 required positional argument: 
'offset'.

Expected (as with pyspark-client 4.3.0)
  [{'a': 6, 'b': 40, 'c': [50], 'd': [('k', 12)]}]
and the UDT whose constructor needs an argument serializes with the object of 
the schema, as
the other UDTs do.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to