Hyukjin Kwon created SPARK-62015:
------------------------------------

             Summary: Convert the local rows of createDataFrame with the type 
objects of the schema, inferred too, and read tuple rows by their len and iter, 
as PySpark does
                 Key: SPARK-62015
                 URL: https://issues.apache.org/jira/browse/SPARK-62015
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
PySpark 4.3.0's createDataFrame of local rows converts them with converters of 
the type objects of
the schema (LocalDataToArrowConversion of python/pyspark/sql/conversion.py), 
and reads the rows
with the Python protocol. The drop-in's conversion 
(crates/pyspark-rs/src/infer.rs, and
create_from_local of crates/pyspark-rs/src/session.rs) differs in three ways:

1. The values of a user-defined type of an inferred schema. PySpark's 
_infer_type returns
   obj.__UDT__ (types.py:2654-2655), _merge_type keeps the first of two UDTs of 
one class
   (types.py:2921-2974), and the converter serializes every value with that 
object
   (conversion.py:846-877). The drop-in serializes each value with its own 
__UDT__ (udt_of), as the
   core type of an inferred schema holds no object. Values whose __UDT__ has 
other parameters, such
   as a scale, send other data, without an error: in a field, an array, a map, 
a struct, and the
   SQL type of another UDT.

2. Rows of a tuple subclass of its own __len__ or __iter__. PySpark's to_row 
takes a tuple by
   len(item), which must be the number of fields, and tuple(item) 
(conversion.py:944-973);
   _infer_schema names a tuple by len(row) and zips the names with row 
(types.py:2810-2821); the
   converter of a struct reads a nested tuple by len(value) and value[i] 
(conversion.py:620-640).
   The drop-in reads the values as the tuple stores them (row_values, 
convert_struct), while its
   inference iterates the row. Such a row sends its storage, a __len__ of 
another length or that
   raises raises nothing, and an __iter__ of values of other types than the 
storage fails with
   ArrowInvalid.

3. The type objects of a given schema (KD-types C7). PySpark's converters name 
the type object in
   their errors (f"input for {dataType} must not be None", 
conversion.py:599-925), and the
   converter of a spatial type calls dataType.toInternal(value) 
(conversion.py:893-916). The
   drop-in names its core type (BooleanType() of a subclass SubBool of 
BooleanType), and does not
   call the toInternal of a subclass of GeometryType or GeographyType.

Repro (master 50594b2e, against a Spark 4.3.0 server)
  class ScaleUDT(UserDefinedType):
      def __init__(self, scale=1): self.scale = scale
      @classmethod
      def sqlType(cls): return LongType()
      @classmethod
      def module(cls): return __name__
      def serialize(self, obj): return obj.v * self.scale
      def deserialize(self, datum): return datum
  class Val:
      def __init__(self, v, s): self.v, self.s = v, s
      @property
      def __UDT__(self): return ScaleUDT(self.s)
  class T(tuple):
      def __iter__(self): return iter((99, "z"))
  class Len3(tuple):
      def __len__(self): return 3
  class SubBool(BooleanType): pass

  >>> spark.createDataFrame([(Val(3, 2),), (Val(4, 10),)]).toArrow().to_pylist()
  [{'_1': 6}, {'_1': 40}]
  >>> spark.createDataFrame([T((1, "a"))], "a long, b string").collect()
  [Row(a=1, b='a')]
  >>> spark.createDataFrame([Len3((1, "a"))], "a long, b string").collect()
  [Row(a=1, b='a')]
  >>> spark.createDataFrame([(None,)], StructType([StructField("a", SubBool(), 
False)]))
  PySparkValueError: input for BooleanType() must not be None
A GeometryType subclass whose toInternal returns another dict sends the value's 
own SRID and WKB,
and its toInternal is not called.

Expected (as with pyspark-client 4.3.0)
  [{'_1': 6}, {'_1': 8}]
  [Row(a=99, b='z')]
  PySparkValueError: [AXIS_LENGTH_MISMATCH] Length mismatch: Expected axis has 
2 elements, new
  values have 3 elements.
  PySparkValueError: input for SubBool() must not be None
and the toInternal of the GeometryType subclass is called, whose result is sent.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to