Hyukjin Kwon created SPARK-62015:
------------------------------------
Summary: Convert the local rows of createDataFrame with the type
objects of the schema, inferred too, and read tuple rows by their len and iter,
as PySpark does
Key: SPARK-62015
URL: https://issues.apache.org/jira/browse/SPARK-62015
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
PySpark 4.3.0's createDataFrame of local rows converts them with converters of
the type objects of
the schema (LocalDataToArrowConversion of python/pyspark/sql/conversion.py),
and reads the rows
with the Python protocol. The drop-in's conversion
(crates/pyspark-rs/src/infer.rs, and
create_from_local of crates/pyspark-rs/src/session.rs) differs in three ways:
1. The values of a user-defined type of an inferred schema. PySpark's
_infer_type returns
obj.__UDT__ (types.py:2654-2655), _merge_type keeps the first of two UDTs of
one class
(types.py:2921-2974), and the converter serializes every value with that
object
(conversion.py:846-877). The drop-in serializes each value with its own
__UDT__ (udt_of), as the
core type of an inferred schema holds no object. Values whose __UDT__ has
other parameters, such
as a scale, send other data, without an error: in a field, an array, a map,
a struct, and the
SQL type of another UDT.
2. Rows of a tuple subclass of its own __len__ or __iter__. PySpark's to_row
takes a tuple by
len(item), which must be the number of fields, and tuple(item)
(conversion.py:944-973);
_infer_schema names a tuple by len(row) and zips the names with row
(types.py:2810-2821); the
converter of a struct reads a nested tuple by len(value) and value[i]
(conversion.py:620-640).
The drop-in reads the values as the tuple stores them (row_values,
convert_struct), while its
inference iterates the row. Such a row sends its storage, a __len__ of
another length or that
raises raises nothing, and an __iter__ of values of other types than the
storage fails with
ArrowInvalid.
3. The type objects of a given schema (KD-types C7). PySpark's converters name
the type object in
their errors (f"input for {dataType} must not be None",
conversion.py:599-925), and the
converter of a spatial type calls dataType.toInternal(value)
(conversion.py:893-916). The
drop-in names its core type (BooleanType() of a subclass SubBool of
BooleanType), and does not
call the toInternal of a subclass of GeometryType or GeographyType.
Repro (master 50594b2e, against a Spark 4.3.0 server)
class ScaleUDT(UserDefinedType):
def __init__(self, scale=1): self.scale = scale
@classmethod
def sqlType(cls): return LongType()
@classmethod
def module(cls): return __name__
def serialize(self, obj): return obj.v * self.scale
def deserialize(self, datum): return datum
class Val:
def __init__(self, v, s): self.v, self.s = v, s
@property
def __UDT__(self): return ScaleUDT(self.s)
class T(tuple):
def __iter__(self): return iter((99, "z"))
class Len3(tuple):
def __len__(self): return 3
class SubBool(BooleanType): pass
>>> spark.createDataFrame([(Val(3, 2),), (Val(4, 10),)]).toArrow().to_pylist()
[{'_1': 6}, {'_1': 40}]
>>> spark.createDataFrame([T((1, "a"))], "a long, b string").collect()
[Row(a=1, b='a')]
>>> spark.createDataFrame([Len3((1, "a"))], "a long, b string").collect()
[Row(a=1, b='a')]
>>> spark.createDataFrame([(None,)], StructType([StructField("a", SubBool(),
False)]))
PySparkValueError: input for BooleanType() must not be None
A GeometryType subclass whose toInternal returns another dict sends the value's
own SRID and WKB,
and its toInternal is not called.
Expected (as with pyspark-client 4.3.0)
[{'_1': 6}, {'_1': 8}]
[Row(a=99, b='z')]
PySparkValueError: [AXIS_LENGTH_MISMATCH] Length mismatch: Expected axis has
2 elements, new
values have 3 elements.
PySparkValueError: input for SubBool() must not be None
and the toInternal of the GeometryType subclass is called, whose result is sent.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]