[
https://issues.apache.org/jira/browse/SPARK-61106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61106:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Convert the values of local rows of a schema, and raise their errors, as
> PyArrow does in createDataFrame in the Python client
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61106
> URL: https://issues.apache.org/jira/browse/SPARK-61106
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{spark.createDataFrame}} of local rows with a schema
> passes the values of most atomic types to PyArrow as they are. PyArrow then
> converts them, or raises its own errors:
> * An int, date, day-time interval or time field takes any value that has
> {{__index__}}, or else {{__int__}}: a {{Decimal}}, a float or a NumPy scalar.
> A bool raises {{ArrowTypeError}} ("Expected integer, got bool").
> ** {{Decimal('1.5')}} is 1 in an int field, 1970-01-02 in a date field and 1
> microsecond in a day-time interval field.
> ** A NaN in a date, day-time interval or time field raises "cannot convert
> float NaN to integer".
> * A value that PyArrow cannot convert raises {{pyarrow.lib.ArrowInvalid}}, or
> {{pyarrow.lib.ArrowTypeError}}, of PyArrow's message:
> ** "Could not convert 'x' with type str: tried to convert to int32"
> ** "tried to convert to float32" for a float field
> ** "Value 1099511627776 too large to fit in C integer type"
> ** "PyLong is too large to fit int64" for an int beyond int64 in a float or
> double field
> ** "Integer too large for date32"
> ** "object of type <class 'str'> cannot be converted to int"
> The Python client converts these values in Rust, emulating PyArrow. It
> instead:
> * raised plain {{ValueError}}s and {{TypeError}}s
> * named float32 "float"
> * raised for a {{Decimal}} in an int, date or day-time interval field
> * raised {{OverflowError}} for a large int in a date field
> * took a bool, or a NaN, in a time field
> Of 374 cases of a value of each kind in a field of each atomic type, 119
> differed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
