Hyukjin Kwon created SPARK-61351:
------------------------------------
Summary: Convert pandas frames with nullable or Arrow-backed
column labels through Arrow, as PySpark does
Key: SPARK-61351
URL: https://issues.apache.org/jira/browse/SPARK-61351
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A pandas frame whose column labels are an Index of pandas' nullable dtypes
(Int64, Float64, boolean: a pivot, unstack or .T of convert_dtypes() data) or
of Arrow-backed dtypes (int64[pyarrow], timestamp[pyarrow], date32[pyarrow],
...), or a MultiIndex level or CategoricalIndex of them, took the native row
path of createDataFrame in the Python drop-in, which converts cells differently
from PySpark 4.3.0, which converts the frame through Arrow. For example, an
object column [1, 2.5] failed with "Type mismatch in row data" (PySpark:
double), a dict cell gave a map (PySpark: struct), a NaN in a list stayed NaN
(PySpark: None), np.int32 gave bigint (PySpark: int), and under
spark.sql.session.timeZone=Asia/Seoul with a client in Asia/Seoul, a datetime64
column under an Int64 label with an explicit timestamp_ntz schema read back
2020-01-01 18:00 (PySpark: 2020-01-02 03:00).
SPARK-61314 fixed the same for DatetimeIndex, TimedeltaIndex, PeriodIndex and
IntervalIndex labels. The fix accepts the exact IntegerArray, FloatingArray,
BooleanArray and ArrowExtensionArray storages of a label Index on the Arrow
routes, with the same checks as for the string labels: pandas' own classes and
hooks, shadowed by no instance attribute, in pandas 2 and 3.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]