[
https://issues.apache.org/jira/browse/SPARK-61351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61351:
-----------------------------------
Labels: pull-request-available (was: )
> Convert pandas frames with nullable or Arrow-backed column labels through
> Arrow, as PySpark does
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-61351
> URL: https://issues.apache.org/jira/browse/SPARK-61351
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> A pandas frame whose column labels are an Index of pandas' nullable dtypes
> (Int64, Float64, boolean: a pivot, unstack or .T of convert_dtypes() data) or
> of Arrow-backed dtypes (int64[pyarrow], timestamp[pyarrow], date32[pyarrow],
> ...), or a MultiIndex level or CategoricalIndex of them, took the native row
> path of createDataFrame in the Python drop-in, which converts cells
> differently from PySpark 4.3.0, which converts the frame through Arrow. For
> example, an object column [1, 2.5] failed with "Type mismatch in row data"
> (PySpark: double), a dict cell gave a map (PySpark: struct), a NaN in a list
> stayed NaN (PySpark: None), np.int32 gave bigint (PySpark: int), and under
> spark.sql.session.timeZone=Asia/Seoul with a client in Asia/Seoul, a
> datetime64 column under an Int64 label with an explicit timestamp_ntz schema
> read back 2020-01-01 18:00 (PySpark: 2020-01-02 03:00).
> SPARK-61314 fixed the same for DatetimeIndex, TimedeltaIndex, PeriodIndex and
> IntervalIndex labels. The fix accepts the exact IntegerArray, FloatingArray,
> BooleanArray and ArrowExtensionArray storages of a label Index on the Arrow
> routes, with the same checks as for the string labels: pandas' own classes
> and hooks, shadowed by no instance attribute, in pandas 2 and 3.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]