[ 
https://issues.apache.org/jira/browse/SPARK-61351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61351:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Convert pandas frames with nullable or Arrow-backed column labels through 
> Arrow, as PySpark does
> ------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61351
>                 URL: https://issues.apache.org/jira/browse/SPARK-61351
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> A pandas frame whose column labels are an Index of pandas' nullable dtypes 
> (Int64, Float64, boolean: a pivot, unstack or .T of convert_dtypes() data) or 
> of Arrow-backed dtypes (int64[pyarrow], timestamp[pyarrow], date32[pyarrow], 
> ...), or a MultiIndex level or CategoricalIndex of them, took the native row 
> path of createDataFrame in the Python drop-in, which converts cells 
> differently from PySpark 4.3.0, which converts the frame through Arrow. For 
> example, an object column [1, 2.5] failed with "Type mismatch in row data" 
> (PySpark: double), a dict cell gave a map (PySpark: struct), a NaN in a list 
> stayed NaN (PySpark: None), np.int32 gave bigint (PySpark: int), and under 
> spark.sql.session.timeZone=Asia/Seoul with a client in Asia/Seoul, a 
> datetime64 column under an Int64 label with an explicit timestamp_ntz schema 
> read back 2020-01-01 18:00 (PySpark: 2020-01-02 03:00).
> SPARK-61314 fixed the same for DatetimeIndex, TimedeltaIndex, PeriodIndex and 
> IntervalIndex labels. The fix accepts the exact IntegerArray, FloatingArray, 
> BooleanArray and ArrowExtensionArray storages of a label Index on the Arrow 
> routes, with the same checks as for the string labels: pandas' own classes 
> and hooks, shadowed by no instance attribute, in pandas 2 and 3.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to