[
https://issues.apache.org/jira/browse/SPARK-61314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61314:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Convert pandas frames with DatetimeIndex, TimedeltaIndex, PeriodIndex or
> IntervalIndex labels through Arrow, as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61314
> URL: https://issues.apache.org/jira/browse/SPARK-61314
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> A pandas frame whose column labels are an exact DatetimeIndex,
> TimedeltaIndex, PeriodIndex or IntervalIndex (or a MultiIndex with such a
> level, or a CategoricalIndex of such categories) took the native row path of
> createDataFrame, which converts cells differently from PySpark 4.3.0. Repro
> under spark.sql.session.timeZone=Asia/Seoul with a client in Asia/Seoul: a
> datetime64 column under a DatetimeIndex label with an explicit timestamp_ntz
> schema reads back 2020-01-01 18:00 (PySpark: 2020-01-02 03:00). Also an
> object column [1, 2.5] fails with "Type mismatch in row data" (PySpark:
> double), and a dict cell gives a map (PySpark: struct).
> Official PySpark converts these frames through Arrow, as it does for str
> labels; the fix sends them through the same Arrow routes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
