Hyukjin Kwon created SPARK-61314:
------------------------------------
Summary: Convert pandas frames with DatetimeIndex, TimedeltaIndex,
PeriodIndex or IntervalIndex labels through Arrow, as PySpark does
Key: SPARK-61314
URL: https://issues.apache.org/jira/browse/SPARK-61314
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A pandas frame whose column labels are an exact DatetimeIndex, TimedeltaIndex,
PeriodIndex or IntervalIndex (or a MultiIndex with such a level, or a
CategoricalIndex of such categories) took the native row path of
createDataFrame, which converts cells differently from PySpark 4.3.0. Repro
under spark.sql.session.timeZone=Asia/Seoul with a client in Asia/Seoul: a
datetime64 column under a DatetimeIndex label with an explicit timestamp_ntz
schema reads back 2020-01-01 18:00 (PySpark: 2020-01-02 03:00). Also an object
column [1, 2.5] fails with "Type mismatch in row data" (PySpark: double), and a
dict cell gives a map (PySpark: struct).
Official PySpark converts these frames through Arrow, as it does for str
labels; the fix sends them through the same Arrow routes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
