Hyukjin Kwon created SPARK-61244:
------------------------------------
Summary: Convert pandas frames with MultiIndex or CategoricalIndex
labels through Arrow, as PySpark does
Key: SPARK-61244
URL: https://issues.apache.org/jira/browse/SPARK-61244
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The Python drop-in's createDataFrame of a pandas frame whose column labels are
a plain MultiIndex or CategoricalIndex takes the native row path, which
converts its cells differently from PySpark.
A list of naive datetimes becomes array<timestamp> (the session time zone), not
array<timestamp_ntz>: under Asia/Seoul, datetime(2020, 1, 2, 3) reads back as
12:00.
A list of Decimals takes decimal(38,s) instead of the smallest precision, and a
dict cell of mixed value types fails with "Type mismatch in row data" (PySpark
gives a struct).
PySpark converts these frames with Arrow, as frames with default labels already
do; the Row scan of object columns on the row path also becomes cheap.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
