Bryan Cutler created SPARK-21375:
------------------------------------
Summary: Add date and timestamp support to ArrowConverters for
toPandas() collection
Key: SPARK-21375
URL: https://issues.apache.org/jira/browse/SPARK-21375
Project: Spark
Issue Type: Improvement
Components: PySpark, SQL
Affects Versions: 2.3.0
Reporter: Bryan Cutler
Date and timestamp are not yet supported in DataFrame.toPandas() using
ArrowConverters. These are common types for data analysis used in both Spark
and Pandas and should be supported.
There is a discrepancy with the way that PySpark and Arrow store timestamps,
without timezone specified, internally. PySpark takes a UTC timestamp that is
adjusted to local time and Arrow is in UTC time. Hopefully there is a clean
way to resolve this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
