Anton Daitche created SPARK-25130:
-------------------------------------
Summary: Wrong timestamp returned by toPandas
Key: SPARK-25130
URL: https://issues.apache.org/jira/browse/SPARK-25130
Project: Spark
Issue Type: Bug
Components: Input/Output
Affects Versions: 2.3.1, 2.3.0
Environment: Tested with version 2.3.1 on OSX and 2.3.0 on Linux.
Reporter: Anton Daitche
The code snippet
{code:java}
import datetime
df = spark.createDataFrame([(datetime.datetime(1901, 1, 1, 0, 0, 0),)], ["ts"])
print(df.collect())
print(df.toPandas())
{code}
prints
{code:java}
collect: 1901-01-01 00:00:00
toPandas: 1900-12-31 23:53:00{code}
Hence the toPandas methods seems to convert the timestamp wrongly.
The problem disappears for "1902-01-01 00:00:00" and later dates (I didn't do
an exhaustive test though).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
