[
https://issues.apache.org/jira/browse/SPARK-25130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782846#comment-16782846
]
Sean Owen commented on SPARK-25130:
-----------------------------------
[~maxgekk] is this likely fixed by your overhaul of time parsing?
> [Python] Wrong timestamp returned by toPandas
> ---------------------------------------------
>
> Key: SPARK-25130
> URL: https://issues.apache.org/jira/browse/SPARK-25130
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.3.0, 2.3.1
> Environment: Tested with version 2.3.1 on OSX and 2.3.0 on Linux.
> Reporter: Anton Daitche
> Priority: Major
>
> The code snippet
> {code:java}
> import datetime
> df = spark.createDataFrame([(datetime.datetime(1901, 1, 1, 0, 0, 0),)],
> ["ts"])
> print("collect:", df.collect()[0][0])
> print("toPandas:", df.toPandas().iloc[0, 0])
> {code}
> prints
> {code:java}
> collect: 1901-01-01 00:00:00
> toPandas: 1900-12-31 23:53:00{code}
> Hence the toPandas methods seems to convert the timestamp wrongly.
> The problem disappears for "1902-01-01 00:00:00" and later dates (I didn't do
> an exhaustive test though).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
