[
https://issues.apache.org/jira/browse/SPARK-31758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17115786#comment-17115786
]
Hyukjin Kwon commented on SPARK-31758:
--------------------------------------
This was fixed in the current master:
{code}
>>> schema = StructType([StructField("timestamp", TimestampType())])
>>> df = spark.createDataFrame([('{"timestamp":"2020-01-01T20:00:00.900125Z"}',
>>> )], ["body"])
>>> df.select(from_json("body", schema)).collect()
[Row(from_json(body)=Row(timestamp=datetime.datetime(2020, 1, 2, 5, 0, 0,
900125)))]
{code}
probably by switching the Java 7 datetime APIs to Java 8 datetime APIs, which
can't be ported back into 2.4.x. Let's leave it resolved as technically it
can't be reproduced in the current master.
> Incorrect timestamp parsing from JSON
> -------------------------------------
>
> Key: SPARK-31758
> URL: https://issues.apache.org/jira/browse/SPARK-31758
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.5
> Reporter: Tomi Ruokola
> Priority: Major
>
> Parsing a json string into a TimestampType can give incorrect results.
> {code:python}
> schema = StructType([StructField("timestamp", TimestampType())])
> df = spark.createDataFrame([('{"timestamp":"2020-01-01T20:00:00.900125Z"}',
> )], ["body"])
> df.select(from_json("body", schema)).collect(){code}
> Output:
> {code:python}
> datetime.datetime(2020, 1, 1, 20, 15, 0, 125000){code}
> This seems to happen when the timestamp has sub-millisecond precision and a Z
> suffix. For example, if the fraction is .900125, the output fraction is .125
> and 900 seconds is added to the timestamp.
> Workaround: Adding the timestampFormat option fixes the problem, even if the
> format string is not exactly correct.
> {code:python}
> df.select(from_json("body", schema, {"timestampFormat": "yyyy-MM-dd
> HH:mm:ss"})).collect()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
